Tools used in this guide:

• Screaming Frog
• AHREFs
• MajesticSEO
• SALT’s redirect tool

Whether you’re migrating to a new website with a different URL structure, or just to a new domain or TLD (Top level domain), it is crucial that you redirect all existing links AND backlinks to the new destination or else you risk losing the entire history and authority that your domain has.

Backlinks are still an important part of Google’s ranking algorithm, and if they are handled incorrectly during a migration, this will negatively impact your site’s performance in the SERPs.

In this article, I am going to show you the best methods on how to properly manage large scale redirects (I’m talking at least 10,000+) in SEO site migrations, as well as some common issues to look out for, and how to best implement the redirects.

There are plenty of guides out there already that cover all areas of a SEO migration, but today I am going focus specifically on large scale redirects and some of the best ways to cope with them that I’ve learned from experience.

Step One – Gather All Necessary Data (URLs)

Start with crawling your current site and the staging site and put both sets of data into a spreadsheet under different tabs named appropriately, then you will want to filter on the current site export for ‘301s’ and cut and paste these URLs into another sheet named appropriately too.

Remove & Avoid Redirect Chains!

We are probably going to want to kill all these existing redirects and redo them ourselves to avoid redirect chains – this is something a lot of people forget to do in my experience.

Next we want to gather all the backlinks pointing to our site and sort them in our spreadsheet, to do this you will need to use third party tools such as MajesticSEO (both fresh and historic index) and AHREFs, but also make sure to get the data from Google Search Console and Bing (this is valuable!).

It’s important to get as much backlink data as possible in order not to lose any link equity, and this can in fact boost your site’s performance as you’ll find you’ll be repairing a lot old links that have 404’d because they haven’t been redirected properly in the past. It’s also important to remember that not all tools pick up all backlinks, so using a variety is necessary.

Organising the data

Now we have all our data, we need to clean it up a little and make it easier to work with, especially if we are handling tens of thousands of URLs, so the first step is to get rid of everything we don’t need – we only need the destination of where the links are pointing to, so everything else can be deleted.

We also want to know the source of all these URLs when combining them all together, so add another column called ‘source’ in each sheet and fill down where the data has come from.

Finally, all the URLs need to be merged together and de-duplicated in another sheet that we’ll call ‘Master’, then you’ll be left with a pretty definitive list of all URLs connected to your site that will need redirecting, this will give us a strong foundation for our migration, your excel sheet tabs should look something like this:

Step Two – Mapping New URL Destinations

Now depending on how many URLs you have in this list; it can seem daunting to map all these thousands of URLs and it will be harrowing if you go through them all one by one to do this.

Note: You should not be redirecting images as these can all be (and should be) ported over across to the new destination.

We are going to try to find patterns that will enable us to strike off large amounts of redirects in one swoop, how easy this is will depend on how well you know the site and how much the URL structure has changed. So, if all you’re doing is a change of TLD or domain, you can simply do a find replace on all your URLs and swap out the bits you need to, but if your URL structure is changing because it’s a new website or an architectural revamp, it becomes a bit more difficult.

Filtering

Let’s assume the latter and start by ordering the list in alphabetical order, this enables us to get a grasp of the list better, and will group the categories together.

Now filter the list to focus on specific sections of the site, for example the ‘magazine’ or ‘blog’, breaking it down into chunks like this makes it a lot easier to manage.

You’ll typically find in the blog section of a site the URL of the blog itself has not changed, this makes life very easy for us because you then do a find & replace on all the URL up to the blog and update it to the new path.

You may also find this with products if it’s an eCommerce site, it may not be as simple as one find & replace though as each category for each product may have changed, but it’s the same idea and will be a lot faster than individual redirects.

Link destinations that have changed completely/old back links that haven’t been redirected already should be redirected with as close as possible one on one equivalency between the old and new page, these will most likely have to be done individually.

Tip: A good way to see if you have lots of file names already matching (the last part of the URL) is to use this formula to strip down the URLs to just the last part for both the current destination and the target location:

=RIGHT(A1,LEN(A1)-FIND(“::”,SUBSTITUTE(A1,”/”,”::”,LEN(A1)-LEN(SUBSTITUTE(A1,”/”,””)))))

So now you’ll have the two final parts of the URLs, can you perform a VLOOKUP in a new column and return the full URL if the two file names are matching using this formula:

=IFNA(VLOOKUP([@[Old URL (relative)]],Table2[[New URL (relative)]:[New URLs]],2,FALSE),”Not There”)

The finished table will look something like this, where the ‘(relative)’ URL is the first formula and ‘Target URL’ is the second formula:

Step Three – Implementing & Testing

Now all the hard work is done (how long this took varies massively depending on many factors) it’s time get these redirects implemented and tested.

There are a few ways of doing this, I have used formulas in excel before that automatically generate the .htaccess rewrite rule, but this can become a problem when you begin to get to a high number of URLs, depending on how many columns you have, because it can make the spreadsheet lag.

Instead, a much better way to generate these redirects is using a mass redirect generator tool, such as the one we have created here, this can generate up to 100,000 .htaccess redirects at a time (which is enough for most cases).

Testing

Once these have been implemented it is then time to test them – the best way to do this is to crawl your original list of URLs for the current site (the ones from step one) and look at the status code for each URL – if the redirects have worked they should all redirect a 301.

Some things to look out for that will require fixing are broken links (404 pages) and links to the staging site, you can fix both these issues in the same way by identifying the offending link(s) in Screaming Frog.

If you see a 404 or a link to the staging click on the URL in the list and at the bottom click on the ‘InLinks’ tab and it will display every internal link pointing to that page and even what the anchor text is, making it easier to find and update.

Summary

To summarise, there’s no magic way (short of being a REGEX wizard) to instantly generate all your redirects that point to all the right locations – you must put the work in but you can make sure you’re putting the work in the right places to get it done as soon and accurately as possible.

Casting your net big enough to gather as many link signals as you can is crucial, as is avoiding redirect chains and loops, and finally being smart about spotting patterns and not wasting huge amounts of time in the wrong places.