Technical SEO Executive

  • 0
  • 0

Tools used in this guide:

Whether you’re migrating to a new website with a different URL structure, or just to a new domain or TLD (Top level domain), it is crucial that you redirect all existing links AND backlinks to the new destination or else you risk losing the entire history and authority that your domain has.

Backlinks are still an important part of Google’s ranking algorithm, and if they are handled incorrectly during a migration, this will negatively impact your site’s performance in the SERPs.

In this article, I am going to show you the best methods on how to properly manage large scale redirects (I’m talking at least 10,000+) in SEO site migrations, as well as some common issues to look out for, and how to best implement the redirects.

There are plenty of guides out there already that cover all areas of a SEO migration, but today I am going focus specifically on large scale redirects and some of the best ways to cope with them that I’ve learned from experience.

Step One – Gather All Necessary Data (URLs)

Start with crawling your current site and the staging site and put both sets of data into a spreadsheet under different tabs named appropriately, then you will want to filter on the current site export for ‘301s’ and cut and paste these URLs into another sheet named appropriately too.

Remove & Avoid Redirect Chains!

We are probably going to want to kill all these existing redirects and redo them ourselves to avoid redirect chains – this is something a lot of people forget to do in my experience.

Next we want to gather all the backlinks pointing to our site and sort them in our spreadsheet, to do this you will need to use third party tools such as MajesticSEO (both fresh and historic index) and AHREFs, but also make sure to get the data from Google Search Console and Bing (this is valuable!).

It’s important to get as much backlink data as possible in order not to lose any link equity, and this can in fact boost your site’s performance as you’ll find you’ll be repairing a lot old links that have 404’d because they haven’t been redirected properly in the past. It’s also important to remember that not all tools pick up all backlinks, so using a variety is necessary.

Organising the data

Now we have all our data, we need to clean it up a little and make it easier to work with, especially if we are handling tens of thousands of URLs, so the first step is to get rid of everything we don’t need – we only need the destination of where the links are pointing to, so everything else can be deleted.

We also want to know the source of all these URLs when combining them all together, so add another column called ‘source’ in each sheet and fill down where the data has come from.

Finally, all the URLs need to be merged together and de-duplicated in another sheet that we’ll call ‘Master’, then you’ll be left with a pretty definitive list of all URLs connected to your site that will need redirecting, this will give us a strong foundation for our migration, your excel sheet tabs should look something like this:

Your excel workbook tabs should be organised like this

Step Two – Mapping New URL Destinations

Now depending on how many URLs you have in this list; it can seem daunting to map all these thousands of URLs and it will be harrowing if you go through them all one by one to do this.

Note: You should not be redirecting images as these can all be (and should be) ported over across to the new destination.

We are going to try to find patterns that will enable us to strike off large amounts of redirects in one swoop, how easy this is will depend on how well you know the site and how much the URL structure has changed. So, if all you’re doing is a change of TLD or domain, you can simply do a find replace on all your URLs and swap out the bits you need to, but if your URL structure is changing because it’s a new website or an architectural revamp, it becomes a bit more difficult.


Let’s assume the latter and start by ordering the list in alphabetical order, this enables us to get a grasp of the list better, and will group the categories together.

Now filter the list to focus on specific sections of the site, for example the ‘magazine’ or ‘blog’, breaking it down into chunks like this makes it a lot easier to manage.

You’ll typically find in the blog section of a site the URL of the blog itself has not changed, this makes life very easy for us because you then do a find & replace on all the URL up to the blog and update it to the new path.

You may also find this with products if it’s an eCommerce site, it may not be as simple as one find & replace though as each category for each product may have changed, but it’s the same idea and will be a lot faster than individual redirects.

Link destinations that have changed completely/old back links that haven’t been redirected already should be redirected with as close as possible one on one equivalency between the old and new page, these will most likely have to be done individually.

Tip: A good way to see if you have lots of file names already matching (the last part of the URL) is to use this formula to strip down the URLs to just the last part for both the current destination and the target location:


So now you’ll have the two final parts of the URLs, can you perform a VLOOKUP in a new column and return the full URL if the two file names are matching using this formula:

=IFNA(VLOOKUP([@[Old URL (relative)]],Table2[[New URL (relative)]:[New URLs]],2,FALSE),”Not There”)

The finished table will look something like this, where the ‘(relative)’ URL is the first formula and ‘Target URL’ is the second formula:

Example plotting redirects using formulas in Excel

Step Three – Implementing & Testing

Now all the hard work is done (how long this took varies massively depending on many factors) it’s time get these redirects implemented and tested.

There are a few ways of doing this, I have used formulas in excel before that automatically generate the .htaccess rewrite rule, but this can become a problem when you begin to get to a high number of URLs, depending on how many columns you have, because it can make the spreadsheet lag.

Instead, a much better way to generate these redirects is using a mass redirect generator tool, such as the one we have created here, this can generate up to 100,000 .htaccess redirects at a time (which is enough for most cases).


Once these have been implemented it is then time to test them – the best way to do this is to crawl your original list of URLs for the current site (the ones from step one) and look at the status code for each URL – if the redirects have worked they should all redirect a 301.

Some things to look out for that will require fixing are broken links (404 pages) and links to the staging site, you can fix both these issues in the same way by identifying the offending link(s) in Screaming Frog.

If you see a 404 or a link to the staging click on the URL in the list and at the bottom click on the ‘InLinks’ tab and it will display every internal link pointing to that page and even what the anchor text is, making it easier to find and update.

Screaming Frog ‘InLinks’ tab is useful for spotting where offending links are coming from


To summarise, there’s no magic way (short of being a REGEX wizard) to instantly generate all your redirects that point to all the right locations – you must put the work in but you can make sure you’re putting the work in the right places to get it done as soon and accurately as possible.

Casting your net big enough to gather as many link signals as you can is crucial, as is avoiding redirect chains and loops, and finally being smart about spotting patterns and not wasting huge amounts of time in the wrong places.





  • 0
  • 0
  • Aarron,

    I’ve looked for years for articles explaining how other SEOs deal with redirections in large scale migration projects. You’ve put out a piece with real meat in it. A few nitpicks if I may:

    When you’ve got a website migration project with millions of redirections, you just cannot do without regular expressions.

    If you produce a large .htaccess file with millions of one-to-one redirections you’ll incur a performance penalty. With regular expressions you’ll get the job done with a much smaller amount of URL rewriting/redirection rules.

    The URL rewriting/redirection rules on most platforms are executed line by line (rule by rule) and sequencing them properly is of critical importance as the logic behind the Nth rule relies on a previous rule catching some types of links. Some lazy dev teams make their lives easier by simplifying the redirection/rewriting rules in a way that results in redirection chains (in fact most of the times I’ve let developers do the regular expression rules – they produce chains of redirections).

    Add a bit of complexity from the handling of www and non-www URL variants of the pre-migration URLs and the task is far from trivial.

    In the last testing phase, it is not enough to just check the status code of the pre-migration URLs as they can be redirected to the wrong final URL.

    Anyway, it all comes down to writing and using custom tools to do the auditing of the redirections.

    Years ago I wrote a custom tool that allowed me to write the mapping rules once and it would check all URLs, follow the redirections and grade every URL (pass/fail). The tool would catch 302, 404, redirect chains, redirecting to the wrong final URL, non-200 status codes for the final URL etc.

    Last year I turned it into a software product. You can read how the redirections get tested using mapping rules here: