How to use Unavailable_After tags to increase crawl efficiency
Working with large websites (1 million+ pages), or websites with high levels of time-sensitive content can sometimes be tricky when it comes to ensuring new content is crawled (and processed) so that users can discover it as quickly as possible.
This is where the unavailable_after meta robots tag can come in handy.
Unavailable_after tags provide a signal to search engines, including Google, that lets them know that they shouldn’t crawl a specific page after a certain date.
Unlike noindex, however, unavailable_after is a hint and not a directive, meaning it can technically be ignored and Google.
When respected, a search engine bot will not crawl the page as frequently and instead repurpose its crawl resource on avenues left open.
It’s important to note that this isn’t the same as a noindex, and isn’t a method for removal from the index, so the page may be visited occasionally to register the unavailable_after is still in force.
The page still has the potential to appear within search results pages.
Formatting The Tag
Unavailable_after can be implemented using different date and time formats, but from experience, it’s good to keep these consistent to not cause any issues in configuration or troubleshooting.
These are the ways the date/time can be formatted:
And will look something like this:
unavailable_after: 8/5/2021 13:00:00 +8
You don’t need to specify the exact time or timezone, but:
- If you specify the time, but not a timezone, the rule will default to the user timezone.
- If you don’t specify either, the default time and timezone will be used.
These can be implemented with other crawl optimization tactics, such as dynamic HTTP Headers to implement 304 Not Modified, or using tools like Cloudflare Workers or Fastly to implement SEO fixes through cloud computing (edge SEO).