HTTP status codes for SEO: 404, 410, 503, and 429

A status code is the first thing a crawler reads on every request, before it gets to your content, your schema, or your meta tags. Get the wrong one and Googlebot, Bingbot, or an AI crawler draws the wrong conclusion about a page that might otherwise rank or get cited. Most sites handle 200, 301, and a plain 404 correctly and stop there. The codes that quietly cause damage are the ones teams rarely think about: 410, 503, and 429.

Why this matters more now

AI crawlers hit sites on a different schedule than Googlebot, often with less patience and a smaller crawl budget per site. If a maintenance window returns the wrong code, or a removed page returns something a crawler reads as "try again later" instead of "gone," you can end up with stale pages sitting in an index, or good pages skipped entirely. The fix costs nothing. It just means using the code that matches what actually happened on the server.

404 vs 410: when a page is gone for good

A 404 says a URL was not found right now. It leaves the door open, so crawlers tend to keep checking back for a while before they drop the page from the index. A 410 says the resource is gone and is not coming back. Google has said both eventually lead to the same outcome, but 410 tends to get acted on faster because it removes the ambiguity.

• Use 404 for a page that might come back, or one you have not decided about yet.
• Use 410 for a page you deleted on purpose and will never restore, such as a discontinued product or an expired event.
• Do not serve a 404 or 410 on a URL that still has content on it. That is a soft 404 and it confuses crawlers about which signal to trust.

503 vs 200: taking a page down without losing it

During a deploy or a maintenance window, it is tempting to just show a "back soon" message on the normal URL. If that page returns a 200, crawlers may index the maintenance message as the page's actual content, which can knock a ranking page out of the index for something it does not even talk about anymore.

• Return a 503 status during any planned downtime, not a 200.
• Add a Retry-After header with a realistic time so well-behaved crawlers know to come back rather than treat the page as gone.
• Keep 503 windows short. A 503 that lasts for days starts to read as a permanent outage instead of maintenance.

429: slowing crawlers down instead of blocking them

A lot of site owners reach straight for robots.txt disallow rules when an AI crawler is hitting the server too hard. That is a blunt tool: it also removes any chance of being cited by that engine. A 429 (too many requests) is a softer signal. It tells a crawler to back off and try again later, without shutting the door completely.

Most well-built crawlers, including the major AI bots, follow standard HTTP conventions and slow down when they see 429 responses, though behavior is not guaranteed for every crawler and is worth confirming in your own logs rather than assuming. If you are seeing load problems from a specific bot, try rate limiting with 429 before you block it outright.

A quick reference

• Page removed on purpose, never coming back: 410
• Page missing, might return, or you have not decided: 404
• Planned downtime or deploy in progress: 503 with Retry-After
• Crawler sending too many requests too fast: 429, not a robots.txt block
• Page moved permanently to a new URL: 301, covered in our redirects guide

Pull a sample of status codes straight from your server logs or a crawl tool once a quarter. It takes a few minutes and it is the fastest way to catch a maintenance page stuck at 200 or a graveyard of soft 404s before they cost you rankings or citations.