HTTP caching and 304 responses: how they help crawlers
Conditional requests let crawlers skip pages that haven't changed. Here is how ETag, Last-Modified, and 304 responses work, and how to check yours.
Every time a crawler fetches a page it already has, your server does work and the crawler downloads bytes it doesn't need. HTTP has a built-in fix: the conditional request. The crawler asks, "has this changed since I last saw it?" and your server can answer with a tiny 304 instead of the full page. Most site owners never check whether theirs does.
How conditional requests work
When your server sends a page, it can attach a validator. That is a small piece of data the client can send back later to prove which version it holds. There are two common validators.
- ETag: an opaque string that identifies one version of the response, often a hash of the content.
- Last-Modified: a timestamp for when the content last changed.
On the next visit, the client sends the validator back. An ETag comes back in an If-None-Match header. A timestamp comes back in an If-Modified-Since header. If nothing changed, the server replies with status 304 Not Modified and no body. The client reuses its saved copy.
If the page did change, the server just sends a normal 200 with the new content. Nothing breaks. The 304 is purely a shortcut.
Why this matters for SEO
Google has said in its crawling documentation that Googlebot supports conditional requests with ETag and Last-Modified. It has also said that many servers don't answer them correctly, so the savings often go unused. Treat that as a reason to test your own setup, not as a promise that fixing it will raise your rankings. It won't do that directly.
What it can do is lower the cost of being crawled. A 304 is cheap for your server and cheap for the crawler. On large sites with many stable pages, that frees capacity for new and changed URLs. On a small site, the effect is small, and you probably have bigger problems to fix first. If you are unsure whether crawl capacity applies to you, read the crawl budget guide before spending time here.
AI crawlers are less clear. Some vendors publish little about how their bots handle caching headers, and behavior may change. Sending correct validators costs you nothing and does no harm, but don't assume every AI bot will use them.
Common ways it goes wrong
Dynamic pages that never return 304
Many CMS and framework setups build the whole page on every request and send it with a 200, even when the client sent a valid validator. The page is identical, but the server never checks. Static files usually handle this well. HTML often doesn't.
ETags that change on every request
If your ETag includes something that varies, like a timestamp, a nonce, or a server-specific value behind a load balancer, it will never match. Each request looks like a new version. The fix is to base the ETag on the content, or drop it and rely on Last-Modified.
A Last-Modified date that lies
Some systems send the current time as Last-Modified on every response. That tells crawlers the page changes constantly, which can waste their time. Others send a date that never moves, even after real edits, so crawlers may keep a stale copy. Make the header match the last real content change.
A 304 that hides a real change
If your cache or CDN returns 304 based on the template and not the content, an edited page can look unchanged. A crawler would then keep the old version. This is rare but worth testing after any caching change.
How to check your site
You don't need special tools. Use curl or the network tab in your browser's developer tools.
- Request a page and note the ETag and Last-Modified headers in the response. If neither is there, your server isn't offering validators at all.
- Request the same page again, this time sending the validator back with If-None-Match or If-Modified-Since.
- Look at the status. A 304 with no body is the goal. A full 200 means your server ignores the validator.
- Repeat the test through your CDN and directly against your origin. They can behave differently.
- Edit the page, then repeat step two with the old validator. You should now get a 200 with the new content.
You can also see the effect in your server logs. The log file analysis guide shows how to read them. Look for 304 responses to Googlebot on pages that rarely change. If you see only 200s for the same URLs over and over, conditional requests aren't working.
A correct 304 saves a download. A wrong one can freeze a stale page in a crawler's memory.
What to do next
Run the five-step test on your five most-crawled templates: the home page, a category page, an article, a product page, and your sitemap. Fix any that send no validators or ignore them. Prefer a content-based ETag or an honest Last-Modified date. Then re-test after every CDN or caching change, since that is when these quietly break.