Crawl Stats report: what Google says about your server
The Crawl Stats report in Search Console shows how Googlebot treats your server. Here is how to read it and which patterns point to real problems.
Most people open Search Console to check clicks and rankings. Few open the Crawl Stats report. That is a shame, because it is the one place Google tells you how it is treating your server. If pages are slow to get indexed, this report often shows why.
What the Crawl Stats report shows
You find it under Settings, then Crawl stats. It covers roughly the last 90 days of Googlebot requests to your site. It only counts Google's own crawlers, so it will not show GPTBot or other AI crawlers. For those you need server logs.
The report gives you three headline charts:
- Total crawl requests: how many URLs Google fetched each day.
- Total download size: how many bytes it pulled.
- Average response time: how long your server took to answer.
Below the charts are breakdowns by response code, file type, purpose, and Googlebot type. These are where the useful signals are.
How to read the breakdowns
By response
Look at the share of 200 responses first. A healthy site is mostly 200s, with some 301s and a few 404s. A large share of 3xx means Google is spending requests on redirects. A rising count of 5xx or 429 means your server is struggling, and Google will slow down to be polite.
By purpose
Google labels each request as discovery (a URL it had not seen) or refresh (a recrawl of a known URL). A new site should show mostly discovery. A mature site should show mostly refresh. If you publish daily and see almost no discovery, Google may not be finding your new pages. Check your internal links and your sitemap.
By file type
This shows how much crawl goes to HTML versus JavaScript, CSS, images, and other files. If a big slice goes to files that never matter for search, such as old API responses or huge scripts, you have something to trim.
By Googlebot type
You will see smartphone, desktop, image, and others. Since Google indexes mobile-first, smartphone should dominate. If desktop leads, check that your mobile pages are reachable and not blocked.
Host status: the part people miss
The Host status section reports whether Google could fetch your robots.txt, resolve your DNS, and connect to your server. Each shows a pass or a problem over the last 90 days. A failed robots.txt fetch matters most. If Google cannot read robots.txt, it may pause crawling the whole site until it can. A firewall rule that blocks Googlebot can cause this while your pages still load fine for you.
Patterns worth acting on
- Response time climbing while crawl requests fall. Google is backing off because your server is slow. Look at hosting, caching, and heavy database queries.
- A spike in 404s. Often a deploy broke internal links or a migration missed redirects.
- Lots of requests to parameter URLs. Faceted navigation or tracking parameters are eating requests. Use canonical tags, robots.txt rules, or cleaner links.
- Flat discovery after a big content push. New pages are not being linked or listed in the sitemap.
- Failed host status checks. Test with the URL Inspection tool and check your firewall.
Click any breakdown row to see example URLs. Those samples turn a vague trend into a concrete list you can fix.
Limits to keep in mind
The report is sampled and aggregated. It will not list every request, and it only reports for the root of a domain property or a root-level URL prefix property. If you only verified a subfolder, you will not see it. It also says nothing about why Google chose not to index a page. For that, use the Pages report.
Also, crawl volume is not a quality score. More crawling does not mean better rankings. Crawl rate mostly follows server health and how often your content changes. For most small sites, the totals matter less than the trends.
A ten-minute monthly routine
- Open Crawl stats and check that response time is flat or falling.
- Scan the response breakdown for new 404, 5xx, or 429 spikes.
- Check that the host status shows no recent failures.
- Compare discovery and refresh against what you published.
- Open example URLs for anything odd and fix the cause.
Do this once a month and you will catch server and crawl problems weeks before they show up as lost traffic. Start today: open the report, note your current average response time, and use it as your baseline.