CCBot
AI training crawlerWeb dataset / archive crawler
Identification
- User-Agent token
- CCBot
- Published IP ranges
- 6
- Operator documentation
- https://commoncrawl.org/ccbot
Requests claiming this User-Agent are verified against the operator's published ranges.
robots.txt behaviour
- Honours robots.txt
- Yes
- Honours crawl-delay
- Yes
- Honours Allow rules
- Yes
- Reads Sitemap directives
- Yes
Observed activity
- Distinct IPs observed
- 22
- TLS fingerprints observed
- 2
- First seen
- 2026-06-06
- Last seen
- 2026-08-09
Aggregated across traffic monitored by urlcap. Updated continuously.
Where it crawls from
Top countries (by distinct IPs)
US
22
Top networks
Unknown network
22
Verify CCBot on your own traffic
One API call checks a visitor's IP against published ranges, reverse DNS and TLS fingerprints — so spoofed User-Agents don't fool you.