Special-case crawlers
Search indexing crawlerSpecialized search crawler
Identification
- User-Agent token
- —
- Published IP ranges
- 270
Requests claiming this User-Agent are verified against the operator's published ranges.
robots.txt behaviour
- Honours robots.txt
- Yes
- Honours crawl-delay
- No
- Honours Allow rules
- Yes
- Reads Sitemap directives
- Yes
Observed activity
- Distinct IPs observed
- 186
- TLS fingerprints observed
- 1
- First seen
- 2026-05-21
- Last seen
- 2026-08-09
Aggregated across traffic monitored by urlcap. Updated continuously.
Where it crawls from
Top countries (by distinct IPs)
US
186
Top networks
Unknown network
186
Verify Special-case crawlers on your own traffic
One API call checks a visitor's IP against published ranges, reverse DNS and TLS fingerprints — so spoofed User-Agents don't fool you.