Directory
Known bots
127 crawlers, fetchers and bots tracked by urlcap —
who they are, whether they honour robots.txt,
and how much of their activity we observe. Verified search-engine crawlers are marked;
their IP ranges are served by the
bot-detection API.
| Bot | UA token | Category | robots.txt | Published ranges | Observed IPs | Last seen |
|---|---|---|---|---|---|---|
| Bingbot search engine | bingbot | Search indexing crawler | Honoured | 28 | 1,251 | 2026-08-09 |
| User-triggered fetchers (Google) search engine | — | User-initiated fetcher, Commercial platform crawler | Honoured | 494 | 909 | 2026-08-09 |
| Yandex search engine | YandexBot | Search indexing crawler | Honoured | 16 | 623 | 2026-08-09 |
| Googlebot search engine | Googlebot | Search indexing crawler | Honoured | 315 | 619 | 2026-08-09 |
| Baiduspider search engine | Baiduspider | — | Honoured | 3 | 463 | 2026-08-09 |
| Applebot search engine | Applebot | Search indexing crawler, Commercial platform crawler | Honoured | 12 | — | 2026-05-28 |
| DuckDuckBot search engine | DuckDuckBot | Search indexing crawler | Honoured | 481 | — | — |
| Amazonbot | Amazonbot | Search indexing crawler, Commercial platform crawler | Honoured | 524 | 432 | 2026-08-09 |
| Claude bot | ClaudeBot | AI training crawler, AI search or answering bot | Honoured | 20 | 336 | 2026-08-09 |
| ChatGPT-User | ChatGPT-User | AI search or answering bot, User-initiated fetcher | Honoured | 258 | 312 | 2026-08-09 |
| OAI-SearchBot | OAI-SearchBot | AI search or answering bot | Honoured | 35 | 270 | 2026-08-09 |
| AhrefsBot | AhrefsBot | SEO / marketing analytics crawler | Honoured | 51 | 188 | 2026-08-08 |
| Special-case crawlers | — | Search indexing crawler, Specialized search crawler | Honoured | 270 | 186 | 2026-08-09 |
| GPTBot | GPTBot | AI training crawler | Honoured | 21 | 175 | 2026-08-09 |
| DuckAssistBot | DuckAssistBot | Specialized search crawler, AI search or answering bot | Honoured | 481 | 90 | 2026-08-09 |
| User-triggered fetchers | — | User-initiated fetcher | Honoured | 1,056 | 96 | 2026-08-09 |
| CCBot | CCBot | AI training crawler, Web dataset / archive crawler | Honoured | 6 | 22 | 2026-08-09 |
| DataForSeoBot | DataForSeoBot | — | Honoured | 7 | 13 | 2026-08-09 |
| PerplexityBot | PerplexityBot | AI training crawler, AI search or answering bot | Ignored | 8 | 9 | 2026-08-09 |
| Perplexity-User | Perplexity-User | AI search or answering bot, User-initiated fetcher | Ignored | 4 | 1 | 2026-07-19 |
| ABEvalBot | ABEvalBot | — | Unknown | — | — | — |
| AI2Bot | AI2Bot | — | Unknown | — | — | — |
| AccordResearchBot | AccordResearchBot | — | Unknown | — | — | — |
| AffsignalCrawler | AffsignalCrawler | — | Unknown | — | — | — |
| Amzn-SearchBot | Amzn-SearchBot | — | Unknown | — | — | — |
| AtlasBot | AtlasBot | — | Unknown | — | — | — |
| AwarioBot | AwarioBot | — | Unknown | — | — | — |
| AwarioSmartBot | AwarioSmartBot | — | Unknown | — | — | — |
| AzureAI-SearchBot | AzureAI-SearchBot | — | Unknown | — | — | — |
| BitSightBot | BitSightBot | — | Unknown | — | — | — |
| Bravebot | Bravebot | — | Unknown | — | — | — |
| BundlabsBot | BundlabsBot | — | Unknown | — | — | — |
| CaesarBot | CaesarBot | — | Unknown | — | — | — |
| Claude-SearchBot | Claude-SearchBot | — | Unknown | — | — | — |
| Cliqzbot | Cliqzbot | — | Unknown | — | — | — |
| ClueWeb-Crawler | ClueWeb-Crawler | — | Unknown | — | — | — |
| CriteoBot | CriteoBot | — | Unknown | — | — | — |
| DNSCrawler | DNSCrawler | — | Unknown | — | — | — |
| DataPoisonerBot | DataPoisonerBot | — | Unknown | — | — | — |
| DeepSeekBot | DeepSeekBot | — | Unknown | — | — | — |
| DeepcrawlBot | DeepcrawlBot | — | Unknown | — | — | — |
| Diffbot | Diffbot | — | Unknown | — | — | — |
| DomainStatsBot | DomainStatsBot | — | Unknown | — | — | — |
| DotBot | DotBot | — | Unknown | — | — | — |
| EasmBot | EasmBot | — | Unknown | — | — | — |
| ExaBot | ExaBot | — | Unknown | — | — | — |
| ExaSearchBot | ExaSearchBot | — | Unknown | — | — | — |
| FAST-WebCrawler | FAST-WebCrawler | — | Unknown | — | — | — |
| FlowIQLabsBot | FlowIQLabsBot | — | Unknown | — | — | — |
| Gaisbot | Gaisbot | — | Unknown | — | — | — |
| GrantOpsOutreachBot | GrantOpsOutreachBot | — | Unknown | — | — | — |
| HouseJesterBot | HouseJesterBot | — | Unknown | — | — | — |
| HubSeedsBot | HubSeedsBot | — | Unknown | — | — | — |
| IMATA-Bot | IMATA-Bot | — | Unknown | — | — | — |
| ISSCyberRiskCrawler | ISSCyberRiskCrawler | — | Unknown | — | — | — |
| IbouBot | IbouBot | — | Unknown | — | — | — |
| ImageBot | ImageBot | — | Unknown | — | — | — |
| ImageMetaBot | ImageMetaBot | — | Unknown | — | — | — |
| JobResearchBot | JobResearchBot | — | Unknown | — | — | — |
| KairosBot | KairosBot | — | Unknown | — | — | — |
| KulenkovGroup-ProspectBot | KulenkovGroup-ProspectBot | — | Unknown | — | — | — |
| KvKResearchBot | KvKResearchBot | — | Unknown | — | — | — |
| LeadFinderBot | LeadFinderBot | — | Unknown | — | — | — |
| LeadSignalAudienceBot | LeadSignalAudienceBot | — | Unknown | — | — | — |
| LinkedInBot | LinkedInBot | — | Unknown | — | — | — |
| LinkupBot | LinkupBot | — | Unknown | — | — | — |
| ListbuildBot | ListbuildBot | — | Unknown | — | — | — |
| LocalDomainCrawler | LocalDomainCrawler | — | Unknown | — | — | — |
| LyonlBot | LyonlBot | — | Unknown | — | — | — |
| MeasureboardResearchBot | MeasureboardResearchBot | — | Unknown | — | — | — |
| Meta-ExternalFetcher | Meta-ExternalFetcher | — | Unknown | — | — | — |
| MistralBot | MistralBot | — | Unknown | — | — | — |
| MojeekBot | MojeekBot | — | Unknown | — | — | — |
| MomentumFetcher | MomentumFetcher | — | Unknown | — | — | — |
| OdklBot | OdklBot | — | Unknown | — | — | — |
| Orbbot | Orbbot | — | Unknown | — | — | — |
| Pinterestbot | Pinterestbot | — | Unknown | — | — | — |
| PurityBot | PurityBot | — | Unknown | — | — | — |
| Querit-SearchBot | Querit-SearchBot | — | Unknown | — | — | — |
| QueritBot | QueritBot | — | Unknown | — | — | — |
| Qwantbot | Qwantbot | — | Unknown | — | — | — |
| ReyilBot | ReyilBot | — | Unknown | — | — | — |
| S33DBot | S33DBot | — | Unknown | — | — | — |
| SEAPbot | SEAPbot | — | Unknown | — | — | — |
| SEOJuice-SearchBot | SEOJuice-SearchBot | — | Unknown | — | — | — |
| SERankingBacklinksBot | SERankingBacklinksBot | — | Unknown | — | — | — |
| SPGlobalWebBot | SPGlobalWebBot | — | Unknown | — | — | — |
| SaaSBrowserBot | SaaSBrowserBot | — | Unknown | — | — | — |
| ScoutJet | ScoutJet | Legacy / unwanted crawler | Unknown | — | — | — |
| SemanticVisionsBot | SemanticVisionsBot | — | Unknown | — | — | — |
| SemrushBot | SemrushBot | — | Unknown | — | — | — |
| ShapBot | ShapBot | — | Unknown | — | — | — |
| SiteAuditBot | SiteAuditBot | — | Unknown | — | — | — |
| SiteLensBot | SiteLensBot | — | Unknown | — | — | — |
| SitemapCrawler | SitemapCrawler | — | Unknown | — | — | — |
| SleepBot | SleepBot | — | Unknown | — | — | — |
| StartmeBot | StartmeBot | — | Unknown | — | — | — |
| StartupFundraisingBot | StartupFundraisingBot | — | Unknown | — | — | — |
| SumvinUCPCrawler | SumvinUCPCrawler | — | Unknown | — | — | — |
| SuperBot | SuperBot | — | Unknown | — | — | — |
| SurPhish-Reproduction-Crawler | SurPhish-Reproduction-Crawler | — | Unknown | — | — | — |
| SurdotlyBot | SurdotlyBot | — | Unknown | — | — | — |
| Thinkbot | Thinkbot | — | Unknown | — | — | — |
| Timpibot | Timpibot | — | Unknown | — | — | — |
| Twitterbot | Twitterbot | User-initiated fetcher, Social preview fetcher | Unknown | — | — | — |
| UK-Archive-Crawler | UK-Archive-Crawler | — | Unknown | — | — | — |
| UmaiBot | UmaiBot | — | Unknown | — | — | — |
| UptimeRobot | UptimeRobot | — | Unknown | — | — | — |
| VelenPublicWebCrawler | VelenPublicWebCrawler | — | Unknown | — | — | — |
| WebCrawlBot | WebCrawlBot | — | Unknown | — | — | — |
| WebCrawler | WebCrawler | — | Unknown | — | — | — |
| WellKnownBot | WellKnownBot | — | Unknown | — | — | — |
| XtractBot | XtractBot | — | Unknown | — | — | — |
| YaDirectFetcher | YaDirectFetcher | — | Unknown | — | — | — |
| YandexAccessibilityBot | YandexAccessibilityBot | — | Unknown | — | — | — |
| YandexMobileBot | YandexMobileBot | — | Unknown | — | — | — |
| YandexRenderResourcesBot | YandexRenderResourcesBot | — | Unknown | — | — | — |
| YisouSpider | YisouSpider | — | Unknown | — | — | — |
| YouBot | YouBot | — | Unknown | — | — | — |
| bidswitchbot | bidswitchbot | — | Unknown | — | — | — |
| everyfeed-spider | everyfeed-spider | — | Unknown | — | — | — |
| iaskspider | iaskspider | — | Unknown | — | — | — |
| kagi-fetcher | kagi-fetcher | — | Unknown | — | — | — |
| msnbot | msnbot | — | Unknown | — | — | — |
| peer39_crawler | peer39_crawler | — | Unknown | — | — | — |
| webinfo-bot | webinfo-bot | — | Unknown | — | — | — |
| xAI-SearchBot | xAI-SearchBot | — | Unknown | — | — | — |
"Published ranges" counts the IP ranges each operator publishes for verification; "Observed IPs" is distinct IPs urlcap has attributed to the bot across monitored traffic. Traffic-pattern groups (vulnerability scanners, unnamed scrapers) are tracked separately and not listed here.
Is that visitor really Googlebot?
User-Agent strings are trivially spoofed. urlcap verifies crawlers by published IP ranges and forward-confirmed reverse DNS — one API call, one honest answer.