Directory
Known bots
211 crawlers, fetchers and bots tracked by urlcap —
who they are, whether they honour robots.txt,
and how much of their activity we observe. Verified search-engine crawlers are marked;
their IP ranges are served by the
bot-detection API.
| Bot | UA token | Category | robots.txt | Published ranges | Observed IPs | Last seen |
|---|---|---|---|---|---|---|
| User-triggered fetchers (Google) search engine | — | User-initiated fetcher, Commercial platform crawler | Honoured | 504 | 1,722 | 2026-10-17 |
| Bingbot search engine | bingbot | Search indexing crawler | Honoured | 28 | 1,686 | 2026-10-17 |
| Googlebot search engine | Googlebot | Search indexing crawler | Honoured | 317 | 1,223 | 2026-10-17 |
| ChatGPT-User search engine | ChatGPT-User | AI search or answering bot, User-initiated fetcher | Honoured | 234 | 897 | 2026-10-16 |
| Yandex search engine | YandexBot | Search indexing crawler | Honoured | 16 | 667 | 2026-10-17 |
| OAI-SearchBot search engine | OAI-SearchBot | AI search or answering bot | Honoured | 39 | 544 | 2026-10-17 |
| Baiduspider search engine | Baiduspider | — | Honoured | 3 | 463 | 2026-10-17 |
| Claude-SearchBot search engine | Claude-SearchBot | — | Honoured | 38 | 378 | 2026-10-17 |
| Claude-User search engine | Claude-User | — | Honoured | 38 | 364 | 2026-10-17 |
| DuckAssistBot search engine | DuckAssistBot | Specialized search crawler, AI search or answering bot | Honoured | 486 | 105 | 2026-10-17 |
| PerplexityBot search engine | PerplexityBot | AI training crawler, AI search or answering bot | Honoured | 8 | 11 | 2026-10-17 |
| Perplexity-User search engine | Perplexity-User | AI search or answering bot, User-initiated fetcher | Honoured | 4 | 3 | 2026-10-02 |
| Applebot search engine | Applebot | Search indexing crawler, Commercial platform crawler | Honoured | 24 | — | 2026-05-28 |
| DuckDuckBot search engine | DuckDuckBot | Search indexing crawler | Honoured | 486 | — | — |
| Meta-ExternalFetcher search engine | Meta-ExternalFetcher | — | Honoured | — | — | — |
| MistralAI-User search engine | MistralAI-User | — | Honoured | — | — | — |
| AhrefsBot | AhrefsBot | SEO / marketing analytics crawler | Honoured | 81 | 871 | 2026-10-16 |
| Special-case crawlers | — | Search indexing crawler, Specialized search crawler | Honoured | 272 | 535 | 2026-10-17 |
| Amazonbot | Amazonbot | Search indexing crawler, Commercial platform crawler | Honoured | 1,292 | 432 | 2026-10-17 |
| User-triggered fetchers | — | User-initiated fetcher | Honoured | 1,068 | 375 | 2026-10-17 |
| Claude bot | ClaudeBot | AI training crawler, AI search or answering bot | Honoured | 38 | 336 | 2026-08-18 |
| GPTBot | GPTBot | AI training crawler | Honoured | 18 | 299 | 2026-10-17 |
| CCBot | CCBot | AI training crawler, Web dataset / archive crawler | Honoured | 5 | 23 | 2026-10-16 |
| DataForSeoBot | DataForSeoBot | — | Honoured | 7 | 16 | 2026-10-17 |
| ABEvalBot | ABEvalBot | — | Unknown | — | — | — |
| AI2Bot | AI2Bot | — | Unknown | — | — | — |
| AccordResearchBot | AccordResearchBot | — | Unknown | — | — | — |
| AffsignalCrawler | AffsignalCrawler | — | Unknown | — | — | — |
| AgentTrustBot | AgentTrustBot | — | Unknown | — | — | — |
| AionBot | AionBot | — | Unknown | — | — | — |
| AmazonAgentZBot | AmazonAgentZBot | — | Unknown | — | — | — |
| Amzn-SearchBot | Amzn-SearchBot | — | Unknown | — | — | — |
| ApcosysBot | ApcosysBot | — | Unknown | — | — | — |
| Aranet-SearchBot | Aranet-SearchBot | — | Unknown | — | — | — |
| ArtistLeadBot | ArtistLeadBot | — | Unknown | — | — | — |
| AstraBot | AstraBot | — | Unknown | — | — | — |
| AtlasBot | AtlasBot | — | Unknown | — | — | — |
| AwarioBot | AwarioBot | — | Unknown | — | — | — |
| AwarioSmartBot | AwarioSmartBot | — | Unknown | — | — | — |
| AzureAI-SearchBot | AzureAI-SearchBot | — | Unknown | — | — | — |
| BedrijvengidsBot | BedrijvengidsBot | — | Unknown | — | — | — |
| BidflowAdsTxtCrawler | BidflowAdsTxtCrawler | — | Unknown | — | — | — |
| BitSightBot | BitSightBot | — | Unknown | — | — | — |
| Bravebot | Bravebot | — | Unknown | — | — | — |
| BundlabsBot | BundlabsBot | — | Unknown | — | — | — |
| CaesarBot | CaesarBot | — | Unknown | — | — | — |
| ChatGLM-Spider | ChatGLM-Spider | — | Unknown | — | — | — |
| Cliqzbot | Cliqzbot | — | Unknown | — | — | — |
| CloudflareBrowserRenderingCrawler | CloudflareBrowserRenderingCrawler | — | Unknown | — | — | — |
| ClueWeb-Crawler | ClueWeb-Crawler | — | Unknown | — | — | — |
| CodeMMPretrainCrawler | CodeMMPretrainCrawler | — | Unknown | — | — | — |
| CompanyResearchBot | CompanyResearchBot | — | Unknown | — | — | — |
| CompassBot | CompassBot | — | Unknown | — | — | — |
| CookieYesbot | CookieYesbot | — | Unknown | — | — | — |
| CriteoBot | CriteoBot | — | Unknown | — | — | — |
| DNSCrawler | DNSCrawler | — | Unknown | — | — | — |
| DataPoisonerBot | DataPoisonerBot | — | Unknown | — | — | — |
| DeccanCorpusBot | DeccanCorpusBot | — | Unknown | — | — | — |
| DeepSeekBot | DeepSeekBot | — | Unknown | — | — | — |
| DeepcrawlBot | DeepcrawlBot | — | Unknown | — | — | — |
| Diffbot | Diffbot | — | Unknown | — | — | — |
| Discordbot | Discordbot | — | Unknown | — | — | — |
| DittaBot | DittaBot | — | Unknown | — | — | — |
| DomainStatsBot | DomainStatsBot | — | Unknown | — | — | — |
| DotBot | DotBot | — | Unknown | — | — | — |
| EasmBot | EasmBot | — | Unknown | — | — | — |
| ExaBot | ExaBot | — | Unknown | — | — | — |
| ExaSearchBot | ExaSearchBot | — | Unknown | — | — | — |
| FAST-WebCrawler | FAST-WebCrawler | — | Unknown | — | — | — |
| FacebookBot | FacebookBot | — | Unknown | — | — | — |
| FartcoinResearchBot | FartcoinResearchBot | — | Unknown | — | — | — |
| FindgBot | FindgBot | — | Unknown | — | — | — |
| FlowIQLabsBot | FlowIQLabsBot | — | Unknown | — | — | — |
| FokalsBot | FokalsBot | — | Unknown | — | — | — |
| FundraisingFoxBot | FundraisingFoxBot | — | Unknown | — | — | — |
| Gaisbot | Gaisbot | — | Unknown | — | — | — |
| GoldiBot | GoldiBot | — | Unknown | — | — | — |
| GrantOpsOutreachBot | GrantOpsOutreachBot | — | Unknown | — | — | — |
| GrokBot | GrokBot | — | Unknown | — | — | — |
| HiWolfsBot | HiWolfsBot | — | Unknown | — | — | — |
| HighstreetPresenceBot | HighstreetPresenceBot | — | Unknown | — | — | — |
| HoldingsCrawler | HoldingsCrawler | — | Unknown | — | — | — |
| HouseJesterBot | HouseJesterBot | — | Unknown | — | — | — |
| HubSeedsBot | HubSeedsBot | — | Unknown | — | — | — |
| HylandsBot | HylandsBot | — | Unknown | — | — | — |
| IMATA-Bot | IMATA-Bot | — | Unknown | — | — | — |
| ISSCyberRiskCrawler | ISSCyberRiskCrawler | — | Unknown | — | — | — |
| IbouBot | IbouBot | — | Unknown | — | — | — |
| ImageBot | ImageBot | — | Unknown | — | — | — |
| ImageMetaBot | ImageMetaBot | — | Unknown | — | — | — |
| InfrawatchCrawler | InfrawatchCrawler | — | Unknown | — | — | — |
| InsuranceMovementsBot | InsuranceMovementsBot | — | Unknown | — | — | — |
| JobResearchBot | JobResearchBot | — | Unknown | — | — | — |
| KairosBot | KairosBot | — | Unknown | — | — | — |
| KakaoBot | KakaoBot | — | Unknown | — | — | — |
| KeenableBot | KeenableBot | — | Unknown | — | — | — |
| Kimi-SearchBot | Kimi-SearchBot | — | Unknown | — | — | — |
| KimiBot | KimiBot | — | Unknown | — | — | — |
| KulenkovGroup-ProspectBot | KulenkovGroup-ProspectBot | — | Unknown | — | — | — |
| KvKResearchBot | KvKResearchBot | — | Unknown | — | — | — |
| LanternBot | LanternBot | — | Unknown | — | — | — |
| LeadEnrichmentBot | LeadEnrichmentBot | — | Unknown | — | — | — |
| LeadFinderBot | LeadFinderBot | — | Unknown | — | — | — |
| LeadSignalAudienceBot | LeadSignalAudienceBot | — | Unknown | — | — | — |
| LedgerlineBot | LedgerlineBot | — | Unknown | — | — | — |
| LinkedInBot | LinkedInBot | — | Unknown | — | — | — |
| LinkupBot | LinkupBot | — | Unknown | — | — | — |
| ListbuildBot | ListbuildBot | — | Unknown | — | — | — |
| LocalDomainCrawler | LocalDomainCrawler | — | Unknown | — | — | — |
| LonglistResearchBot | LonglistResearchBot | — | Unknown | — | — | — |
| LyonlBot | LyonlBot | — | Unknown | — | — | — |
| MaoxiaoWaterfallBot | MaoxiaoWaterfallBot | — | Unknown | — | — | — |
| Maps-bot | Maps-bot | — | Unknown | — | — | — |
| MathPicDatasetCrawler | MathPicDatasetCrawler | — | Unknown | — | — | — |
| MeasureboardResearchBot | MeasureboardResearchBot | — | Unknown | — | — | — |
| MidacreekBot | MidacreekBot | — | Unknown | — | — | — |
| MintyResearchBot | MintyResearchBot | — | Unknown | — | — | — |
| MistralBot | MistralBot | — | Unknown | — | — | — |
| MochiImageCrawler | MochiImageCrawler | — | Unknown | — | — | — |
| MojeekBot | MojeekBot | — | Unknown | — | — | — |
| MomentumFetcher | MomentumFetcher | — | Unknown | — | — | — |
| MoonshotBot | MoonshotBot | — | Unknown | — | — | — |
| NovablePipelineCrawler | NovablePipelineCrawler | — | Unknown | — | — | — |
| OAI-AdsBot | OAI-AdsBot | — | Unknown | — | — | — |
| OdklBot | OdklBot | — | Unknown | — | — | — |
| OngoingAICrawler | OngoingAICrawler | — | Unknown | — | — | — |
| Orbbot | Orbbot | — | Unknown | — | — | — |
| OutreachBetaBot | OutreachBetaBot | — | Unknown | — | — | — |
| PageSourceSearchBot | PageSourceSearchBot | — | Unknown | — | — | — |
| PanguBot | PanguBot | — | Unknown | — | — | — |
| Pinterestbot | Pinterestbot | — | Unknown | — | — | — |
| PoweredByBot | PoweredByBot | — | Unknown | — | — | — |
| PublicWWWBot | PublicWWWBot | — | Unknown | — | — | — |
| PurityBot | PurityBot | — | Unknown | — | — | — |
| QlyzeBot | QlyzeBot | — | Unknown | — | — | — |
| Querit-SearchBot | Querit-SearchBot | — | Unknown | — | — | — |
| QueritBot | QueritBot | — | Unknown | — | — | — |
| Qwantbot | Qwantbot | — | Unknown | — | — | — |
| Qwenbot | Qwenbot | — | Unknown | — | — | — |
| RecruetBot | RecruetBot | — | Unknown | — | — | — |
| Reflectionbot | Reflectionbot | — | Unknown | — | — | — |
| ResearchBot | ResearchBot | — | Unknown | — | — | — |
| RevicBot | RevicBot | — | Unknown | — | — | — |
| ReyilBot | ReyilBot | — | Unknown | — | — | — |
| S33DBot | S33DBot | — | Unknown | — | — | — |
| SEAPbot | SEAPbot | — | Unknown | — | — | — |
| SEOJuice-SearchBot | SEOJuice-SearchBot | — | Unknown | — | — | — |
| SERankingBacklinksBot | SERankingBacklinksBot | — | Unknown | — | — | — |
| SMTBot | SMTBot | — | Unknown | — | — | — |
| SPGlobalWebBot | SPGlobalWebBot | — | Unknown | — | — | — |
| SaaSBrowserBot | SaaSBrowserBot | — | Unknown | — | — | — |
| SaluteBot | SaluteBot | — | Unknown | — | — | — |
| ScoutJet | ScoutJet | Legacy / unwanted crawler | Unknown | — | — | — |
| SemanticVisionsBot | SemanticVisionsBot | — | Unknown | — | — | — |
| SemrushBot | SemrushBot | — | Unknown | — | — | — |
| SerusIconBot | SerusIconBot | — | Unknown | — | — | — |
| SeznamBot | SeznamBot | — | Unknown | — | — | — |
| ShapBot | ShapBot | — | Unknown | — | — | — |
| SiteAuditBot | SiteAuditBot | — | Unknown | — | — | — |
| SiteLensBot | SiteLensBot | — | Unknown | — | — | — |
| SiteRocketBot | SiteRocketBot | — | Unknown | — | — | — |
| SitemapCrawler | SitemapCrawler | — | Unknown | — | — | — |
| SleepBot | SleepBot | — | Unknown | — | — | — |
| SofyaBot | SofyaBot | — | Unknown | — | — | — |
| StartmeBot | StartmeBot | — | Unknown | — | — | — |
| StartupFundraisingBot | StartupFundraisingBot | — | Unknown | — | — | — |
| StewardComplianceBot | StewardComplianceBot | — | Unknown | — | — | — |
| SumvinUCPCrawler | SumvinUCPCrawler | — | Unknown | — | — | — |
| SuperBot | SuperBot | — | Unknown | — | — | — |
| SupplierDirectoryBot | SupplierDirectoryBot | — | Unknown | — | — | — |
| SurPhish-Reproduction-Crawler | SurPhish-Reproduction-Crawler | — | Unknown | — | — | — |
| SurdotlyBot | SurdotlyBot | — | Unknown | — | — | — |
| TelegramBot | TelegramBot | — | Unknown | — | — | — |
| Thinkbot | Thinkbot | — | Unknown | — | — | — |
| Timpibot | Timpibot | — | Unknown | — | — | — |
| Top1mBot | Top1mBot | — | Unknown | — | — | — |
| TrawlBot | TrawlBot | — | Unknown | — | — | — |
| Twitterbot | Twitterbot | User-initiated fetcher, Social preview fetcher | Unknown | — | — | — |
| UK-Archive-Crawler | UK-Archive-Crawler | — | Unknown | — | — | — |
| UmaiBot | UmaiBot | — | Unknown | — | — | — |
| UptimeRobot | UptimeRobot | — | Unknown | — | — | — |
| VelenPublicWebCrawler | VelenPublicWebCrawler | — | Unknown | — | — | — |
| WanScannerBot | WanScannerBot | — | Unknown | — | — | — |
| WebCrawlBot | WebCrawlBot | — | Unknown | — | — | — |
| WebCrawler | WebCrawler | — | Unknown | — | — | — |
| WebSearchCrawler | WebSearchCrawler | — | Unknown | — | — | — |
| WebSiphonBot | WebSiphonBot | — | Unknown | — | — | — |
| WebStatBot | WebStatBot | — | Unknown | — | — | — |
| WellKnownBot | WellKnownBot | — | Unknown | — | — | — |
| XtractBot | XtractBot | — | Unknown | — | — | — |
| YaDirectFetcher | YaDirectFetcher | — | Unknown | — | — | — |
| YandexAccessibilityBot | YandexAccessibilityBot | — | Unknown | — | — | — |
| YandexMobileBot | YandexMobileBot | — | Unknown | — | — | — |
| YandexRenderResourcesBot | YandexRenderResourcesBot | — | Unknown | — | — | — |
| YisouSpider | YisouSpider | — | Unknown | — | — | — |
| YouBot | YouBot | — | Unknown | — | — | — |
| agentcensus-crawler | agentcensus-crawler | — | Unknown | — | — | — |
| baananas-bot | baananas-bot | — | Unknown | — | — | — |
| bidswitchbot | bidswitchbot | — | Unknown | — | — | — |
| everyfeed-spider | everyfeed-spider | — | Unknown | — | — | — |
| hostio-bot | hostio-bot | — | Unknown | — | — | — |
| iaskspider | iaskspider | — | Unknown | — | — | — |
| kagi-fetcher | kagi-fetcher | — | Unknown | — | — | — |
| msnbot | msnbot | — | Unknown | — | — | — |
| osentix-crawler | osentix-crawler | — | Unknown | — | — | — |
| peer39_crawler | peer39_crawler | — | Unknown | — | — | — |
| research-bot | research-bot | — | Unknown | — | — | — |
| taste-research-crawler | taste-research-crawler | — | Unknown | — | — | — |
| tphotobot | tphotobot | — | Unknown | — | — | — |
| webinfo-bot | webinfo-bot | — | Unknown | — | — | — |
| xAI-SearchBot | xAI-SearchBot | — | Unknown | — | — | — |
"Published ranges" counts the IP ranges each operator publishes for verification; "Observed IPs" is distinct IPs urlcap has attributed to the bot across monitored traffic. Traffic-pattern groups (vulnerability scanners, unnamed scrapers) are tracked separately and not listed here.
Is that visitor really Googlebot?
User-Agent strings are trivially spoofed. urlcap verifies crawlers by published IP ranges and forward-confirmed reverse DNS — one API call, one honest answer.