urlcap

API for developers

One developer API for HTTP, bots, DNS, and robots.txt.

Capture any HTTP request, extract structured data, identify known crawlers, look up DNS, watch robots.txt — over one versioned API. Try it below, no account needed.

5 free requests / 24 h · API docs · pricing

Try a capture

Send an HTTP request to any URL and inspect the full response — status, headers, cookies, query parameters and body — right here in the browser.

No key? You get 5 free trial requests / 24 h. Sign up for 1,000/month free.

Programmatic use: GET /api/v1/capture?url=… or POST /api/v1/capture with a JSON body — see the API reference.

Endpoints at a glance

Capture

Send any HTTP request to any URL. Inspect status, headers, cookies, redirects, query parameters, body, and timings — like your browser's network panel, as an API call.

/api/v1/capture

Extract

Run browser-style extraction jobs over a page. Pull structured data with CSS, XPath, or JSONPath selectors; paginate, follow links, and submit results to a callback.

/api/v1/extract

Bot detection

Check whether an IP belongs to Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, and other known crawlers — backed by a daily-refreshed registry of official CIDR ranges.

/api/v1/is_bot

Reverse DNS

PTR lookup for an IPv4 or IPv6 address, with optional forward-confirm (FCrDNS) to verify a self-claimed crawler. TTL-cached.

/api/v1/reverse_dns

Robots

Fetch and parse a site's robots.txt, check whether a URL is allowed for a user-agent, and watch a site for changes — with HMAC-signed webhook notifications on every change.

/api/v1/robots

Web Bot Auth

Verify a bot's RFC 9421 HTTP Message Signature against the operator's public key directory. The cryptographic "is this really Googlebot?" check — Ed25519, TTL-cached directory fetches.

/api/v1/web_bot_auth/verify

Schedules

Run a capture or an extract job once at a fixed time or on a cron schedule. Every run is stored and queryable, so you can diff results, build feeds, or alert on changes.

/api/v1/schedules

Datasets

Maintain your own collections of URLs or IP/CIDR ranges. Look up membership in O(1), keep full revision history, and re-feed them into is-bot or capture/extract pipelines.

/api/v1/datasets

TOTP

Generate the rotating 6-digit codes used by Google Authenticator, Authy and 1Password. Compute the current code from an otpauth:// URI — handy for automating 2FA in tests and scripts. Stateless: secrets aren't stored.

/api/v1/totp

Real-world guides

Concrete problems, working curl

All guides →