Guides
How to solve real developer problems with urlcap — bot detection from server logs, TOTP automation
for CI, recurring HTTP checks without running cron, and more. Every article is built around working
curl
examples you can paste into a terminal.
Bot detection
A walk through verifying every crawler IP that hits your site: forward-confirmed reverse DNS, the CIDR ranges Google / Bing / OpenAI / Anthropic publish, and an Nginx allowlist that lets search engines through while blocking GPTBot, ClaudeBot and PerplexityBot.
Updated /api/v1/is_bot
Bot detection
Verify bot IPs from your Nginx, Apache or application logs against a daily-refreshed registry of known crawler CIDRs — Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot and more.
Updated /api/v1/is_bot
TOTP
Generate TOTP codes on demand from an otpauth:// URI so your CI, Playwright, or QA scripts can pass 2FA in staging — safely, without bundling secrets into frontend code.
Updated /api/v1/totp
Schedules
Run a capture or extract job on a cron schedule, store every run's result, and use it for uptime monitoring, compliance snapshots, or content-change detection — no cron worker to maintain.
Updated /api/v1/schedules
Drop us a note at info@urlcap.com and we'll consider it for the next round of guides.
