urlcap

Changelog

What's new in the urlcap API. Breaking changes ship under a new version path; everything additive lands within the current version. Subscribe to updates by emailing info@urlcap.com.

1. 2026-08-24 v1 · new

   Invocation parameters — run a template by id with different inputs

   A run-now can now carry a variables object — {"action":"run-now","variables":{"country":"US"}} — that merges over the template's stored variables seed for that one invocation. So the same saved template, invoked by id, can run with different entry parameters each time — driving {{...}} interpolation and conditional-action guards. The stored template is untouched. See the docs.

2. 2026-08-26 v1 · new

   Seed cookies on an extract job

   Extract jobs accept a cookies map that's seeded into the jar before the first request (scoped to the start URL's host), sent on every same-host request after. For sites that gate the session on a consent cookie their JavaScript would set — which a headless run never clicks through — just seed it, e.g. "cookies": {"CHG-cookie-accepted": "true"}. See the docs.

3. 2026-08-26 v1 · new

   Conditional actions — run a step only when a variable matches

   Any extract action can now carry a when guard so it runs only if a condition holds against a variable — an entry parameter or a value captured mid-flow. Checks: existence (exists), exact value (equals/notEquals/in), and numeric range (min/max). Skipped actions show in the trace as "skipped — condition not met". See the docs.

4. 2026-08-24 v1 · new

   Playground: view the page source after each step

   The extract playground now captures the page source after every step; each line in the execution list gets a view HTML link showing the exact DOM at that point — so you can see precisely what the page looked like when a click matched nothing. On the API it's the new "snapshots": true flag (each result.trace step gains a snapshot index); bounded to 30 per run, 2 MB each, served as text/plain. See the docs.

5. 2026-08-24 v1 · new

   Edit schedules in the playground + schedule update endpoint

   Extract schedules now have an Edit in playground link that opens the playground pre-loaded with their spec — tweak it, run it to watch the trace, then Save to schedule to update it in place or Save as new template to spin off a fresh on-demand job. Backed by a new PUT /api/v1/schedules/{id} that replaces a schedule's task spec (and optionally its name) while keeping its id, status, schedule and run history. See the docs.

6. 2026-08-24 v1 · new

   Extract playground — build a job and watch it run live

   A new playground in your account lets you paste an extract job model, run it on the default engine, and watch each step stream live — the initial load, every action's outcome (matched element / no match, click navigation, fill applied, {{secret.*}} values shown as «secret»), then the extractor results. It uses your account's secrets, proxies and certificates, so it mirrors a real run — ideal for dialing in selectors (e.g. a click on a button matched by text) before you save the job. The same trace is available on the API with "trace": true (streamed in progress.trace, echoed in result.trace). See the docs.

7. 2026-08-24 v1 · new

   On-demand schedule templates — save a task, invoke it by id

   Create a schedule with "active": false and no cron/runAt to get a saved, named task that never fires on its own but runs whenever you trigger it by id with POST /api/v1/schedules/{id} {"action":"run-now"} — it runs once and stays inactive. On-demand extract templates now resolve proxy pools, {{secret.*}} placeholders and the clientCertificate selector at each invocation (account-scoped, same as a direct /extract call), so templated logins with stored credentials work. New activate/deactivate actions toggle an existing schedule between live and on-demand. See the docs.

8. 2026-08-24 v1 · new

   Account secrets — login credentials out of job specs

   Store login passwords and tokens once (POST /api/v1/secrets, up to 200 per account) and reference them from extract jobs as {{secret.name}} in any action value, URL, selector or url_template. The target site receives the real credential at run time, but the submitted job, the stored task record, the result.variables echo and the recorded outbound request bodies carry only the placeholder or a «secret» marker. Values are AES-256-GCM-encrypted at rest and never returned by any endpoint after creation; rotate with "overwrite": true. See the docs.

9. 2026-08-24 v1 · new

   Client certificates (mTLS) on /capture and /extract

   Store up to 20 client certificates per account (POST /api/v1/client_certificates — a PEM pair or a PKCS#12 export) and present one during the TLS handshake of any capture or extract job with "clientCertificate": {"alias": "..."} — for partner APIs, corporate gateways and other mutual-TLS targets. Keys are verified against the leaf certificate at upload, stored AES-256-GCM-encrypted, and never returned by any endpoint; the list API exposes metadata only (subject, fingerprint, expiry). Extract jobs present the certificate on every HTTPS request they make. Requires an https target; not yet supported with the real-browser engine. Verified end-to-end against an mTLS reference host: 400 without the certificate, 200 with it, on both engines. See the docs.

10. 2026-08-24 v1 · new

    /extract: mid-flow extraction into a per-job variable store

    A new extract action type captures values at any point in the action sequence — from the DOM, a response header (from: "header:Name"), the cookie jar (from: "cookie:name"), or the current URL (from: "url", optionally with a regex) — and stores them in a variable store scoped to that one job. Later actions reuse them via {{name}} placeholders in value / url / selector — grab a CSRF token before a form, fill it back, build the next URL from an id on the page. Values land in a single per-job namespace by default (rewrites overwrite); naming the action stores them as name.field to record where they were captured. Jobs can seed initial values with a top-level variables object, URL substitutions are URL-encoded automatically (|raw opts out), and the store’s final state is echoed as result.variables for chaining into a follow-up job. List variables can also drive visit_each pagination: url_template + items_var build one detail URL per list element ({{item}} / {{item.field}}) instead of harvesting links off the page — extract the ids mid-flow, then walk the detail pages built from them, in one job. See the docs.

11. 2026-08-23 v1 · improved

    known-bot-ranges: one row per CIDR, unambiguous verified-engine flag

    The known-bot-ranges feed now emits exactly one row per distinct CIDR. Some operators publish a single egress range for several of their bots — a range could previously arrive once per owning bot, with definitive_search_engine reflecting each owner separately, leaving the merge (and the verified-engine verdict) up to the consumer. Each row now carries the OR of definitive_search_engine and honours_robots across owners, bot_group names a verified engine when one owns the range, and a new bot_groups array lists every owner ({id, name}). If your integration allow-lists verified engines from this feed, no change is needed — the flag is now simply correct on shared ranges.

12. 2026-08-15 v1 · new

    Configurable per-task socket timeout for /extract

    Extract jobs accept an optional socketTimeoutMs that raises the per-request connect/read timeout from the 30 s default up to a 120 s ceiling. This keeps a slow upstream proxy — e.g. an unlocker-style endpoint that can take 20–60 s to return a single page — from false-timing-out mid-crawl, while still capping any one stuck socket so it can’t pin a worker indefinitely. Unset or 0 preserves the previous 30 s behaviour.

    Verified end-to-end: an extract routed through an account proxy with socketTimeoutMs: 90000 completed a real page fetch and returned parsed content with no premature timeout.

13. 2026-07-30 v1 · fix

    Outcome timestamps stored in true UTC

    Outcomes posted to /api/v1/ingest/{site}/outcomes were being stored with ts one hour ahead during British Summer Time (a server-side timezone bug in the insert path — client-supplied timestamps were never at fault). Fixed; new outcomes are stored in true UTC. Historical rows from BST periods retain the +1h offset, and every row's arrival time was always recorded correctly alongside.

14. 2026-07-29 v1 · improved

    Human-outcome grace — logged-in customers stop looking like scrapers

    A heavy repeat visitor with a fully warm browser cache issues zero asset requests — at the log layer that's indistinguishable from a "high volume, no assets" scraper. Bot discovery now cross-references the outcomes channel: an IP that recently completed a purchase (purchase/completed), authenticated (auth/authenticated or signup), or solved a real captcha is exempt from scraper attribution and captcha-candidate flagging for a 72-hour window.

    Guard rails: grace stops above a request-volume ceiling (default 3,000 reqs/hour — one past purchase on a NAT egress can't shield a scraper behind the same IP), and a high suspicious-validation rate keeps the flag regardless (captcha farms solve captchas). Ship auth and purchase outcomes with the request_id of the originating request to opt your real users into the exemption.

15. 2026-07-29 site · new

    Known-bot directory — /bots

    A public directory of the crawlers, fetchers and bots urlcap tracks: urlcap.com/bots. Each bot has a profile page — User-Agent token, category, whether it honours robots.txt (and crawl-delay / Allow / Sitemap), verified-search-engine status, published IP-range counts, and aggregated observed activity (distinct IPs, TLS fingerprints, top countries and networks). The underlying verification data powers /is_bot and the known-bot-ranges feed.

16. 2026-07-29 site · new

    Live User-Agent statistics — /user-agents

    urlcap.com/user-agents shows the most-used User-Agent strings observed across urlcap (2.8M+ distinct UAs), all-time and trailing-7-day, with breakdowns by device class, browser, operating system and top robots. Parsed with Yauaa, recomputed hourly from a snapshot so the page is always instant.

17. 2026-07-29 site

    Light theme by default + a new typeface

    The site now defaults to the light theme everywhere; dark mode remains one click away on the topbar toggle and is remembered per browser. The whole site also moved to Courier Prime — a screen-optimised take on the classic typewriter face, self-hosted, with no third-party font requests.

18. 2026-05-28 v1 · new

    Candidate-IP feed — /sites/{id}/bot-ip-candidates

    IP sibling of the JA4-side /bot-ja4s feed: a small list of IPs exhibiting abusive behaviour on your site, ranked by a 5-component composite (block-ratio 40%, volume 20%, path breadth 15%, JA4 churn 15%, vuln-probe hits 10%). Already-classified IPs (in bot_observed_ips) and trust-list IPs are excluded from scoring entirely. Default min_score=0.50 score floor keeps the list small by construction.

    format flag supports json (default operator shape), txt (one IP per line) and cidr (each IP as /32 or /128) — the text formats are text/plain so you can feed straight into ipset, iptables, or a Cloudflare IP list.

19. 2026-05-28 internal

    Bot-candidate auto-promote, auto-dismiss, de-promote & reopen

    The candidate queue at /account/admin/bot-candidates is now self-maintaining. A daily 04:30 UTC job auto-promotes obvious bots that pass eight conservative gates (score ≥ 0.85, reqs ≥ 10k, distinct_ips ≥ 100, sites_seen ≥ 5, asset_ratio < 0.01, purchases = 0, age ≥ 48h, status = pending) into a single umbrella bot_group Auto-flagged scraper (discovery), and auto-dismisses stale rows (last_seen > 14 days ago + score < 0.7) to keep the queue bounded.

    The admin UI gains a header banner with live queue counts (pending / promoted-auto-vs-manual / dismissed) and two new per-row actions: De-promote (deletes the corresponding bot_observed_ja4s row, garbage-collects an orphaned per-candidate bot_group if any, sends the candidate back to pending) and Reopen (dismissed → pending for re-review).

    Both automated paths are independently switchable via system_config: bot.discovery.auto_promote.enabled (default false) and bot.discovery.auto_dismiss.enabled (default true). Audit trail on each promoted candidate (promotion_kind, promoted_to_bot_group_id, promoted_at) and on bot_groups created by manual promotion (discovery_kind = 'manual-from-candidate' or 'auto-umbrella').

20. 2026-05-27 v1 · new

    Site-level quick-check endpoints — /url-stats & /bot-traffic

    Two new helper endpoints on the sites surface that bottle up the most common investigation queries:

    • GET /api/v1/sites/{id}/url-stats?host=&path=&days=N — totals + status breakdown + per-day chart + a 25-row non-2xx sample for a specific URL. Answers "any rejections on URL X today?" without writing ClickHouse by hand.
    • GET /api/v1/sites/{id}/bot-traffic?bot_group=&days=N — recent traffic from a named bot_group (substring-match), with status mix and a sample of recent requests. Answers "are we blocking Google?" in one call. Bot identification matches every JA4 the discovery system has attributed to the bot_group via bot_observed_ja4s.
21. 2026-05-26 v1 · new

    Phase-level timings on /capture and monitor checks

    /api/v1/capture's response.timings object now carries a phase breakdown: dnsMs, connectMs (TCP + TLS), requestSendMs, ttfbMs, bodyMs, plus the resolvedIp the socket actually landed on. Phase fields are omitted when their lifecycle hook didn't fire — pooled keep-alive reuse skips DNS / connect, and followRedirects=true only captures the first leg's timings.

    Capture-kind URL monitors persist the same fields per check on monitor_checks and surface them on /monitors/{id}/checks + the monitor detail page. Extract-kind monitors don't get phase timings — HtmlUnit's connection layer doesn't expose them. While we were there: fixed the persisted check size_bytes column reading 0 on capture monitors (it was looking at sizeBytes in the response envelope, which is actually bodyBytes).

22. 2026-05-26 v1 · new

    URL monitors — up/down checks with webhook + email alerts

    New /api/v1/monitors (plus /checks, /uptime, /pause, /resume) let you schedule a recurring capture or extract against any URL and get notified on state transitions. Same primitives as UptimeRobot — plus full headless-browser checks, JSON-API validation via the extract path, and the User-Agent personas from /user_agent_profiles.

    Sub-minute polling on paid plans (30 s minimum on Startup + Business, 60 s on Developer, 300 s on Free). State transitions fire HMAC-SHA256-signed webhooks AND email through the platform mailer. Flapping is debounced via alertFailureThreshold — consecutive failures required before flipping to down, so single-shot 502s don't page anyone. Check history retained 30 days; sweep runs daily.

    Manage from the UI at /account/monitors — create, edit, view recent checks, see uptime over the last 30 days.

23. 2026-05-26 v1 · new

    User-Agent profiles for capture & extract

    New GET /api/v1/user_agent_profiles plus two new fields on capture + extract requests — userAgent (raw string override) and userAgentProfile (key into the catalogue). The catalogue is an operator-managed MySQL table, seeded with the common browser personas (chrome-latest-mac, firefox-latest-win, edge-latest-win, safari-latest-mac, …) plus googlebot and the default urlcap.

    On /extract the profile selects both the UA string AND the HtmlUnit BrowserVersion (Chrome / Firefox / Edge) so the in-page navigator.userAgent and the JS engine flavour line up. safari-* profiles still run on Chromium (HtmlUnit has no Safari engine) — navigator.vendor will read Google Inc.. googlebot only sets the UA string; for cryptographic urlcap attribution use webBotAuth=true.

24. 2026-05-23 v1 · new

    Per-minute traffic-shape signals on the bot-JA4 blocklist

    /api/v1/sites/{id}/bot-ja4s now returns reqs_per_minute_peak, reqs_per_minute_mean (over active minutes only), active_minutes and burstiness (coefficient of variation) per JA4. Useful for catching attack-shape JA4s that hide under a moderate asset_ratio but spike to thousands of req/min during their active windows — the case where score >= 0.70 and asset_ratio sits between 0.05 and 0.50 so neither Tier 1 nor Tier 2 fires.

25. 2026-05-22 v1 · new

    Bot-JA4 blocklist endpoint — one feed, edge-blocklist ready

    New GET /api/v1/sites/{id}/bot-ja4s: returns every JA4 the discovery system has flagged on a site over a sliding window (1..90 days), labelled known_bot with bot-group attribution or candidate with discovery score. Browser-shaped fingerprints are excluded; the strings are the keys you feed into nginx, Cloudflare WAF, or any edge map. Each row also carries cross_customer_action — how many other sites have explicitly blocked, allowed, or challenged this JA4 via edge_action outcomes — plus block_likely_on_this_site inferred from your own status-code mix.

26. 2026-05-22 v1 · new

    Single-IP intelligence endpoint

    GET /api/v1/ip/intelligence?ip=… composes every signal urlcap has on a single IP into one response — MaxMind geo, PTR records, bot-CIDR membership, every bot_observed_ips attribution row (cidr_match / ua_match / vuln_match / manual), trust-list status, and cross-site behavioural data over the last 1–7 days: request count, distinct JA4s/UAs/sites, status-code mix, asset ratio, vuln-probe hits, every JA4 with classification, top user agents, and cross-customer edge_action votes summed across every JA4 the IP has used. The operator-grade investigation endpoint.

27. 2026-05-22 v1 · quality

    Base62 site public ids + per-domain kind

    Every site now carries a 12-char base62 publicId alongside its numeric id. Use it wherever a site appears in a URL — /api/v1/sites/2DrxGfsYW0jv/bot-ja4s, /api/v1/ingest/2DrxGfsYW0jv/events — to avoid leaking enumeration order to anyone holding a URL. The numeric form remains accepted. Domains now carry a kind hint (doc / asset / api); informational only for v1, but expect future signals to lean on it (assets-vs-docs ratio per host, etc.).

28. 2026-05-21 v1 · new

    Outcomes channel: tell urlcap what happened next

    New POST /api/v1/ingest/{site}/outcomes accepts asynchronous verdicts keyed by the original event's request_id — kinds js_challenge / auth / purchase / edge_action. urlcap resolves the JA4 server-side at ingest time and denormalises it onto the outcome row so verdicts survive the 7-day raw-event TTL. Eight new outcome-derived signals surface on /api/v1/ja4/intelligence (js_challenge_attempts / passes / pass_rate, auth_observations, distinct_users, purchases, total_purchase_cents, last_purchase_at) plus ja4_ua_consistency — UA-spoofing tell where a JA4 keeps claiming different agent/OS values.

    Plain-HTTP traffic (no TLS handshake, no JA4) now also flows through ingest — IP-keyed aggregates and vuln-probe / bot-discovery still see those rows; JA4-keyed materialised views filter them out.

29. 2026-05-21 v1 · new

    Vuln-probe attribution + trust list

    Every ingested request is tagged at write-time with a vuln_probe_id if its path matches the curated vuln-scan registry (52 paths across 9 categories — exposed WP plugins, exposed .git, common dropper paths, etc.). IPs hitting three or more distinct probes are auto-attributed to the synthetic "Vulnerability scanner" bot_group; the discovery loop currently catches ~50–70 new scanner IPs per hour.

    New per-user trust list of IPs / CIDRs / domains the discovery scan never flags — protect monitoring probes, office IPs, and partner integrations from being auto-classified as scanners. Trust applies globally (signal modulation) while ownership stays per-user (admin audit).

30. 2026-05-20 v1 · new

    JA4 intelligence: OS / agent / device classification + 25-metric snapshot

    Three new read endpoints on top of the long-window aggregates:

    • /api/v1/ja4/intelligence — the rolled-up profile for one JA4 over 7 / 30 / 90 days. Returns the top OS / agent / device guess with per-dimension confidence (the score IS the confidence — apply your own threshold), diversity scores, and the new outcome-derived counts. Hourly recompute.
    • /api/v1/ja4/profile — drill into one dimension (os_name, agent_name, device_class, country, asn and 4 more), top-N values with HLL-merged distinct counts.
    • /api/v1/ja4/metrics — on-demand "prioritised 25" snapshot: trailing-hour rollup (counts + 10 ratios), optional IP+JA4 sub-block when ip= is supplied (with is_new_24h), and a top-10 JA4×UA breakdown over 24h with each UA's share of the JA4's volume.
    • /api/v1/ip/profile — per-IP behavioural summary on one site over N days (distinct JA4s/UAs/paths/hosts + heuristic / challenge counts). The proxy / NAT detection lookup.
31. 2026-05-20 v1 · new

    Bot discovery: auto-classify what your traffic actually contains

    A background loop now ingests every request, attributes IPs and JA4s to known bot groups (CIDR match, UA self-identification, vuln-probe pattern), and queues unknown candidates for review. UA-self-identifying bots are auto-classified into new bot_groups — MistralBot, Meta-ExternalFetcher, DeepSeekBot and others have been added without human intervention. The bot-IP index rebuilds every hour and feeds /api/v1/is_bot; pending candidates surface in the admin queue.

32. 2026-05-19 v1 · new

    Customer sites + ingest channel + JA4 signals

    The headline release: you can now ship request events from your edge to urlcap and get per-JA4 signals back. New site management surface:

    • /api/v1/sites — list and create sites (one per customer-facing property).
    • /api/v1/sites/{id}/domains — attach hostnames; UNIQUE across all sites (a hostname can only belong to one site).
    • /api/v1/sites/{id}/ingest_keys — mint per-site ingest tokens. Cleartext is returned exactly once at mint-time; urlcap only stores the SHA-256 hash.
    • POST /api/v1/ingest/{site}/events — NDJSON batches of request events, authed with the per-site token (not your urlcap API key). 1000 events / 4 MB per batch; partial failures never fail the batch.

    Reads land in /api/v1/ja4/signals: Cloudflare-style "what does this JA4 look like right now?" snapshot, recomputed every minute — 10 ratios (h2h3, browser, cache, error etc.), 4 per-site ranks, and 2 quantiles. Plus GeoIP (country / city / ASN) enrichment at ingest from MaxMind GeoLite2, also surfaced on /api/v1/is_bot, /api/v1/ip/lookup and /api/v1/ip/geo.

33. 2026-05-16 v1 · new

    Capture & extract can now sign outbound requests (Web Bot Auth)

    Pass webBotAuth: true to /api/v1/capture or /api/v1/extract and urlcap signs every outbound HTTP request it makes with its own Ed25519 keypair — the same RFC 9421 / Web Bot Auth machinery /web_bot_auth/verify already understands. Target sites that allow known crawlers but block unknown bots can verify our identity against the JWKS published at /.well-known/http-message-signatures-directory and choose to allow urlcap-attributed traffic. Signature covers @authority, signature-agent, @method and @target-uri; 60-second lifetime.

34. 2026-05-16 v1 · new

    Web Bot Auth signature verification

    New endpoint /api/v1/web_bot_auth/verify verifies an inbound HTTP request's RFC 9421 HTTP Message Signature against the operator's published Ed25519 key directory — the cryptographic identity check the Web Bot Auth draft proposes as a successor to IP-range and reverse-DNS checks. urlcap parses the Signature-Input + Signature + Signature-Agent headers, fetches the JWKS-style directory (TTL-cached 1 h), rebuilds the canonical signature base, and verifies with Ed25519 using the right keyid. Pairs naturally with /is_bot and /reverse_dns for a three-layer "is this bot really who it says?" check.

35. 2026-05-16 v1 · new

    robots.txt suite — fetch, allow/deny check, change-tracker with webhooks

    Four new endpoints under /api/v1/robots: fetch + parse a site's robots.txt, decide whether a URL is allowed for a given user-agent per RFC 9309, and create per-user watches that re-fetch on a schedule and POST an HMAC-SHA256-signed notification whenever the content changes. Snapshots are only stored on change, so a watch's history is the file's actual edit log. Fetched bodies are TTL-cached in-process for 1 hour.

    /api/v1/is_bot matches now carry a botGroup.honoursRobots block with four booleans — robots, crawlDelay, allow, sitemap — for whether each known bot has ever been caught skipping that aspect of robots.txt. Seeded conservatively: null = not researched yet, false = at least one credible violation report.

36. 2026-05-15 v1 · new

    Reverse DNS (PTR) with FCrDNS confirmation

    New endpoint /api/v1/reverse_dns resolves IPv4 / IPv6 addresses to their PTR names. With forwardConfirm=true, every PTR is re-resolved (A + AAAA) and we report whether the original IP is in the answer — the standard FCrDNS check used to verify crawlers like Googlebot or Bingbot. Results are cached in-process for the upstream record's TTL (clamped 30s–1h; negative answers 5 minutes), and every response reports ttlSeconds remaining plus cached. /is_bot picked up a matching reverseDns=true option for the common "is this IP a real bot and what's its PTR?" case.

37. 2026-05-13 v1 · update

    Bot detection: date & historical

    /api/v1/is_bot learned two point-in-time / lifetime parameters: date (an ISO-8601 as-of-date — answers "was this IP a known bot on this date?", including CIDRs that have since been retired) and historical=true (returns every CIDR ever published by any bot, regardless of whether it's currently active). Each match now carries addedAt, removedAt and active, so you can see when a CIDR was retired and why. Same in-memory index, same sub-millisecond lookups — see the reference.

38. 2026-05-13 v1 · new

    Bot detection — /api/v1/is_bot

    One call to tell if an address is a known web crawler. GET /api/v1/is_bot?ip=66.249.66.1 (or ?ips=a,b,c, or a JSON body for batches up to 200) returns isBot plus, on a match, the bot's search engine, bot group, the matching CIDR, and its category codes (SEARCH_INDEXING, AI_TRAINING, AI_SEARCH_OR_ANSWERING, …) with per-link confidence. Backed by a daily-refreshed in-memory CIDR index — no DB hit per request. Covers Googlebot, Bingbot, Yandex, DuckDuckBot & DuckAssistBot, Applebot, GPTBot & ChatGPT-User & OAI-SearchBot, ClaudeBot, PerplexityBot & Perplexity-User, AhrefsBot, Amazonbot, CCBot, and more. See the reference.

39. 2026-05-13 v1 · new

    Datasets — /api/v1/datasets

    Keep your own named, de-duplicated collections of IP / CIDR ranges or URLs: create a dataset, POST items to add, PUT to replace the whole set, DELETE items or the dataset itself, and ask GET /contains whether a value is in it (for IP datasets that includes "is this address covered by some CIDR in the dataset?"). Optional history mode keeps the items dropped by each replace, with the date they were deactivated — useful for tracking a set as it evolves. Per-plan caps for number of datasets, max items each, and whether history is allowed (free / developer / startup / business). See the reference.

40. 2026-05-12 v1 · new

    Extract: JSON content & JSONPath

    The extract tool now handles JSON responses: point a job at a URL that returns JSON (a REST API, say) and pull out values with JSONPath — selector becomes a JSONPath expression (e.g. $.results[*].name), with the familiar value / list / items shapes. JSON is auto-detected from the response, or force it with "content": "json". See the JSON content reference.

41. 2026-05-12 v1 · new

    Scheduled tasks — /api/v1/schedules

    Run a capture or an extract job at a future time — once (runAt) or on a recurring cron schedule (classic 5-field crontab, optional seconds field, @daily & co., evaluated in a time zone you choose). Each execution stores its full JSON result; fetch it from /api/v1/schedules/{id}/runs/{n}. Pause / resume / run-now / cancel, plus a Scheduled tasks page in your account. See the reference.

42. 2026-05-12 v1 · new

    Accounts & sign-in

    Self-service accounts: email/password sign-up with verification, password reset, per-user API keys, plans & billing, and one-click Sign in with Google, GitHub or Microsoft. API requests now also accept Authorization: Bearer api_… per-user keys alongside the legacy X-API-Key.

43. 2026-05-11 v1 · new

    IP & CIDR tool — /api/v1/ip

    Added IPv4/IPv6 tooling: /api/v1/ip/contains (is an address in a CIDR? — single or batch), /api/v1/ip/lookup (which of your stored ranges contain an address?), and /api/v1/ip/ranges to manage a list of named ranges. Stored in an optimised table — 16-byte IPv4-mapped keys, B-tree-indexed range bounds. See the IP & CIDR reference.

44. 2026-05-11 v1 · new

    Extract tool — /api/v1/extract

    Added an async extract tool: submit a job model (load a URL, perform actions — fill / click / wait / navigate / select — and pull out data with CSS/XPath selectors, optionally across paginated pages) and get a taskId; poll GET /api/v1/extract/{taskId} for the result. Every HTTP request the engine performs while running a job is recorded against the task. See the Extract reference.

45. 2026-05-11 v1 · new

    Capture tool — /api/v1/capture & the workbench

    Added the capture tool — send an HTTP request to any URL (with full control over method, headers and their order, body, redirects) and get the parsed response: status, headers, cookies, query parameters, body, timings. The home page is now a capture workbench. Every captured request and response (and all headers) is recorded. See the Capture reference.

46. 2026-05-11 v1

    Versioned API & the /api/v1/totp endpoint

    Introduced the versioned API base at https://urlcap.com/api/v1. The first endpoint, /api/v1/totp, mirrors the existing TOTP functionality but returns a structured JSON envelope — { version, requestId, data } — including the code, digit count, period, algorithm and seconds-until-rotation. Accepts both GET and POST.

    • New consistent error envelope: { version, error: { type, message } } with 400 / 401 statuses.
    • The legacy plain-text endpoint at /auth is unchanged and will stay that way.
47. 2026-05-11 site

    New developer site & API reference

    Launched the redesigned urlcap site — a marketing landing page and a full API reference at /docs — with a dark-mode toggle, copy-to-clipboard code samples, and a responsive layout.

48. earlier legacy

    TOTP at /auth

    The original endpoint: GET /auth?uri=otpauth://totp/... with an X-API-Key header, returning the current code as plain text. Still available, frozen for compatibility.

Looking for what's coming next? Custom HTTP request endpoints — full control over method, headers (including order), and body — are in progress.