urlcap

Changelog

What's new in the urlcap API. Breaking changes ship under a new version path; everything additive lands within the current version. Subscribe to updates by emailing info@urlcap.com.

1. 2026-10-09 v1 · new

   Translation API

   POST /api/v1/translate translates text between 34 languages (European languages plus Chinese, Japanese, Korean and Arabic) with neural machine translation running on urlcap's own infrastructure — nothing is sent to a third party. Send a urlcap-style {"text": …, "to": "en"} object, or the Azure Translator v3 array shape with ?to= in the query for a drop-in migration (change the base URL, send your key as X-API-Key). Source language auto-detects when from is omitted; textType=html keeps markup intact. Companion endpoints GET /translate/languages and POST /translate/detect. Usage counts one request per 200 characters; the anonymous trial allows requests of up to 1,000 characters. See Translate text.

2. 2026-10-09 v1 · fix

   run-now returns the run it created; more parallel runs

   Task runs now execute up to 8 at a time per account (16 server-wide); further runs queue and start as slots free. Previously two runs at a time were executed for everybody, so concurrent run-nows waited on each other.

   A run-now that starts immediately now answers with data.run — the execution this request created, with its id and runNo taken from the insert. Previously the number had to be read from data.schedule.runs, a counter re-read after the start, so two simultaneous run-nows on the same schedule could both report the later number. Run objects everywhere also carry id now. See Schedules.

3. 2026-10-08 v1 · new

   Shared, system-wide external APIs

   External APIs now come in two scopes. Besides the ones you store on your account, urlcap can provide definitions that every account may call from a call_api action without storing anything first. GET /api/v1/apis lists both, each tagged "scope": "account" or "scope": "system"; a name or apiId resolves to your own definition first and otherwise to the shared one, so you can always shadow a shared API with your own. Shared definitions are read-only (PUT / DELETE only touch your own); their {{secret.*}} placeholders resolve against the calling account's secrets, so a shared vendor API can be bring-your-own-key. Run results report apiScope per step. See External APIs.

4. 2026-10-08 v1 · fix

   Unknown /api paths now return 404

   A request to an /api/... path that is not a documented endpoint used to answer 200 with a placeholder body. It now returns 404 with the standard error envelope ({"version":"1","error":{"type":"not_found", ...}}), the same shape every other v1 error uses. Nothing documented changes; if a client relied on a 200 from a typo'd path, it will now see the error instead. The legacy GET /auth TOTP endpoint is unaffected.

5. 2026-10-05 v1 · new

   External APIs — the call_api action

   Extract jobs can now call your own services mid-flow. Store an endpoint once — method, URL, default headers and body, and a credential kept encrypted (bearer, basic, header or query) — with POST /api/v1/apis or on the dashboard. A call_api action then sends values from the current page (input extractors) and from earlier steps ({{...}} in path, query, headers, body), stores the response as variables ({{name.status}}, {{name.body}}, one per JSONPath extractor) and can write it into the page DOM — target + mode (text, html, append, prepend, before, after, replace, attr:name) — so later extractors see the enriched page. Credentials never appear in the job, the trace or the result. Select a definition by alias (api) or by its stable numeric apiId, like proxies and certificates. See External APIs.

6. 2026-09-16 v1 · new

   post action, and POST downloads

   Some sites lay a form out across table cells; the engine parses that with the submit button outside the form, so a click matches the button and sends nothing. The new post action submits the form yourself: url (absolute or relative) plus fields, form-encoded, through the job's session, with the response becoming the current page. download gains method: "POST" + fields for a file behind such a button, and its url may now be relative. See actions.

7. 2026-09-16 v1 · improved

   result.session.id — the resumeFrom handle on every run

   The session block now names the run’s own task id. A direct /extract caller already had it as taskId, but a scheduled run’s run record carries no task id, so a run-now caller had no way to resume the session it had just created — and a skipGroups run without resumeFrom starts with an empty jar, logged out. Take result.session.id from the login run and pass it as resumeFrom alongside skipGroups. See reuse a session.

8. 2026-09-15 v1 · new

   Action groups, skipGroups, and extractor flags

   Actions take a group name; a job (or a run-now invocation of an on-demand template) passes skipGroups to switch whole groups off for that run — skipped steps are recorded on result.steps and never trip strict. Extractors take sets: when one succeeds it raises the named variable to true, so when guards can skip the login steps on a run that resumed a still-valid session, and result.variables tells you whether it was. run-now also accepts resumeFrom now. See groups and flags.

9. 2026-09-15 v1 · new

   Resume a previous run’s session — same cookies, same exit IP

   Every succeeded default-engine extract run now keeps its end-of-run browser state for 30 days: the complete cookie jar and the proxy it went out through. Pass that run’s taskId as resumeFrom on a later job and it starts with the same cookies (all hosts, with their flags) and — unless the job names its own proxy — the same exit IP. Log in once, fetch many times; a resumed run stores its own state, so a chain continues from the latest run. Results carry a session block (reusable, cookies, retainedUntil, resumedFrom). Default engine only for now. See reuse a session.

10. 2026-09-14 v1 · new

    Extract runs are no longer one status — per-action outcomes, match counts, strict mode, timing

    Every extract result now carries, without trace: steps (one entry per action — matched element count, performed, the url after the step and whether it navigated, durationMs, error), matches (per extractor: nodes matched, plus per-field empty counts on items — the difference between an empty listing and a rotted selector), timing (load / actions / extractors / pagination, per action type and per your own phase labels) and notFound (every selector lookup that found nothing). Set strict: true to fail the run when an unguarded action doesn't perform (error.type strict_action_failed, error.action = index). Downloads gain filename from Content-Disposition; the job model gains an optional name, shown in the dashboard instead of the URL; scheduled-run listings carry a notFound count. See per-action outcomes and notFound.

11. 2026-08-24 v1 · new

    Invocation parameters — run a template by id with different inputs

    A run-now can now carry a variables object — {"action":"run-now","variables":{"country":"US"}} — that merges over the template's stored variables seed for that one invocation. So the same saved template, invoked by id, can run with different entry parameters each time — driving {{...}} interpolation and conditional-action guards. The stored template is untouched. See the docs.

12. 2026-08-26 v1 · new

    Seed cookies on an extract job

    Extract jobs accept a cookies map that's seeded into the jar before the first request (scoped to the start URL's host), sent on every same-host request after. For sites that gate the session on a consent cookie their JavaScript would set — which a headless run never clicks through — just seed it, e.g. "cookies": {"CHG-cookie-accepted": "true"}. See the docs.

13. 2026-08-26 v1 · new

    Conditional actions — run a step only when a variable matches

    Any extract action can now carry a when guard so it runs only if a condition holds against a variable — an entry parameter or a value captured mid-flow. Checks: existence (exists), exact value (equals/notEquals/in), and numeric range (min/max). Skipped actions show in the trace as "skipped — condition not met". See the docs.

14. 2026-08-24 v1 · new

    Playground: view the page source after each step

    The extract playground now captures the page source after every step; each line in the execution list gets a view HTML link showing the exact DOM at that point — so you can see precisely what the page looked like when a click matched nothing. On the API it's the new "snapshots": true flag (each result.trace step gains a snapshot index); bounded to 30 per run, 2 MB each, served as text/plain. See the docs.

15. 2026-08-24 v1 · new

    Edit schedules in the playground + schedule update endpoint

    Extract schedules now have an Edit in playground link that opens the playground pre-loaded with their spec — tweak it, run it to watch the trace, then Save to schedule to update it in place or Save as new template to spin off a fresh on-demand job. Backed by a new PUT /api/v1/schedules/{id} that replaces a schedule's task spec (and optionally its name) while keeping its id, status, schedule and run history. See the docs.

16. 2026-08-24 v1 · new

    Extract playground — build a job and watch it run live

    A new playground in your account lets you paste an extract job model, run it on the default engine, and watch each step stream live — the initial load, every action's outcome (matched element / no match, click navigation, fill applied, {{secret.*}} values shown as «secret»), then the extractor results. It uses your account's secrets, proxies and certificates, so it mirrors a real run — ideal for dialing in selectors (e.g. a click on a button matched by text) before you save the job. The same trace is available on the API with "trace": true (streamed in progress.trace, echoed in result.trace). See the docs.

17. 2026-08-24 v1 · new

    On-demand schedule templates — save a task, invoke it by id

    Create a schedule with "active": false and no cron/runAt to get a saved, named task that never fires on its own but runs whenever you trigger it by id with POST /api/v1/schedules/{id} {"action":"run-now"} — it runs once and stays inactive. On-demand extract templates now resolve proxy pools, {{secret.*}} placeholders and the clientCertificate selector at each invocation (account-scoped, same as a direct /extract call), so templated logins with stored credentials work. New activate/deactivate actions toggle an existing schedule between live and on-demand. See the docs.

18. 2026-08-24 v1 · new

    Account secrets — login credentials out of job specs

    Store login passwords and tokens once (POST /api/v1/secrets, up to 200 per account) and reference them from extract jobs as {{secret.name}} in any action value, URL, selector or url_template. The target site receives the real credential at run time, but the submitted job, the stored task record, the result.variables echo and the recorded outbound request bodies carry only the placeholder or a «secret» marker. Values are AES-256-GCM-encrypted at rest and never returned by any endpoint after creation; rotate with "overwrite": true. See the docs.

19. 2026-08-24 v1 · new

    Client certificates (mTLS) on /capture and /extract

    Store up to 20 client certificates per account (POST /api/v1/client_certificates — a PEM pair or a PKCS#12 export) and present one during the TLS handshake of any capture or extract job with "clientCertificate": {"alias": "..."} — for partner APIs, corporate gateways and other mutual-TLS targets. Keys are verified against the leaf certificate at upload, stored AES-256-GCM-encrypted, and never returned by any endpoint; the list API exposes metadata only (subject, fingerprint, expiry). Extract jobs present the certificate on every HTTPS request they make. Requires an https target; not yet supported with the real-browser engine. Verified end-to-end against an mTLS reference host: 400 without the certificate, 200 with it, on both engines. See the docs.

20. 2026-08-24 v1 · new

    /extract: mid-flow extraction into a per-job variable store

    A new extract action type captures values at any point in the action sequence — from the DOM, a response header (from: "header:Name"), the cookie jar (from: "cookie:name"), or the current URL (from: "url", optionally with a regex) — and stores them in a variable store scoped to that one job. Later actions reuse them via {{name}} placeholders in value / url / selector — grab a CSRF token before a form, fill it back, build the next URL from an id on the page. Values land in a single per-job namespace by default (rewrites overwrite); naming the action stores them as name.field to record where they were captured. Jobs can seed initial values with a top-level variables object, URL substitutions are URL-encoded automatically (|raw opts out), and the store’s final state is echoed as result.variables for chaining into a follow-up job. List variables can also drive visit_each pagination: url_template + items_var build one detail URL per list element ({{item}} / {{item.field}}) instead of harvesting links off the page — extract the ids mid-flow, then walk the detail pages built from them, in one job. See the docs.

21. 2026-08-23 v1 · improved

    known-bot-ranges: one row per CIDR, unambiguous verified-engine flag

    The known-bot-ranges feed now emits exactly one row per distinct CIDR. Some operators publish a single egress range for several of their bots — a range could previously arrive once per owning bot, with definitive_search_engine reflecting each owner separately, leaving the merge (and the verified-engine verdict) up to the consumer. Each row now carries the OR of definitive_search_engine and honours_robots across owners, bot_group names a verified engine when one owns the range, and a new bot_groups array lists every owner ({id, name}). If your integration allow-lists verified engines from this feed, no change is needed — the flag is now simply correct on shared ranges.

22. 2026-08-15 v1 · new

    Configurable per-task socket timeout for /extract

    Extract jobs accept an optional socketTimeoutMs that raises the per-request connect/read timeout from the 30 s default up to a 120 s ceiling. This keeps a slow upstream proxy — e.g. an unlocker-style endpoint that can take 20–60 s to return a single page — from false-timing-out mid-crawl, while still capping any one stuck socket so it can’t pin a worker indefinitely. Unset or 0 preserves the previous 30 s behaviour.

    Verified end-to-end: an extract routed through an account proxy with socketTimeoutMs: 90000 completed a real page fetch and returned parsed content with no premature timeout.

23. 2026-07-30 v1 · fix

    Outcome timestamps stored in true UTC

    Outcomes posted to /api/v1/ingest/{site}/outcomes were being stored with ts one hour ahead during British Summer Time (a server-side timezone bug in the insert path — client-supplied timestamps were never at fault). Fixed; new outcomes are stored in true UTC. Historical rows from BST periods retain the +1h offset, and every row's arrival time was always recorded correctly alongside.

24. 2026-07-29 v1 · improved

    Human-outcome grace — logged-in customers stop looking like scrapers

    A heavy repeat visitor with a fully warm browser cache issues zero asset requests — at the log layer that's indistinguishable from a "high volume, no assets" scraper. Bot discovery now cross-references the outcomes channel: an IP that recently completed a purchase (purchase/completed), authenticated (auth/authenticated or signup), or solved a real captcha is exempt from scraper attribution and captcha-candidate flagging for a 72-hour window.

    Guard rails: grace stops above a request-volume ceiling (default 3,000 reqs/hour — one past purchase on a NAT egress can't shield a scraper behind the same IP), and a high suspicious-validation rate keeps the flag regardless (captcha farms solve captchas). Ship auth and purchase outcomes with the request_id of the originating request to opt your real users into the exemption.

25. 2026-07-29 site · new

    Known-bot directory — /bots

    A public directory of the crawlers, fetchers and bots urlcap tracks: urlcap.com/bots. Each bot has a profile page — User-Agent token, category, whether it honours robots.txt (and crawl-delay / Allow / Sitemap), verified-search-engine status, published IP-range counts, and aggregated observed activity (distinct IPs, TLS fingerprints, top countries and networks). The underlying verification data powers /is_bot and the known-bot-ranges feed.

26. 2026-07-29 site · new

    Live User-Agent statistics — /user-agents

    urlcap.com/user-agents shows the most-used User-Agent strings observed across urlcap (2.8M+ distinct UAs), all-time and trailing-7-day, with breakdowns by device class, browser, operating system and top robots. Parsed with Yauaa, recomputed hourly from a snapshot so the page is always instant.

27. 2026-07-29 site

    Light theme by default + a new typeface

    The site now defaults to the light theme everywhere; dark mode remains one click away on the topbar toggle and is remembered per browser. The whole site also moved to Courier Prime — a screen-optimised take on the classic typewriter face, self-hosted, with no third-party font requests.

28. 2026-05-28 v1 · new

    Candidate-IP feed — /sites/{id}/bot-ip-candidates

    IP sibling of the JA4-side /bot-ja4s feed: a small list of IPs exhibiting abusive behaviour on your site, ranked by a 5-component composite (block-ratio 40%, volume 20%, path breadth 15%, JA4 churn 15%, vuln-probe hits 10%). Already-classified IPs (in bot_observed_ips) and trust-list IPs are excluded from scoring entirely. Default min_score=0.50 score floor keeps the list small by construction.

    format flag supports json (default operator shape), txt (one IP per line) and cidr (each IP as /32 or /128) — the text formats are text/plain so you can feed straight into ipset, iptables, or a Cloudflare IP list.

29. 2026-05-28 internal

    Bot-candidate auto-promote, auto-dismiss, de-promote & reopen

    The candidate queue at /account/admin/bot-candidates is now self-maintaining. A daily 04:30 UTC job auto-promotes obvious bots that pass eight conservative gates (score ≥ 0.85, reqs ≥ 10k, distinct_ips ≥ 100, sites_seen ≥ 5, asset_ratio < 0.01, purchases = 0, age ≥ 48h, status = pending) into a single umbrella bot_group Auto-flagged scraper (discovery), and auto-dismisses stale rows (last_seen > 14 days ago + score < 0.7) to keep the queue bounded.

    The admin UI gains a header banner with live queue counts (pending / promoted-auto-vs-manual / dismissed) and two new per-row actions: De-promote (deletes the corresponding bot_observed_ja4s row, garbage-collects an orphaned per-candidate bot_group if any, sends the candidate back to pending) and Reopen (dismissed → pending for re-review).

    Both automated paths are independently switchable via system_config: bot.discovery.auto_promote.enabled (default false) and bot.discovery.auto_dismiss.enabled (default true). Audit trail on each promoted candidate (promotion_kind, promoted_to_bot_group_id, promoted_at) and on bot_groups created by manual promotion (discovery_kind = 'manual-from-candidate' or 'auto-umbrella').

30. 2026-05-27 v1 · new

    Site-level quick-check endpoints — /url-stats & /bot-traffic

    Two new helper endpoints on the sites surface that bottle up the most common investigation queries:

    • GET /api/v1/sites/{id}/url-stats?host=&path=&days=N — totals + status breakdown + per-day chart + a 25-row non-2xx sample for a specific URL. Answers "any rejections on URL X today?" without writing ClickHouse by hand.
    • GET /api/v1/sites/{id}/bot-traffic?bot_group=&days=N — recent traffic from a named bot_group (substring-match), with status mix and a sample of recent requests. Answers "are we blocking Google?" in one call. Bot identification matches every JA4 the discovery system has attributed to the bot_group via bot_observed_ja4s.
31. 2026-05-26 v1 · new

    Phase-level timings on /capture and monitor checks

    /api/v1/capture's response.timings object now carries a phase breakdown: dnsMs, connectMs (TCP + TLS), requestSendMs, ttfbMs, bodyMs, plus the resolvedIp the socket actually landed on. Phase fields are omitted when their lifecycle hook didn't fire — pooled keep-alive reuse skips DNS / connect, and followRedirects=true only captures the first leg's timings.

    Capture-kind URL monitors persist the same fields per check on monitor_checks and surface them on /monitors/{id}/checks + the monitor detail page. Extract-kind monitors don't get phase timings — HtmlUnit's connection layer doesn't expose them. While we were there: fixed the persisted check size_bytes column reading 0 on capture monitors (it was looking at sizeBytes in the response envelope, which is actually bodyBytes).

32. 2026-05-26 v1 · new

    URL monitors — up/down checks with webhook + email alerts

    New /api/v1/monitors (plus /checks, /uptime, /pause, /resume) let you schedule a recurring capture or extract against any URL and get notified on state transitions. Same primitives as UptimeRobot — plus full headless-browser checks, JSON-API validation via the extract path, and the User-Agent personas from /user_agent_profiles.

    Sub-minute polling on paid plans (30 s minimum on Startup + Business, 60 s on Developer, 300 s on Free). State transitions fire HMAC-SHA256-signed webhooks AND email through the platform mailer. Flapping is debounced via alertFailureThreshold — consecutive failures required before flipping to down, so single-shot 502s don't page anyone. Check history retained 30 days; sweep runs daily.

    Manage from the UI at /account/monitors — create, edit, view recent checks, see uptime over the last 30 days.

33. 2026-05-26 v1 · new

    User-Agent profiles for capture & extract

    New GET /api/v1/user_agent_profiles plus two new fields on capture + extract requests — userAgent (raw string override) and userAgentProfile (key into the catalogue). The catalogue is an operator-managed MySQL table, seeded with the common browser personas (chrome-latest-mac, firefox-latest-win, edge-latest-win, safari-latest-mac, …) plus googlebot and the default urlcap.

    On /extract the profile selects both the UA string AND the HtmlUnit BrowserVersion (Chrome / Firefox / Edge) so the in-page navigator.userAgent and the JS engine flavour line up. safari-* profiles still run on Chromium (HtmlUnit has no Safari engine) — navigator.vendor will read Google Inc.. googlebot only sets the UA string; for cryptographic urlcap attribution use webBotAuth=true.

34. 2026-05-23 v1 · new

    Per-minute traffic-shape signals on the bot-JA4 blocklist

    /api/v1/sites/{id}/bot-ja4s now returns reqs_per_minute_peak, reqs_per_minute_mean (over active minutes only), active_minutes and burstiness (coefficient of variation) per JA4. Useful for catching attack-shape JA4s that hide under a moderate asset_ratio but spike to thousands of req/min during their active windows — the case where score >= 0.70 and asset_ratio sits between 0.05 and 0.50 so neither Tier 1 nor Tier 2 fires.

35. 2026-05-22 v1 · new

    Bot-JA4 blocklist endpoint — one feed, edge-blocklist ready

    New GET /api/v1/sites/{id}/bot-ja4s: returns every JA4 the discovery system has flagged on a site over a sliding window (1..90 days), labelled known_bot with bot-group attribution or candidate with discovery score. Browser-shaped fingerprints are excluded; the strings are the keys you feed into nginx, Cloudflare WAF, or any edge map. Each row also carries cross_customer_action — how many other sites have explicitly blocked, allowed, or challenged this JA4 via edge_action outcomes — plus block_likely_on_this_site inferred from your own status-code mix.

36. 2026-05-22 v1 · new

    Single-IP intelligence endpoint

    GET /api/v1/ip/intelligence?ip=… composes every signal urlcap has on a single IP into one response — MaxMind geo, PTR records, bot-CIDR membership, every bot_observed_ips attribution row (cidr_match / ua_match / vuln_match / manual), trust-list status, and cross-site behavioural data over the last 1–7 days: request count, distinct JA4s/UAs/sites, status-code mix, asset ratio, vuln-probe hits, every JA4 with classification, top user agents, and cross-customer edge_action votes summed across every JA4 the IP has used. The operator-grade investigation endpoint.

37. 2026-05-22 v1 · quality

    Base62 site public ids + per-domain kind

    Every site now carries a 12-char base62 publicId alongside its numeric id. Use it wherever a site appears in a URL — /api/v1/sites/2DrxGfsYW0jv/bot-ja4s, /api/v1/ingest/2DrxGfsYW0jv/events — to avoid leaking enumeration order to anyone holding a URL. The numeric form remains accepted. Domains now carry a kind hint (doc / asset / api); informational only for v1, but expect future signals to lean on it (assets-vs-docs ratio per host, etc.).

38. 2026-05-21 v1 · new

    Outcomes channel: tell urlcap what happened next

    New POST /api/v1/ingest/{site}/outcomes accepts asynchronous verdicts keyed by the original event's request_id — kinds js_challenge / auth / purchase / edge_action. urlcap resolves the JA4 server-side at ingest time and denormalises it onto the outcome row so verdicts survive the 7-day raw-event TTL. Eight new outcome-derived signals surface on /api/v1/ja4/intelligence (js_challenge_attempts / passes / pass_rate, auth_observations, distinct_users, purchases, total_purchase_cents, last_purchase_at) plus ja4_ua_consistency — UA-spoofing tell where a JA4 keeps claiming different agent/OS values.

    Plain-HTTP traffic (no TLS handshake, no JA4) now also flows through ingest — IP-keyed aggregates and vuln-probe / bot-discovery still see those rows; JA4-keyed materialised views filter them out.

39. 2026-05-21 v1 · new

    Vuln-probe attribution + trust list

    Every ingested request is tagged at write-time with a vuln_probe_id if its path matches the curated vuln-scan registry (52 paths across 9 categories — exposed WP plugins, exposed .git, common dropper paths, etc.). IPs hitting three or more distinct probes are auto-attributed to the synthetic "Vulnerability scanner" bot_group; the discovery loop currently catches ~50–70 new scanner IPs per hour.

    New per-user trust list of IPs / CIDRs / domains the discovery scan never flags — protect monitoring probes, office IPs, and partner integrations from being auto-classified as scanners. Trust applies globally (signal modulation) while ownership stays per-user (admin audit).

40. 2026-05-20 v1 · new

    JA4 intelligence: OS / agent / device classification + 25-metric snapshot

    Three new read endpoints on top of the long-window aggregates:

    • /api/v1/ja4/intelligence — the rolled-up profile for one JA4 over 7 / 30 / 90 days. Returns the top OS / agent / device guess with per-dimension confidence (the score IS the confidence — apply your own threshold), diversity scores, and the new outcome-derived counts. Hourly recompute.
    • /api/v1/ja4/profile — drill into one dimension (os_name, agent_name, device_class, country, asn and 4 more), top-N values with HLL-merged distinct counts.
    • /api/v1/ja4/metrics — on-demand "prioritised 25" snapshot: trailing-hour rollup (counts + 10 ratios), optional IP+JA4 sub-block when ip= is supplied (with is_new_24h), and a top-10 JA4×UA breakdown over 24h with each UA's share of the JA4's volume.
    • /api/v1/ip/profile — per-IP behavioural summary on one site over N days (distinct JA4s/UAs/paths/hosts + heuristic / challenge counts). The proxy / NAT detection lookup.
41. 2026-05-20 v1 · new

    Bot discovery: auto-classify what your traffic actually contains

    A background loop now ingests every request, attributes IPs and JA4s to known bot groups (CIDR match, UA self-identification, vuln-probe pattern), and queues unknown candidates for review. UA-self-identifying bots are auto-classified into new bot_groups — MistralBot, Meta-ExternalFetcher, DeepSeekBot and others have been added without human intervention. The bot-IP index rebuilds every hour and feeds /api/v1/is_bot; pending candidates surface in the admin queue.

42. 2026-05-19 v1 · new

    Customer sites + ingest channel + JA4 signals

    The headline release: you can now ship request events from your edge to urlcap and get per-JA4 signals back. New site management surface:

    • /api/v1/sites — list and create sites (one per customer-facing property).
    • /api/v1/sites/{id}/domains — attach hostnames; UNIQUE across all sites (a hostname can only belong to one site).
    • /api/v1/sites/{id}/ingest_keys — mint per-site ingest tokens. Cleartext is returned exactly once at mint-time; urlcap only stores the SHA-256 hash.
    • POST /api/v1/ingest/{site}/events — NDJSON batches of request events, authed with the per-site token (not your urlcap API key). 1000 events / 4 MB per batch; partial failures never fail the batch.

    Reads land in /api/v1/ja4/signals: Cloudflare-style "what does this JA4 look like right now?" snapshot, recomputed every minute — 10 ratios (h2h3, browser, cache, error etc.), 4 per-site ranks, and 2 quantiles. Plus GeoIP (country / city / ASN) enrichment at ingest from MaxMind GeoLite2, also surfaced on /api/v1/is_bot, /api/v1/ip/lookup and /api/v1/ip/geo.

43. 2026-05-16 v1 · new

    Capture & extract can now sign outbound requests (Web Bot Auth)

    Pass webBotAuth: true to /api/v1/capture or /api/v1/extract and urlcap signs every outbound HTTP request it makes with its own Ed25519 keypair — the same RFC 9421 / Web Bot Auth machinery /web_bot_auth/verify already understands. Target sites that allow known crawlers but block unknown bots can verify our identity against the JWKS published at /.well-known/http-message-signatures-directory and choose to allow urlcap-attributed traffic. Signature covers @authority, signature-agent, @method and @target-uri; 60-second lifetime.

44. 2026-05-16 v1 · new

    Web Bot Auth signature verification

    New endpoint /api/v1/web_bot_auth/verify verifies an inbound HTTP request's RFC 9421 HTTP Message Signature against the operator's published Ed25519 key directory — the cryptographic identity check the Web Bot Auth draft proposes as a successor to IP-range and reverse-DNS checks. urlcap parses the Signature-Input + Signature + Signature-Agent headers, fetches the JWKS-style directory (TTL-cached 1 h), rebuilds the canonical signature base, and verifies with Ed25519 using the right keyid. Pairs naturally with /is_bot and /reverse_dns for a three-layer "is this bot really who it says?" check.

45. 2026-05-16 v1 · new

    robots.txt suite — fetch, allow/deny check, change-tracker with webhooks

    Four new endpoints under /api/v1/robots: fetch + parse a site's robots.txt, decide whether a URL is allowed for a given user-agent per RFC 9309, and create per-user watches that re-fetch on a schedule and POST an HMAC-SHA256-signed notification whenever the content changes. Snapshots are only stored on change, so a watch's history is the file's actual edit log. Fetched bodies are TTL-cached in-process for 1 hour.

    /api/v1/is_bot matches now carry a botGroup.honoursRobots block with four booleans — robots, crawlDelay, allow, sitemap — for whether each known bot has ever been caught skipping that aspect of robots.txt. Seeded conservatively: null = not researched yet, false = at least one credible violation report.

46. 2026-05-15 v1 · new

    Reverse DNS (PTR) with FCrDNS confirmation

    New endpoint /api/v1/reverse_dns resolves IPv4 / IPv6 addresses to their PTR names. With forwardConfirm=true, every PTR is re-resolved (A + AAAA) and we report whether the original IP is in the answer — the standard FCrDNS check used to verify crawlers like Googlebot or Bingbot. Results are cached in-process for the upstream record's TTL (clamped 30s–1h; negative answers 5 minutes), and every response reports ttlSeconds remaining plus cached. /is_bot picked up a matching reverseDns=true option for the common "is this IP a real bot and what's its PTR?" case.

47. 2026-05-13 v1 · update

    Bot detection: date & historical

    /api/v1/is_bot learned two point-in-time / lifetime parameters: date (an ISO-8601 as-of-date — answers "was this IP a known bot on this date?", including CIDRs that have since been retired) and historical=true (returns every CIDR ever published by any bot, regardless of whether it's currently active). Each match now carries addedAt, removedAt and active, so you can see when a CIDR was retired and why. Same in-memory index, same sub-millisecond lookups — see the reference.

48. 2026-05-13 v1 · new

    Bot detection — /api/v1/is_bot

    One call to tell if an address is a known web crawler. GET /api/v1/is_bot?ip=66.249.66.1 (or ?ips=a,b,c, or a JSON body for batches up to 200) returns isBot plus, on a match, the bot's search engine, bot group, the matching CIDR, and its category codes (SEARCH_INDEXING, AI_TRAINING, AI_SEARCH_OR_ANSWERING, …) with per-link confidence. Backed by a daily-refreshed in-memory CIDR index — no DB hit per request. Covers Googlebot, Bingbot, Yandex, DuckDuckBot & DuckAssistBot, Applebot, GPTBot & ChatGPT-User & OAI-SearchBot, ClaudeBot, PerplexityBot & Perplexity-User, AhrefsBot, Amazonbot, CCBot, and more. See the reference.

49. 2026-05-13 v1 · new

    Datasets — /api/v1/datasets

    Keep your own named, de-duplicated collections of IP / CIDR ranges or URLs: create a dataset, POST items to add, PUT to replace the whole set, DELETE items or the dataset itself, and ask GET /contains whether a value is in it (for IP datasets that includes "is this address covered by some CIDR in the dataset?"). Optional history mode keeps the items dropped by each replace, with the date they were deactivated — useful for tracking a set as it evolves. Per-plan caps for number of datasets, max items each, and whether history is allowed (free / developer / startup / business). See the reference.

50. 2026-05-12 v1 · new

    Extract: JSON content & JSONPath

    The extract tool now handles JSON responses: point a job at a URL that returns JSON (a REST API, say) and pull out values with JSONPath — selector becomes a JSONPath expression (e.g. $.results[*].name), with the familiar value / list / items shapes. JSON is auto-detected from the response, or force it with "content": "json". See the JSON content reference.

51. 2026-05-12 v1 · new

    Scheduled tasks — /api/v1/schedules

    Run a capture or an extract job at a future time — once (runAt) or on a recurring cron schedule (classic 5-field crontab, optional seconds field, @daily & co., evaluated in a time zone you choose). Each execution stores its full JSON result; fetch it from /api/v1/schedules/{id}/runs/{n}. Pause / resume / run-now / cancel, plus a Scheduled tasks page in your account. See the reference.

52. 2026-05-12 v1 · new

    Accounts & sign-in

    Self-service accounts: email/password sign-up with verification, password reset, per-user API keys, plans & billing, and one-click Sign in with Google, GitHub or Microsoft. API requests now also accept Authorization: Bearer api_… per-user keys alongside the legacy X-API-Key.

53. 2026-05-11 v1 · new

    IP & CIDR tool — /api/v1/ip

    Added IPv4/IPv6 tooling: /api/v1/ip/contains (is an address in a CIDR? — single or batch), /api/v1/ip/lookup (which of your stored ranges contain an address?), and /api/v1/ip/ranges to manage a list of named ranges. Stored in an optimised table — 16-byte IPv4-mapped keys, B-tree-indexed range bounds. See the IP & CIDR reference.

54. 2026-05-11 v1 · new

    Extract tool — /api/v1/extract

    Added an async extract tool: submit a job model (load a URL, perform actions — fill / click / wait / navigate / select — and pull out data with CSS/XPath selectors, optionally across paginated pages) and get a taskId; poll GET /api/v1/extract/{taskId} for the result. Every HTTP request the engine performs while running a job is recorded against the task. See the Extract reference.

55. 2026-05-11 v1 · new

    Capture tool — /api/v1/capture & the workbench

    Added the capture tool — send an HTTP request to any URL (with full control over method, headers and their order, body, redirects) and get the parsed response: status, headers, cookies, query parameters, body, timings. The home page is now a capture workbench. Every captured request and response (and all headers) is recorded. See the Capture reference.

56. 2026-05-11 v1

    Versioned API & the /api/v1/totp endpoint

    Introduced the versioned API base at https://urlcap.com/api/v1. The first endpoint, /api/v1/totp, mirrors the existing TOTP functionality but returns a structured JSON envelope — { version, requestId, data } — including the code, digit count, period, algorithm and seconds-until-rotation. Accepts both GET and POST.

    • New consistent error envelope: { version, error: { type, message } } with 400 / 401 statuses.
    • The legacy plain-text endpoint at /auth is unchanged and will stay that way.
57. 2026-05-11 site

    New developer site & API reference

    Launched the redesigned urlcap site — a marketing landing page and a full API reference at /docs — with a dark-mode toggle, copy-to-clipboard code samples, and a responsive layout.

58. earlier legacy

    TOTP at /auth

    The original endpoint: GET /auth?uri=otpauth://totp/... with an X-API-Key header, returning the current code as plain text. Still available, frozen for compatibility.

Looking for what's coming next? Custom HTTP request endpoints — full control over method, headers (including order), and body — are in progress.