Field notes from the people who run the crawlers.
Postmortems, datasets, and reference architecture from real third-party incidents.
One diagram. Every moving part.
The detector is a fleet of isolated Chromium browsers driven by the DevTools Protocol, paired with a per-session MITM proxy, fused with threat-intel providers, and graded by a behavioural signature engine. Everything below downstream is reproducible from raw artifacts.
From schedule tick to alert email.
Each scan walks deterministically through ten stages. Failure at any stage is captured as a hard_failure or warning in the coverage contract — never silently swallowed.
Two independent observers per request.
Every request is captured twice — once by the browser via DevTools Protocol, once by the MITM proxy at the wire. Records are joined into one canonical_flow with a correlation score. A flow seen by only one observer is automatically suspect.
Pattern matched. Intent classified.
Every captured script body, response, and command line is run through a regex engine. Hits are grouped by intent — Execution, Persistence, Obfuscation, Download, Recon — each with its own severity tier feeding the final risk score.
- ExecutionWScript · MSHTA · regsvr32
- PersistenceRun/RunOnce · schtasks · WMI
- Obfuscationeval(atob) · fromCharCode
- DownloadADODB.Stream · certutil · BITS
- Reconnavigator.plugins · canvas fp
Every domain. Every IP. Every relationship.
A typical e-commerce page touches 60–150 third-party domains across a single visit. The crawler records all of them, classifies each one (confirmed contact vs candidate), and graphs the supply-chain hops so a single suspicious script can be traced to every domain it loaded and every IP that served it.
Confirmed vs candidate. Never inflated.
The commercial monitoring contract draws a hard line between domains the browser actually contacted and those that only appeared in source bytes. This tree is the canonical taxonomy used in every report.
Five snapshots per scan. No payload deferred long enough to hide.
The crawler holds the page open and re-observes the live DOM at five offsets — capturing request count, runtime artifacts, and a fresh DOM sha256 each time. Drift between offsets is captured with full provenance.
Every domain gets a number. Show your work.
The per-domain risk score is deterministic: WHOIS age tier × context multiplier × correlation multiplier × signature severity. Every input is preserved in the alert so the score is always reproducible.
- age_score — 0 (2y+) to 35 (today)
- context_mul — script_src +8, iframe +8, redirect +10, sw +8
- corr_mul — intel hit +18, lookalike domain +8
- sig_severity — Critical / High / Medium / Low / Info
risk = age_score + Σ context_mul[i] + Σ corr_mul[j] + Σ sig_severity[k] where: age_score ∈ [0, 35] context_mul[i] ∈ [+8, +10] corr_mul[j] ∈ [+8, +18] sig_severity[k]∈ [+1, +30] verdict = risk ≥ 60 → Critical risk ≥ 40 → High risk ≥ 20 → Medium risk ≥ 5 → Low else → Info
Speak the crawler's language.
Every term used on this page, in alerts, and in raw artifacts — defined precisely. Filter by term, category, or any word in the definition.
- CDP Browser
- Wire protocol used to drive and instrument Chromium. The crawler attaches via CDP to capture every Network.request, Runtime.evaluate, Page.frameNavigated, and Target.attachedToTarget event from inside the browser process.
- MITM Proxy Network
- A per-session HTTPS-intercepting proxy with an ephemeral CA installed only in the throwaway browser profile. Captures full request and response bodies, TLS metadata, and the actual remote IP for every flow at the wire.
- Canonical Flow Data Model
- A single request record produced by joining the CDP-side observation and the proxy-side observation by URL hash and timing. Carries seen_by_cdp, seen_by_proxy, canonical_id, sha256, and a correlation_score.
- Correlation Score Scoring
- Numeric agreement between the two observers (CDP vs proxy) for a single flow. 1.0 means both observers agree on URL, method, status, and body hash. Low scores flag tampering, racing redirects, or single-source ghosts.
- Confirmed Contact Evidence Class
- Top tier of the evidence hierarchy. The browser actually opened a TCP+TLS session to this domain and received bytes. Backed by exact_proxy_remote_ip and confirmed_loaded.
- Candidate Evidence Class
- Tracked but never claimed as loaded. Includes embedded_only, redirect_only, dns_resolution_only, and deobfuscated. Reported separately from confirmed contact so counts are never inflated.
- Embedded Only Evidence Class
- A URL appeared in source bytes (HTML, script, JSON) but the browser never fetched it during the scan window.
- Deobfuscated Evidence Class
- A URL or command recovered from atob, fromCharCode, hex, or base64 inside captured script bodies. Never claimed as loaded unless also confirmed by CDP or proxy.
- Observation Timeline Crawl
- The five re-observation offsets (0s, 5s, 15s, 30s, 60s) at which the crawler snapshots dom_sha256, request_count, runtime_artifact_count, script_count, and iframe_count.
- Resource Graph Data Model
- Typed node-and-edge graph (resource_graph.json) linking URLs, domains, and IPs so a single suspicious script can be traced to every domain it loaded and every IP that served it.
- RDAP Enrichment
- Successor to WHOIS for IP and ASN ownership lookups. Used to attach the canonical RIR handle (ARIN, RIPE, APNIC, LACNIC, AFRINIC) and netblock owner to every third-party IP.
- Signature Detection
- One of the 70+ regex behavioural rules grouped by adversary intent (Execution, Persistence, Obfuscation, Download, Recon). Each hit carries a severity tier feeding the per-domain risk score.
- Coverage Contract Quality
- Self-grading manifest each scan emits as coverage_dashboard_summary.json. Records hard_failures and warnings so a degraded scan never silently passes.
- Risk Score Scoring
- Deterministic per-domain number: age_score + Σ context_mul + Σ corr_mul + Σ sig_severity. Verdict bands at 60 / 40 / 20 / 5 map to Critical / High / Medium / Low / Info.
- Redirect Path Data Model
- Reconstructed multi-hop 30x chain with a stable path_id, external_hop_count, and dual-source confirmation from both proxy and CDP.
- Domain Age Tier Scoring
- WHOIS-derived bucket from 0 (registered 2+ years ago) to 35 (registered today). Newly registered domains carry the highest base risk.