If you last built a scraper around 2019, the mental model was simple: rotate a few IPs, set a believable User-Agent, respect a rate limit, done. In 2026, most commercial sites of any size sit behind a dedicated detection engine (Cloudflare Bot Management, Akamai Bot Manager, DataDome, Kasada, HUMAN) that scores every connection before it reaches the application. The data is still out there, and it's still collectible. What changed is the price of collecting it reliably, and the fact that the price is now set per site, per defense tier, and can move overnight.
This post covers what the detection stack actually does, what each rung of the response ladder costs, and what we learned assessing 252 e-commerce sites this year, including two findings that surprised us.
What the detection stack actually does
Modern bot management doesn't rely on one signal. It stacks several, and each one closes a door that used to be open.
TLS and HTTP/2 fingerprinting. Before you've sent a single header, the way your client negotiates encryption identifies your software. The order of cipher suites, extensions, and HTTP/2 settings forms a fingerprint (JA3 and its successor JA4 are the standard ways to hash it), and Python's requests or Go's default HTTP client look nothing like Chrome. A detection engine can drop you at the handshake: no page, no error message, no clue.
IP reputation tiers. Every IP address arrives pre-scored. Datacenter ranges (AWS, GCP, Hetzner) are effectively flagged on sight; residential and mobile IPs inherit trust from the real subscribers behind them. This scoring is what created the tiered proxy market: the tenfold premium for residential traffic is the price of borrowed trust.
Behavioral scoring. Client-side sensor scripts, which vendors like DataDome and Kasada ship as a core feature, watch how a session behaves: mouse movement, scroll cadence, timing between page loads, whether the JavaScript environment looks like a real browser or a headless one wearing a costume. Individual signals can be faked. Faking all of them consistently, at scale, is expensive.
Managed challenges. The visible tier: Cloudflare's interstitial page, a CAPTCHA, a JavaScript proof-of-work that burns CPU before the page loads. These are solvable by machine, but slowly and at a cost per solve, which is exactly the point. The challenge doesn't need to stop you; it needs to make you unprofitable.
Silent degradation and response poisoning. The tier that should worry you most gives you a clean 200 and wrong data: prices with a hidden offset, product lists quietly thinned, stale cached responses served only to suspected bots. Your pipeline stays green while your dataset rots. This is why "my requests get 200s" stopped meaning "my data is right," and why validation has to be a first-class part of any collection pipeline. We've written about what that takes in what business-ready data means.
The new unit of account: cost per successful request
The old metric was cost per request. The metric that actually matters now is cost per successful request: a fetch that returned complete, correct data. Each defense tier a site runs pushes you up a response ladder, and each rung has a public market price:
| Rung | What it is | Public order-of-magnitude cost |
|---|---|---|
| Direct requests | Your own servers' IPs, plain HTTP clients | $0 in network cost |
| Datacenter proxies | Rented cloud IPs, rotated | ~$0.60/GB |
| Residential proxies | Real-household exit IPs | ~$8/GB (Bright Data lists $8.40) |
| Scraping browser | Full headless browser, residential exit, challenge solving | ~$8–11/GB |
Per-gigabyte prices understate the spread, because a full browser session doesn't just cost more per gigabyte; it also pulls far more gigabytes, loading scripts, styles, and images a plain HTTP fetch would skip. In our fleet, measured per product successfully collected, residential-tier sites cost roughly 8× what datacenter-tier sites do, and browser-tier sites about 18×.
The nastier property of the ladder is that your rung isn't stable. A site that switches on a new vendor, or tightens an existing one, can move your collection up two rungs overnight, and a 5x–50x swing in your bill arrives with no warning and no changelog. If you're doing the build-vs-buy math on data collection, this volatility is the line item most teams miss. We walk through it in the honest build-vs-buy math.
What 300+ assessments taught us
Across 300+ e-commerce brand sites we assessed this year (premium fashion, beauty, and sportswear DTC sites), 71% run detectable anti-bot technology. Among the sites where collection was feasible, the tier split, collected the way we collect them, came out to roughly 40% needing no proxy spend at all, 25% on datacenter proxies, and 35% needing residential IPs or full browser rendering.
Two findings in that data surprised us.
First: about half the sites cost nothing in proxy spend, even though most of them run anti-bot tech. That's not because they're undefended. It's because the route you take through a site matters more than the wall in front of it, and for many sites there's a viable route that the expensive defenses simply aren't guarding. Finding the cheapest viable route per site is most of what our assessment work is, and it's the part of the craft we don't publish.
That conditionality matters, so let's be explicit about it: these are the numbers our methodology produces, and they aren't a DIY benchmark. A team scraping page-by-page through the rendered frontend faces the full ladder far more often, which means residential and browser tiers, the 8×–18× unit costs, and the overnight repricing risk all land on their invoice.
Second: of 300+ assessments, not one infeasible verdict was caused by anti-bot. Every site we marked infeasible failed for structural reasons: no priced catalog on the public web, or commerce living entirely in closed channels like chat-commerce apps and dealer portals. Sufficient spend gets through essentially any defense; no spend conjures data a site never publishes. Anti-bot decides your cost, not your feasibility. Site architecture decides feasibility.
Honest caveats on all of the above: our cost figures cover collection-network bandwidth only, modeled at a monthly refresh cadence (daily refresh scales roughly 30×), and the sample skews premium DTC, a segment that runs heavier defenses than the average online store.
Why this doesn't reverse
It's tempting to read 2026's defenses as a phase that market pressure will soften. The incentives point the other way.
Imperva's 2024 Bad Bot Report measured automated traffic at just under half of all web traffic. For a retailer, every one of those requests costs infrastructure money, skews analytics, and occasionally feeds a competitor, so blocking pays for itself. The AI-crawler backlash made blocking respectable too: when Cloudflare began blocking AI crawlers by default for new sites in 2025, aggressive bot filtering became a default posture rather than a niche choice. And critically, defense is now vendorized: a mid-size shop doesn't build any of this, it ticks a checkbox in a CDN dashboard and inherits the detection research of a company with thousands of customers. Offense has to keep pace per site; defense improves on a vendor release cycle.
For anyone who needs external e-commerce data, that leaves two workable positions. You can staff the treadmill: fingerprint-accurate clients, proxy vendor relationships, per-site validation, and an engineer who owns the pager when a target flips tiers. Or you can hand it to a team that runs that fight across hundreds of sites, where the fixed cost of staying current amortizes into something a single data budget can carry. Both are legitimate; only one of them is your core business.
Find out which tier your targets sit in
The single highest-value fact in this whole topic is site-specific: which rung of the ladder your target sites actually sit on, collected well. That's precisely what our free assessment answers: you name the sites, we come back with the feasibility verdict, the cost tier, and a real data sample you can inspect before spending anything. Request one here, or read the FAQs first if you want the details on scope, pricing, and how we handle compliance.