Ask an engineer whether to build or buy web data collection and you'll usually get an estimate for the demo: an afternoon with Playwright, a working scraper, done. Ask again a year later and you'll hear about the redesigns, the 403s, and the proxy invoice. Neither answer is the math. So here's the math: one concrete scenario, line items from public rate cards, and our own numbers from a year of assessing sites.
The scenario: 10 competitor sites, weekly refresh
A typical mid-size setup for a pricing analyst or a PM: ten direct-to-consumer competitor sites, around 2,000 products each, full catalog refreshed weekly. Prices, availability, names, variants, categories. That's roughly a million page loads a year (10 × 2,000 × 52), and the output has to be structured well enough that someone can actually build a price report on it.
Here's what running that yourself costs, with every assumption stated.
| Line item | Assumption | Annual cost |
|---|---|---|
| Initial build | 2–4 engineer-weeks at ~$3,500/week loaded | $7,000–$14,000 (year one) |
| Ongoing maintenance | 0.2–0.5 FTE of a data engineer at $180k loaded cost | $36,000–$90,000 |
| Proxies / unblocking | ~1M page loads at residential/browser tiers: Bright Data from $8.40/GB or ~$3 per 1k page loads via scraper API; Oxylabs ~$9.40/GB | $3,000–$5,500 |
| Headless-browser infra | Chromium fleet, scheduling, monitoring, storage | $1,200–$4,800 |
| Spike reserve | One or two sites escalate anti-bot mid-year | $2,000–$10,000 |
A few notes on those numbers.
The loaded cost. $180k assumes a mid-level data engineer at roughly $135k base plus benefits, payroll taxes and overhead. If your team is cheaper, scale it down; the shape of the conclusion won't change.
Maintenance is the whole game. 0.2–0.5 FTE isn't padding. Ten sites means ten independently rotting scrapers: layout changes, platform migrations, anti-bot upgrades, and silent schema drift, where the scraper keeps running but the prices are wrong. That last one is the expensive one, because nobody notices until a decision has already been made on bad data.
Price proxies at the residential tier, because that's what DIY pays. A DIY scraper works page by page through the rendered frontend, and the rendered frontend is exactly the surface anti-bot systems watch. That means residential proxies and unblocker products in the $8–11/GB class, or per-page scraper APIs, are the realistic rate card. At those list prices a million page loads costs a few thousand dollars, but list prices are the entry point: Bright Data's enterprise contracts run $25K–$500K+ per year, because hard sites don't cost list price. When a target deploys serious bot management, your cost per successful page can jump 5–50× overnight; we wrote up the mechanics in Anti-bot in 2026: why web data got expensive. That's what the spike reserve is for, and $10k isn't the worst case.
"I'll just use Apify" isn't an exit. Apify's tiers ($29, $99, $249/mo) are honest pricing for what they are: compute and a marketplace of actors. You still select, configure, run and maintain those actors yourself, so the platform fee trims the infra line and leaves the maintenance line untouched.
The honest total: roughly $50k–$125k in year one, and $40k–$110k every year after. Notice what dominates. It's never the proxies. It's the fraction of an engineer you've permanently assigned to keeping the pipes alive.
What 300+ assessments say about the variance
The table assumes you know what your target sites will cost to collect. You don't, and that gap is where DIY budgets actually die.
Across 300+ e-commerce brand sites we assessed this year (premium fashion, beauty, and sportswear DTC sites), 94% were feasible to collect. About half cost nothing in proxy spend, collected the way we collect them. But among the sites that did need paid infrastructure, per-product collection cost spread roughly 110× between the cheapest and most expensive deciles.
Two caveats to keep those figures honest: they cover proxy and collection-network costs only, without engineering time, and they're modeled at monthly refresh. Daily refresh is roughly 30× the bandwidth.
A warning about how to read them: these are not numbers a DIY builder can plug into a budget. They reflect collection methods we've refined across hundreds of sites; a team scraping page by page through the rendered frontend hits the anti-bot cost ladder more often and harder, which is why the table above prices proxies at the residential tier rather than at anything resembling our fleet's costs. What the fleet data does establish is variance. Two sites that look identical in a browser can differ by two orders of magnitude in collection cost, and which kind you're holding is impossible to know from the outside; determining that is precisely what an assessment is for. Most teams budget for the median site, then hit one or two from the expensive decile, and discover their annual proxy estimate was a guess.
When building is the right call
Sometimes it genuinely is, and pretending otherwise would make everything above less believable.
Build if:
- Web data is your product. If you're building a price-comparison engine or a market-data product, collection is your core competency and the capability itself is the moat. Own it.
- You have real idle data-engineering capacity. Not "the team could squeeze it in": actual sustained slack, including the weeks when three scrapers break at once.
- Your needs are tiny and stable. One or two friendly sites, no anti-bot, a refresh cadence measured in months. A cron job and a spreadsheet are fine. You don't need us, and you don't need Bright Data either.
If none of those describe you, the table above is your realistic bill, and buying converts it into a predictable line item, not an open-ended engineering liability.
The middle nobody prices
The market usually presents two options. Infrastructure platforms (Bright Data, Oxylabs, Apify) sell you the pipe; you still carry every row of the table except part of the build. Closed dashboards sell you conclusions inside their box: Crayon and Klue-class competitive-intelligence tools run $16K–$100K+ per year in the vendor pricing research we compiled in June 2026, typically with weeks of onboarding and your underlying data locked in their UI.
There's a third shape: buy the finished dataset itself. Structured, quality-checked, delivered to your warehouse or as files, exportable, yours. What "finished" means in practice (validation, freshness, field-level structure) is its own topic; we covered it in What business-ready data actually means. That's the shape Datka is built around, and the mechanics are on our product page.
On price: we're pre-launch, pricing is per-project and quoted, so we won't paste a rate card here. The honest anchor is this: a scoped managed feed for the ten-site scenario above costs a fraction of the 0.2–0.5 FTE maintenance line alone. Whether or not you ever compare vendors, you're always comparing against that engineer.
Run your own numbers
Everything above uses our assumptions. Yours differ, and the one line that matters most (which cost decile your target sites fall into) is exactly the one you can't estimate from a browser tab. We'll do that part for you: tell us your target sites and we'll come back with a feasibility read plus a real data sample pulled from them, so your build-vs-buy spreadsheet starts from measurements instead of guesses.