Ask an engineer whether to build or buy web data collection and you'll usually get an estimate for the demo: an afternoon of work, a working scraper, done. Ask again a year later and you'll hear about the redesigns that broke it and the infrastructure invoices. Neither answer is the math. This post does the math: one concrete scenario, line items from public rate cards, and our own numbers from a year of assessing sites.
The scenario: 10 competitor sites, weekly refresh
A typical mid-size setup for a pricing analyst or a PM: ten direct-to-consumer competitor sites, around 2,000 products each, full catalog refreshed weekly. Prices, availability, names, variants, categories. That's roughly a million page loads a year (10 × 2,000 × 52), and the output has to be structured well enough that someone can actually build a price report on it.
The table prices running that yourself, with every assumption stated.
| Line item | Assumption | Annual cost |
|---|---|---|
| Initial build | 2 to 4 engineer-weeks at ~$3,500/week loaded | $7,000 to $14,000 (year one) |
| Ongoing maintenance | 0.2 to 0.5 FTE of a data engineer at $180k loaded cost | $36,000 to $90,000 |
| Third-party access services | ~1M page loads billed per GB or per page at public list prices: Bright Data from $8.40/GB or ~$3 per 1k page loads; Oxylabs ~$9.40/GB | $3,000 to $5,500 |
| Run infrastructure | Compute, scheduling, monitoring, storage | $1,200 to $4,800 |
| Spike reserve | One or two sites get much harder to collect mid-year | $2,000 to $10,000 |
A few notes on those numbers.
The loaded cost. $180k assumes a mid-level data engineer at roughly $135k base plus benefits, payroll taxes and overhead. If your team is cheaper, scale it down; the shape of the conclusion won't change.
Maintenance is the whole game. 0.2 to 0.5 FTE isn't padding. Ten sites means ten independently rotting scrapers: layout changes, platform migrations, sites getting harder to access, and silent schema drift, where the scraper keeps running but the prices are wrong. That last one is the expensive one, because nobody notices until a decision has already been made on bad data.
Price access at the rate DIY actually pays. Commercially important sites are rarely easy to collect, so a DIY setup ends up paying third-party access services $8 to $11 per GB, or a fee per page. At those list prices a million page loads costs a few thousand dollars, but list prices are the entry point: Bright Data's enterprise contracts run $25K to $500K+ per year, because hard sites don't cost list price. When a target site becomes much harder to collect, your cost per successful page can jump 5 to 50× overnight. That's what the spike reserve is for, and $10k isn't the worst case.
"I'll just use Apify" isn't an exit. Apify's tiers ($29, $99, $249/mo) are honest pricing for what they are: compute and a marketplace of actors. You still select, configure, run and maintain those actors yourself, so the platform fee trims the infra line and leaves the maintenance line untouched.
The total: roughly $50k to $125k in year one, and $40k to $110k every year after. The largest line is the fraction of an engineer you've permanently assigned to keeping the pipes alive, not the access fees.
What 300+ assessments say about the variance
The table assumes you know what your target sites will cost to collect. You don't, and that gap is where DIY budgets actually die.
Across the 300+ e-commerce brand sites we had assessed as of November 2026 (premium fashion, beauty and sportswear DTC sites), the sites that needed paid infrastructure spread roughly 110× in per-product collection cost between the cheapest and most expensive deciles.
Two caveats on that figure: it covers external collection costs only, without engineering time, and it's modeled at monthly refresh. Daily refresh is roughly 30× the volume.
A warning about how to read it: these are our costs, not a DIY builder's, so the table above prices access at public list rates instead. What the fleet data does establish is variance. Two sites that look identical in a browser can differ by two orders of magnitude in collection cost, and which kind you're holding is impossible to know from the outside. Most teams budget for the median site, then hit one or two from the expensive decile, and discover their annual estimate was a guess.
When building is the right call
Sometimes it genuinely is, and pretending otherwise would make everything above less believable.
Build if:
- Web data is your product. If you're building a price-comparison engine or a market-data product, collection is your core competency and the capability itself is the moat. Own it.
- You have real idle data-engineering capacity. Not "the team could squeeze it in": actual sustained slack, including the weeks when three scrapers break at once.
- Your needs are tiny and stable. One or two sites that are easy to collect, a refresh cadence measured in months. A cron job and a spreadsheet are fine. You don't need us, and you don't need Bright Data either.
If none of those describe you, the table above is your realistic bill. Buying turns it into a fixed line item.
The middle nobody prices
The market usually presents two options. Infrastructure platforms (Bright Data, Oxylabs, Apify) sell you the pipe; you still carry every row of the table except part of the build. Closed dashboards sell you conclusions inside their box: Crayon and Klue-class competitive-intelligence tools run $16K to $100K+ per year in the vendor pricing research we compiled in June 2026, typically with weeks of onboarding and your underlying data locked in their UI.
There's a third shape: buy the finished dataset itself. Structured, quality-checked, delivered to your warehouse or as files, exportable, yours. What "finished" means in practice (quality checks, freshness, field-level structure) is its own topic; we covered it in What business-ready data actually means. That's the shape Datka is built around, and the mechanics are on our product page.
On price: pricing is per project and quoted, so we won't paste a rate card here. As an anchor, a scoped managed feed for the ten-site scenario above costs a fraction of the 0.2 to 0.5 FTE maintenance line alone.
For the same decision from a brand team's side, with what to buy instead, see Buy vs build: competitor data for brand teams.
Run your own numbers
Everything above uses our assumptions. Yours differ, and the one line that matters most (which cost decile your target sites fall into) is exactly the one you can't estimate from a browser tab. Book a consultation and bring your list of target sites; we'll scope a managed feed against your own numbers.