The Real Cost of Scraping: A Cost-Per-Clean-Record Model
Why per-GB and per-IP pricing hide what your pipeline actually costs
If you're pricing out a scraping pipeline, here's the short version: stop comparing proxy providers on cost-per-GB or cost-per-IP. Compare them on cost-per-clean-record, total pipeline spend divided by the number of records that actually made it into your dataset usable. A cheaper proxy with a high ban rate is often the more expensive option once you do this math.
We said we'd walk through this after the AI/ML data collection piece, here's the full model, based on how we see teams running proxy infrastructure at Squid Proxies actually evaluate providers once the initial vendor comparison is over.
Why cost-per-GB is the wrong number
Proxy pricing pages are built around inputs: dollars per gigabyte, dollars per IP, dollars per request. Those are easy to compare across vendors, which is exactly why they get treated as the decision metric.
But none of them tell you what you actually paid for. A pipeline can burn through bandwidth on retries, get rate-limited into returning partial pages, or pull content that fails validation downstream. All of that shows up as "spend" on your invoice and as nothing in your dataset. Two providers with identical per-GB pricing can produce wildly different costs once you account for what each one actually delivers.
The fix is to price the output, not the input.
The formula
Cost per clean record = Total pipeline spend / Usable records
Where "total pipeline spend" is everything the record cost you, not just the proxy bill:
Proxy/bandwidth cost
Retry cost (bandwidth and time spent on requests that had to be repeated)
Compute cost for the scraping infrastructure itself
Cleaning/validation cost (dev or pipeline time spent filtering out malformed, incomplete, or duplicate records)
And "usable records" means records that passed your own validation, not rows returned, not pages fetched. A response that came back with a 200 status but got served a CAPTCHA page isn't a usable record, even though it cost you bandwidth like one.
A worked example
Say you're comparing two providers for a 10-million-page crawl.
Provider A: $0.80/GB, average page size 500KB, 8% ban/retry rate, 3% of returned pages fail validation.
Provider B: $1.40/GB, same page size, 2% ban/retry rate, 1% fail validation.
Raw bandwidth cost looks like it favors Provider A by a wide margin. But once you factor in retries and validation failures:
| Provider A | Provider B | |
|---|---|---|
| Base bandwidth cost | $4,000 | $7,000 |
| Retry overhead (extra fetches) | +$320 (8%) | +$140 (2%) |
| Total spend | $4,320 | $7,140 |
| Usable records (of 10M) | 8.73M | 9.7M |
| Cost per clean record | $0.000495 | $0.000736 |
Provider A still wins here, the ban rate wasn't high enough to flip the outcome. But narrow the gap (say Provider A's ban rate climbs to 20% during a target site's anti-bot update, which happens more than vendors like to admit) and the ranking reverses. The point isn't that cheaper proxies are always worse, it's that you can't know which provider actually wins without running this number, and the answer can change over the life of a project as target sites change their defenses.
Where the hidden costs actually hide
Three places teams consistently under-count:
Retries. A retried request costs bandwidth twice but only produces one record, if it produces one at all. High-retry pipelines can look cheap on a proxy invoice and expensive on a cost-per-record basis.
Silent partial failures. This is the failure mode we flagged in the AI/ML data collection piece: a blocked scraper doesn't error out, it just returns thinner or slightly wrong content. That content often passes a naive completeness check and fails a real one later, downstream, where it's much more expensive to catch.
Rotation mismatch. Rotating too aggressively on targets that expect session continuity, or too rarely on targets that fingerprint fixed IPs, both increase your ban/retry rate, which is exactly the input that erodes cost-per-record. Getting rotation strategy right isn't just a reliability improvement, it's a cost lever.
Turning this into an ongoing number, not a one-time comparison
The model above is useful for picking a provider, but it's more useful as a metric you track continuously. Target sites change their bot detection over time; a provider that scored well in your initial evaluation can drift as ban rates creep up months later.
Two practical ways to keep this number honest:
Log validation pass/fail at the record level, not just HTTP status codes, so "usable records" reflects your real bar rather than a proxy that merely didn't error.
Recompute cost-per-clean-record on a rolling basis (weekly or monthly, depending on volume) rather than only at procurement time, treat it as a health metric for the pipeline, not a one-off vendor comparison.
Once you're tracking this number over time, it also becomes the fastest way to notice when a target site has changed its defenses before your dataset quality degrades far enough to be obvious downstream.
