How to Benchmark Proxy Providers: A Repeatable Test Plan

Most provider trials produce a number that does not survive contact with production. Three free trials get run on three different afternoons, a few hundred requests each, against whatever endpoint was handy. A winner emerges. Two weeks later the success rate on the production crawl sits ten points below the trial figure and nobody can explain the gap.

Measuring a proxy is trivial. Holding everything else still while you measure is the work. Anti-bot defences change their posture during the day, home connections in an exit pool come and go by the hour, and your own uplink is not constant either. Test two providers ninety minutes apart and you have measured the ninety minutes at least as much as the providers.

Fix the variables before you send a single request

Write the variable list down first. Each of these quietly changes the result if it drifts between providers, and each one is easy to fix in advance and impossible to fix afterwards.

• Clock. Every provider runs inside the same wall-clock window, ideally the same minutes, not merely the same afternoon.
• Machine and egress. One host, one uplink, one region. If you want US and EU numbers, that is two complete runs of the same code, not one run split across two locations.
• Target list. The same URLs in the same order for every provider, pinned in a file that does not change mid-trial.
• Concurrency. The same number of in-flight requests per provider. Doubling concurrency on one provider inflates its latency tail and can trip target rate limits that then get logged as provider failures.
• Client stack. Same HTTP library, same header set, same TLS fingerprint, same protocol version. A client on HTTP/2 with a browser-like JA3 gets treated very differently from curl defaults, and that difference can swamp the difference between providers.
• Rotation mode. Per-request rotation everywhere, or sticky sessions of identical length everywhere. Mixing the two across providers is the fastest way to produce a meaningless table. The trade-offs are in rotating vs sticky proxy sessions.
• Timeouts and retries. One connect timeout, one read timeout, one retry policy, applied identically. Log retries separately so a provider that only succeeds on its third attempt cannot hide inside the headline success rate.

Our own harness pins the same variables and then never stops, which is what keeps a comparison honest across months rather than an afternoon. The configuration we use is documented in our methodology.

Run every provider simultaneously

Sequential testing is the most common flaw in published proxy comparisons. Target defences adapt on a timescale of minutes. A site serving clean HTML at 09:00 may be issuing challenges at 11:00 because somebody else hammered it in between. Test provider A at 09:00 and provider B at 11:00, and the difference you write down belongs to the target.

Interleave instead. One process, one task queue, every provider represented in every slice of time.

# One worker pool. Every provider sees the same target in the same minute.
tasks = [(provider, url) for url in target_urls for provider in providers]
random.shuffle(tasks)  # decorrelate provider order from target order

# A per-provider semaphore keeps concurrency identical across providers
# even though they all share one queue.
limits = {p: asyncio.Semaphore(CONCURRENCY_PER_PROVIDER) for p in providers}

Shuffling matters as much as sharing the queue. If provider A always reaches a URL first, it pays for the cold cache and every provider behind it inherits a target that has already seen that URL requested this minute. Randomised order spreads that penalty evenly.

Pick targets that look like your production traffic

Test the sites you are actually going to scrape. Where contractual or legal constraints prevent that, choose stand-ins running the same anti-bot vendor at the same aggressiveness, with the same content shape, and state the substitution in your writeup so the result is not read as something it is not.

A useful target set covers the response types your crawler meets in production, weighted roughly by how much of your traffic hits each one:

• Static HTML behind a CDN edge rule. High volume, cheap per request, and highly sensitive to the reputation of the exit IP.
• A JavaScript-rendered page that needs a real browser, where bytes per request are an order of magnitude higher and the bandwidth bill is decided.
• A JSON API with a documented rate limit. This isolates raw transport quality from anti-bot behaviour, because the rules are published rather than guessed.
• A login-gated page, if your workload holds sessions. Sticky-session stability is a separate failure mode that per-request rotation testing never surfaces.

How many requests before the number means anything

Success rate is a proportion, so its uncertainty is set by sample size and nothing else. Around a true rate of 90 percent, the 95 percent confidence interval behaves like this:

A 200-request trial cannot separate a 91 percent provider from an 85 percent one, and most trials are smaller than 200 requests per target. Treat 1,000 requests per provider per target as the floor for any decision with money attached.

That arithmetic also assumes independent trials, which yours are not. Requests fired at one target inside one minute all share whatever mood that target is in, so the effective sample is smaller than the raw count. Spreading 1,000 requests across several hours, a weekday and a weekend produces a far better estimate than 1,000 requests fired in ninety seconds. We report rolling windows rather than instantaneous rates for that reason, and the long view of every provider we track is plotted in the performance graphs.

Separating provider failures from target failures

One success rate hides two unrelated problems. The proxy network can fail to carry your request at all, or it can carry it flawlessly to a target that refuses you. The first is the provider's fault. The second is a mix of exit reputation and your own crawler behaviour, and it responds to completely different fixes.

Define success with a content assertion

A 200 response is a weak signal. Soft blocks return 200 with a stub page, an empty results array, or a consent wall. Define success as an assertion against the payload: a CSS selector that must resolve, a JSON key that must exist, a byte-length floor for that specific target. Then record which assertion failed.

The gap between status-code success and assertion success is where the interesting findings live, and it varies between providers because it tracks exit reputation rather than transport. Both halves of that story are covered in proxy fraud score explained and proxy success rate explained.

A log schema you will not have to redo

Write one row per attempt rather than one row per URL, as JSON lines, so the whole trial loads into DuckDB or pandas afterwards without anyone writing a parser.

{"ts":"2026-04-28T09:14:02.118Z","run_id":"trial-07","provider":"alpha",
 "target":"listing_page","attempt":1,"session_id":"a7f3c1",
 "exit_ip":"203.0.113.44","exit_country":"US","status":200,
 "assertion":"pass","bytes_down":184320,"bytes_up":1180,
 "connect_ms":312,"ttfb_ms":901,"total_ms":1442,"error":null}

Four fields carry most of the analysis. attempt keeps retries out of the headline rate. bytes_down is what you are billed on, so without it there is no cost calculation at the end. exit_ip gives you the unique-IP count and the reuse rate for free, which is the honest version of the pool-size claim discussed in proxy pool size explained. assertion is the column you actually report.

Split timing into connect, time to first byte and total. A provider with a congested gateway and a provider with slow exits look identical in total time and completely different once you separate the handshake from the wait. Network-level timing deserves its own harness, described in how to test proxy speed.

Cost per successful request

Every other measurement in the trial exists to feed this one. Price per gigabyte is what a provider advertises. Cost per successful request is what your crawl costs.

cost_per_success = (avg_bytes_per_attempt / 1e9) * price_per_gb / assertion_pass_rate

Beta is 40 percent cheaper per gigabyte and about 6 percent more expensive per result. Failures are billable events: the TLS handshake, the challenge page and the retry all cross the meter. Establish during the trial whether your provider charges for failed attempts, because plenty do and it changes the ranking. The mechanics behind the list price are in residential proxy pricing explained.

Turning a one-off trial into an ongoing signal

A benchmark is valid for the window it ran in. Pools churn, providers renegotiate upstream supply, and a network that looked excellent in April can look ordinary by July. Re-run the same harness monthly against the same target list, keep every old log, and read the trend rather than the last run.

That is the entire premise of this site: one test, from US and EU hosts, every minute of every day, published raw. If you want to see the shape of the data before building your own harness, start with the live benchmark table and then line up two candidates on identical axes with the side-by-side comparison tool. Individual provider pages such as Evomi show how one network moves over time.

Once the shortlist is down to two, the open questions stop being technical: contract length, sub-user limits, how fast support answers at 3am. That side of the decision is covered in how to choose a proxy provider.