How to Cut Proxy Bandwidth Costs Without Losing Coverage

Residential proxy contracts meter one thing: bytes across the gateway. Not requests, and not pages. That makes your invoice a direct function of how much data your crawler is willing to accept, and the default settings of every scraping stack accept far more than they need.

None of the techniques below reduce coverage. You visit the same URLs, extract the same fields, and keep the same success rate. What changes is how many bytes cross the meter on the way to the same result.

What the meter is actually counting

Providers count traffic in both directions at the gateway: your request headers going out, the full response coming back, and every redirect hop in between. The number is wire bytes, which means a gzip-compressed body is billed at its compressed size and an uncompressed one is billed at full weight. Some providers also count the bytes of failed and blocked responses, because those bytes really did traverse their network.

Read the metering paragraph in your provider's terms before you optimise anything, because the details differ. A few count only response bodies. Most count everything. The pricing mechanics guide covers how tiers and overage rates interact with that number, and the price per 100GB rankings normalise the rate card itself.

Where a page's weight goes

The HTML document you parse is nearly always the smallest thing a browser downloads for a page. Hero images, product photography, webfonts, autoplaying video, tag managers, session replay scripts and chat widgets make up the rest. A scraper that drives a real browser pays for all of it at residential rates.

Refuse the requests you never needed

Playwright and Puppeteer both let you intercept every outbound request and abort it before a byte is spent. Register the handler on the browser context rather than the page so it survives new tabs and popups.

// Playwright: refuse everything the parser will never look at
const BLOCK_TYPES = new Set(['image', 'media', 'font', 'stylesheet'])
const BLOCK_HOSTS = [
  'google-analytics.com',
  'googletagmanager.com',
  'doubleclick.net',
  'facebook.net',
  'hotjar.com',
]

await context.route('**/*', (route) => {
  const request = route.request()
  if (BLOCK_TYPES.has(request.resourceType())) return route.abort()

  const host = new URL(request.url()).hostname
  if (BLOCK_HOSTS.some((blocked) => host.endsWith(blocked))) return route.abort()

  return route.continue()
})

Two details decide whether this works in production. First, route.abort() is cheaper than route.fulfill() with an empty body, because fulfilling still requires the browser to process a response. Second, the host list beats the type list for third-party tags, since analytics beacons arrive as script and xhr and you rarely want to block those categories wholesale.

Outside a browser the equivalent control is your HTTP client. Never follow a redirect chain you have not bounded, never download a body whose content type you do not intend to parse, and set follow_redirects explicitly rather than inheriting a default.

Call the JSON endpoint instead of rendering the page

Most modern sites fetch their content as JSON after the initial paint. Open the network tab, filter to XHR, reload, and look for the request that carries the fields you want. Replicating that one call replaces a browser session and its dozens of subresource fetches with a single small response, and the JSON schema tends to be more stable than the DOM you would otherwise be selecting against.

• Copy the request faithfully. Right-click the entry in the network tab and copy it as cURL so you inherit the exact headers, then delete headers one at a time until it breaks.
• Watch for signed parameters. Some endpoints require a token minted by the page bundle. If the token has a long expiry, fetch one page per session and reuse it across many API calls.
• Check pagination limits. An endpoint that accepts a page size of 100 costs you a hundredth of the requests that a paginated HTML listing does.
• Prefer the mobile app API where one exists. These are usually leaner responses with fewer presentational fields.

When the JSON path exists, keep the browser only for bootstrapping a session and let cheap API calls carry the data. Wiring that split into a real crawler is covered in the web scraping proxy setup guide.

Compression, conditional requests and deduplication

Advertise only the encodings you can decode

HTML, JSON, CSS and JavaScript compress to a fraction of their raw size, and the meter counts the compressed bytes. Modern clients negotiate this for you: httpx and requests set Accept-Encoding based on which decoders are installed. Hand-writing the header to include br when the brotli package is missing gets you an undecodable body, so install the extra instead of editing the header.

In a headless browser this is already handled. In a raw socket client, a custom Go transport, or anything that sets Accept-Encoding: identity for simplicity, you are paying several times over for every text response.

Ask before you download

A HEAD request returns headers with no body, which is enough to read Content-Length, Content-Type, ETag and Last-Modified. That is how you decide whether a large file is worth pulling. On recrawls, send the stored ETag back as If-None-Match or the stored timestamp as If-Modified-Since. An unchanged resource answers 304 Not Modified with no body at all, and on a site that refreshes a small fraction of its pages each day, revalidation turns most of a recrawl into headers.

Deduplicate before the fetcher sees the URL

Crawlers waste real money re-fetching the same content behind different URLs. Canonicalise aggressively: strip tracking parameters such as utm_source and gclid, sort remaining query parameters, drop fragments, normalise trailing slashes and case-fold the host. Then hash the result and check a seen-set before scheduling. Content hashing the response body on top of that catches the mirrors that URL normalisation misses.

Stop paying for responses you will discard

A blocked response costs the same per byte as a good one. Challenge interstitials are usually small, but a soft block that returns a full styled page, or a 200 response containing an empty result set, can weigh as much as the real thing. Stream the response, inspect the first chunks, and abandon the transfer as soon as you know it is worthless.

import httpx

MAX_WIRE_BYTES = 512 * 1024
SOFT_BLOCK = ('captcha-delivery', 'cf-chl-opt', 'Access Denied', 'Request unsuccessful')

client = httpx.Client(
    proxy='http://customer-country-us:[email protected]:7000',
    timeout=httpx.Timeout(connect=10.0, read=20.0, write=10.0, pool=5.0),
    follow_redirects=True,
    max_redirects=3,
)

def fetch(url: str) -> str | None:
    with client.stream('GET', url) as response:
        content_type = response.headers.get('content-type', '')
        if 'html' not in content_type and 'json' not in content_type:
            return None  # exiting the block closes the stream before the body arrives

        parts = []
        for index, chunk in enumerate(response.iter_text()):
            if index < 4 and any(marker in chunk for marker in SOFT_BLOCK):
                return None  # soft block recognised in the first few kilobytes

            parts.append(chunk)
            if response.num_bytes_downloaded > MAX_WIRE_BYTES:
                raise ValueError('byte cap hit on ' + url)

        record_cost(url, response.num_bytes_downloaded)  # your metrics hook
        return ''.join(parts)

Three things are doing the work here. Leaving the with block early closes the connection so the remaining body is never transferred. num_bytes_downloaded reports the raw bytes off the wire, which is the figure that matches your invoice rather than the decompressed length. And the cap converts a runaway response into a loud failure instead of a silent charge. Which markers to match on is a per-target question covered in how to avoid getting blocked while web scraping.

Split the traffic by what the target actually inspects

Sites that gate their HTML behind bot management usually serve their images from a CDN that checks nothing. If your job genuinely needs the media, fetch the document through the residential pool and the assets through a datacenter pool on the same crawl. Test it with one image: request it over a hosting IP and compare the bytes and the status to the residential fetch. If both return 200 with an identical Content-Length, that entire asset class can move to the cheap tier.

The same split applies to robots.txt, sitemaps, RSS feeds and JSON-LD endpoints, which are published for machines and almost never protected. Sending those over residential bandwidth is a habit rather than a requirement. The full comparison of the two pool types is in residential vs datacenter proxies, and provider pages such as Evomi show which vendors sell both from one account.

What the savings look like in money

Take a crawl of 500,000 pages a month at a residential rate of $4 per gigabyte, and work down the stages. The page weights below are round numbers chosen for legibility; substitute your own from the byte log you started keeping in the previous section.

The second row alone is a 70 percent reduction, worth $2,800 a month or $33,600 a year, and it is roughly twenty lines of route-handling code. The fourth row is only reachable on targets where a plain HTTP request returns the content, so treat it as an upper bound rather than a plan.

One consequence deserves attention. Once a page costs 40 KB instead of 2 MB, bandwidth stops dominating your cost model and success rate takes over, because a failed attempt now wastes a retry slot rather than two megabytes. That is the point at which you should re-derive cost per successful request using the method in proxy success rate explained.

Rate cards move, and the per-gigabyte spread between vendors at the same tier is wide enough to matter once your volume is trimmed. Current numbers for every provider we track sit on the live benchmark table, the side-by-side comparison tool puts two rate cards next to each other, and our methodology documents how the price figures are normalised.