Proxies for Web Scraping: A Practical Setup Guide

A scraper with proxies bolted on late behaves badly in specific, predictable ways: retries that stack on the same broken exit, sessions that rotate in the middle of a paginated flow, and a bandwidth bill nobody can attribute to a domain. All of it comes from configuration decisions made once, early, and never revisited.

This guide covers the wiring. Which pool to point at which host, how to express credentials and targeting, what a retry policy should do, how connection reuse interacts with rotation, and what to record so the system is debuggable. Detection surfaces are a separate subject, handled in how to avoid getting blocked while web scraping.

Choose the pool per target, not per project

Most crawls touch a mix of hosts, and paying residential rates for all of them is the single most common overspend in this field. Route by measured behaviour of each host rather than by a global setting.

• Datacenter for public APIs, documentation, sitemaps, RSS, and any host that answers a hundred test requests from a hosting IP with real content.
• Residential for consumer web targets that reject hosting IPs on the first request, which you confirm by comparing bodies rather than status codes.
• ISP (static residential) when one address has to persist for hours, such as long sessions or ad verification.
• Mobile only where a carrier IP is the entry condition, because the price per gigabyte is the highest of the four.

Store the decision per domain in your scheduler so it survives restarts and can be revised when a host changes its posture. The cost model behind that choice is worked through in residential vs datacenter proxies.

Wiring the gateway into your client

httpx and requests

Both accept a proxy URL with credentials in the userinfo section. The important part is what surrounds it: an explicit timeout for every phase, a bounded connection pool, and a client object that lives longer than one request.

import random
import time
import httpx

GATEWAY = 'gateway.example.net:7000'
USERNAME = 'customer'
PASSWORD = 'SECRET'

RETRY_ON_NEW_EXIT = {403, 408, 429, 503}   # the exit looks like the problem
RETRY_ON_SAME_EXIT = {425, 500, 502, 504}  # the target looks like the problem
# 400, 401, 404, 410 and 451 are terminal. 407 means your credentials are wrong.

_clients: dict[str, httpx.Client] = {}


def new_session() -> str:
    return str(random.getrandbits(32))


def client_for(session_id: str, country: str = 'us') -> httpx.Client:
    """One pooled client per exit, so keep-alive survives between requests."""
    if session_id not in _clients:
        user = USERNAME + '-country-' + country + '-session-' + session_id
        _clients[session_id] = httpx.Client(
            proxy='http://' + user + ':' + PASSWORD + '@' + GATEWAY,
            timeout=httpx.Timeout(connect=10.0, read=25.0, write=10.0, pool=5.0),
            limits=httpx.Limits(max_connections=4, max_keepalive_connections=4),
            follow_redirects=True,
            max_redirects=3,
        )
    return _clients[session_id]


def retire(session_id: str) -> None:
    client = _clients.pop(session_id, None)
    if client is not None:
        client.close()


def fetch(url: str, attempts: int = 4) -> httpx.Response:
    session_id = new_session()
    for attempt in range(attempts):
        try:
            response = client_for(session_id).get(url)
        except (httpx.ConnectError, httpx.ReadTimeout, httpx.RemoteProtocolError):
            retire(session_id)
            session_id = new_session()
            time.sleep(2 ** attempt + random.random())
            continue

        if response.status_code in RETRY_ON_NEW_EXIT:
            retire(session_id)
            session_id = new_session()
            time.sleep(2 ** attempt + random.random())
            continue

        if response.status_code in RETRY_ON_SAME_EXIT:
            time.sleep(2 ** attempt + random.random())
            continue

        return response

    raise RuntimeError('gave up on ' + url)

Scrapy

Scrapy reads request.meta["proxy"], and its built-in HttpProxyMiddleware sits at priority 750. Anything that assigns a proxy has to run before that. Set the credentials yourself as a Proxy-Authorization header so you control exactly which session identifier each request carries.

# middlewares.py
import base64
from itertools import count

GATEWAY = 'http://gateway.example.net:7000'
USERNAME = 'customer'
PASSWORD = 'SECRET'
REQUESTS_PER_SESSION = 25


class GatewaySessionMiddleware:
    def __init__(self):
        self._served = count()

    def process_request(self, request, spider):
        if 'proxy' in request.meta:
            return
        session = request.meta.get('proxy_session')
        if session is None:
            session = next(self._served) // REQUESTS_PER_SESSION

        user = USERNAME + '-country-us-session-' + str(session)
        token = base64.b64encode((user + ':' + PASSWORD).encode()).decode()
        request.meta['proxy'] = GATEWAY
        request.headers['Proxy-Authorization'] = 'Basic ' + token


# settings.py
# DOWNLOADER_MIDDLEWARES = {'myproject.middlewares.GatewaySessionMiddleware': 350}
# CONCURRENT_REQUESTS_PER_DOMAIN = 8
# DOWNLOAD_TIMEOUT = 25
# RETRY_TIMES = 2
# RETRY_HTTP_CODES = [429, 500, 502, 503, 504]
# AUTOTHROTTLE_ENABLED = True

Pass meta={"proxy_session": order_id} from the spider whenever a group of requests has to share one exit, such as a paginated listing or a flow that depends on a cookie set by the first page.

Playwright

Set the proxy on the context, not the page: browser.new_context(proxy={"server": "http://gateway.example.net:7000", "username": "customer-country-us-session-42", "password": "SECRET"}). One context per session identifier gives you a clean cookie jar and a stable exit at the same time. Contexts are cheap; browsers are not, so reuse one browser process across many contexts. On older Chromium builds you may also need a placeholder proxy argument at launch for context-level proxies to take effect.

Where targeting and session identity live

Almost every residential provider encodes options in the username rather than exposing an API. One hostname and port serve the whole network, and a username such as customer-country-de-city-berlin-session-a91f selects a German exit in Berlin and pins it for the provider's sticky window. Separators and keyword names differ per vendor, so read their docs before assuming the format.

• Country and city narrow the exit pool geographically. Deep targeting shrinks the pool, which raises the reuse rate of each address.
• Session identifier pins an exit for a fixed window, usually between one and thirty minutes. Changing the string is how you rotate on demand.
• ASN or carrier is offered by some providers when you need a specific network rather than a country.
• Protocol and port often select behaviour: one port for rotating, another for sticky, and sometimes a separate one for SOCKS5.

Generate session identifiers from your own scheduler so they are reproducible in a log. Random strings are fine; sequential counters are better, because you can replay the exact exit assignment when a job goes wrong. Geographic side effects of deep targeting are covered in the proxy geotargeting guide.

Retry policy that does not multiply the bill

Every retry transfers bytes again. A three-attempt policy applied blindly triples the cost of your worst-performing domains, which are also the domains generating the most retries. Classify first, then decide.

Back off exponentially with jitter, and cap total attempts per URL rather than per error type, so one pathological page cannot consume a whole worker. Requeue with delay instead of sleeping in the worker whenever your architecture allows it, because a sleeping worker holds a concurrency slot for nothing.

Rotate on failure or rotate on a clock

Two rotation strategies exist and they solve different problems. Rotating on failure keeps a working exit for as long as it works, which minimises handshakes and keeps sessions coherent. Rotating on a schedule, say every N requests or every T seconds, spreads load evenly and stops any single address accumulating a suspicious request history.

Run both. Rotate on any block-shaped failure, and rotate anyway after a fixed number of requests even when nothing has gone wrong. The fixed number is target-specific: sites with aggressive per-IP counters want small values, tolerant sites are happy with hundreds. The session-length decision tree lives in rotating vs sticky proxy sessions.

Concurrency, keep-alive and connection reuse

A residential exit is one household device sharing an asymmetric uplink. Providers usually cap in-flight requests per exit to a small number, and exceeding it produces timeouts that look like target-side failures. Set max_connections per session client to a handful and scale throughput by running more sessions, not by pushing harder on one.

That trade-off is real money. Each new tunnel costs a TCP handshake plus a TLS handshake through the proxy, which is several round trips at residential latency. Reusing a connection for twenty requests amortises that cost; reusing it for two thousand makes the exit look like a crawler. Pick a number, log it, and adjust it based on the block rate you measure.

Where your DNS lookups happen

With an HTTP proxy this is settled for you. The client sends CONNECT example.com:443 and the resolution happens at the proxy side, so the target hostname resolves near the exit. With SOCKS5 it depends on the scheme you write. In curl and requests, socks5:// resolves the hostname locally and socks5h:// hands it to the proxy. The difference matters.

Local resolution means a CDN picks an edge close to your server rather than close to the exit. You then present a residential IP in one country while fetching from an edge node in another, which is an inconsistency worth avoiding, and it also exposes your crawl targets to your own resolver logs. Use socks5h:// unless you have a specific reason not to. The protocol differences are covered in HTTP vs SOCKS5 proxies.

Instrument every request

Without per-request telemetry you cannot tell a bad provider from a bad selector. Emit one structured record per attempt, including the failures, and keep it long enough to compare weeks.

1. Identity of the attempt: timestamp, target host, URL pattern, attempt number, provider, session identifier, and the exit IP if the provider returns it in a response header.
2. Outcome: status code, whether a known content selector was present in the body, and the block signature matched if any.
3. Cost: bytes transferred, which is what you are billed for and what feeds the techniques in reduce proxy bandwidth costs.
4. Timing: connect time and total time separately, so you can tell a slow exit from a slow target.

From those four fields you can derive everything worth knowing: success rate per provider and per domain, p95 latency, cost per successful extraction, and the distribution of failures across exits. A body-content check rather than a status check is what makes the success number honest, as described in proxy success rate explained.

We run the same instrumentation continuously against every provider we track, from US and EU hosts, and publish the output on the live benchmark table with the procedure documented in our methodology. Sort by consistency in the most reliable provider rankings, and read individual vendor pages such as Oxylabs for the per-provider history. When your failures turn into challenge pages rather than error codes, continue with why you keep getting CAPTCHAs.