How to Avoid Getting Blocked While Web Scraping

Blocks are not random. Something specific about your traffic was measured and scored, and the response you got tells you which layer did the measuring. Teams lose weeks because they skip that step: they read an article, add ten headers and a stealth plugin, and the block rate stays exactly where it was because the target was filtering on ASN the whole time.

Anti-bot vendors evaluate in cost order. A registry lookup on the source IP costs a memory read. A TLS fingerprint comparison costs almost nothing, since the handshake already happened. A JavaScript challenge costs real compute and a page load, so it is gated behind the cheap checks. Your fix has to target the layer that actually caught you.

Read the signal before you change anything

Capture a failing request in full, including the response headers, before you touch the code. The vendor usually identifies itself, and the body length alone separates a challenge page from a real one.

# Capture the whole response, headers included
curl -sS -o /tmp/body.html -D /tmp/head.txt \
     -w 'status=%{http_code} bytes=%{size_download} time=%{time_total}\n' \
     -x http://USER:[email protected]:7000 \
     -H 'Accept-Language: en-US,en;q=0.9' \
     'https://example.com/search?q=widgets'

# Who answered, and did they set a challenge cookie?
grep -iE 'cf-ray|x-datadome|akamai|server:|set-cookie' /tmp/head.txt

# Does the body contain a challenge rather than content?
grep -icE 'captcha|challenge-platform|access denied|unusual traffic' /tmp/body.html

Record the classification for each attempt in your telemetry so you can see rates rather than anecdotes. A block rate that jumps from 2 percent to 60 percent overnight is a target policy change; one that creeps up over a week is your own pool degrading. Instrumentation for this is in the web scraping proxy setup guide.

IP reputation and the ASN behind your exit

The first check is where your packet came from. Commercial datasets map every address to an autonomous system and a usage type, and a hosting classification is the loudest signal available. No header manipulation survives it, because the decision happens before your headers are parsed.

Past the usage type sits reputation, which is memory of what that address did recently. Shared abuse feeds, proxy lists, and the target's own history of the address all contribute. Residential addresses are not automatically clean: an IP that was part of a botnet last month carries that history into your crawl. This is what fraud score measures, and it is why we sample it continuously across every pool. The mechanics are in proxy fraud score explained, and current standings are sortable in the lowest fraud score rankings and the highest clean rate rankings.

One trap deserves naming. Deep geographic targeting shrinks the reachable pool, sometimes to a few hundred exits in a small country. Your requests then concentrate on a tiny set of addresses, each one accumulating a request history far faster than it would in a wide pool. If your block rate rose the day you added city-level targeting, that is the cause.

TLS handshakes that contradict your user agent

Before a single HTTP byte moves, your client sends a ClientHello containing its cipher suite list, supported curves, signature algorithms, extension list and ALPN preferences. The exact contents and their order are determined by the TLS library, and they differ sharply between OpenSSL, BoringSSL, NSS and Go's crypto stack. Hashing that structure gives a JA3 or JA4 fingerprint.

Python requests on OpenSSL emits a ClientHello that Chrome has never produced. Send it with a Chrome user agent and you have declared two incompatible identities in the same connection, which is a free and unambiguous detection. Chrome also inserts GREASE values at randomised positions, which most scripted clients omit entirely.

The fix is a client that reproduces a real browser handshake rather than a patched user agent string.

from curl_cffi import requests

PROXY = 'http://customer-country-us-session-9f2a:[email protected]:7000'

# impersonate replays a real browser ClientHello and HTTP/2 settings,
# so the handshake matches the User-Agent the library sends with it.
session = requests.Session(
    impersonate='chrome124',
    proxies={'http': PROXY, 'https': PROXY},
    timeout=25,
)

response = session.get('https://example.com/listing')
print(response.status_code, len(response.text))

Pin the impersonation target to a build your installed version actually ships, and update it when you update the library, because a fingerprint from a browser version that no longer exists in the wild is its own anomaly.

HTTP/2 settings and header ordering

HTTP/2 gives away more than HTTP/1.1 did. The SETTINGS frame a client sends at connection start carries specific values for header table size, initial window size and max concurrent streams, and each browser engine uses its own. Window update sizes, whether priority frames are sent, and the order of the pseudo-headers all vary too: Chrome orders them :method, :authority, :scheme, :path, while Firefox uses a different sequence. Together these form a fingerprint that is independent of TLS and just as cheap to check.

Two practical consequences. If your client negotiates HTTP/1.1 against a host that offers HTTP/2 to every real browser, that alone is unusual. And in HTTP/1.1, header order and capitalisation are preserved on the wire, so a client that sends its headers alphabetically or normalises them to lowercase looks nothing like a browser. Libraries that let you control header order are worth choosing for that reason alone.

The headers a browser sends and a script forgets

Missing headers are easy to spot. Inconsistent ones are worse, because they prove the sender is constructing headers rather than emitting them.

• Client hints must match the user agent. A sec-ch-ua value claiming Chrome 124 alongside a user agent string for Chrome 110 is a direct contradiction.
• Fetch metadata is generated by the browser. Sec-Fetch-Site, Sec-Fetch-Mode, Sec-Fetch-Dest and Sec-Fetch-User describe how the request was initiated, and their values differ between a typed URL, a link click and an XHR. Copying one set onto every request is detectable.
• Accept-Language should match your exit geography. A German residential IP sending en-US only is a mismatch that costs nothing to notice.
• Referer should reflect a real path. Requesting a deep product page with no referer, repeatedly, describes a crawler working from a URL list.
• Accept-Encoding should list what you can decode. Advertising brotli without a decoder installed produces broken bodies and an odd traffic pattern.

What a headless browser reveals about itself

Running a real browser fixes the network fingerprints and opens a new surface. navigator.webdriver is set to true by default under automation. Software rendering exposes WebGL vendor and renderer strings such as SwiftShader or llvmpipe, which no consumer GPU reports. Headless builds ship without proprietary media codecs, so codec support probes come back wrong. Screen dimensions, device pixel ratio, available fonts, timezone and reported hardware concurrency all get read and compared against each other.

Launch with --disable-blink-features=AutomationControlled, run the browser headed against a virtual display where you can afford it, and make sure the timezone and locale you report match the country of your exit IP. Resist stacking every stealth patch you can find. A surface that is internally inconsistent, claiming a mobile device with a desktop screen size, scores worse than an unpatched one.

Timing and navigation that no person produces

Behaviour is where the expensive detection lives, and regularity is the giveaway. Requests spaced at exactly 500 milliseconds, a constant rate across 14 hours, no diurnal variation, and page IDs fetched in ascending numeric order are all patterns a human session never contains.

• Jitter your intervals from a distribution rather than adding a fixed sleep. Log-normal is closer to human pacing than uniform.
• Cap requests per exit on a rolling window instead of relying on rotation to spread the load evenly.
• Arrive the way a visitor does. Reach deep pages through the listing that links to them where the crawl structure allows it.
• Vary the order. Shuffle your URL queue so consecutive requests are not adjacent in the site hierarchy.
• Stop overnight if the target has an obvious regional audience and your traffic would otherwise be the only activity at 4am local time.

Honeypots and links that only a parser can see

Some sites plant links that a rendered browser never displays: anchors inside a display: none container, elements positioned off-screen, zero-size images, or form fields hidden with CSS. A human cannot click them. A naive HTML parser follows every href it finds, and requesting one is a self-identification with no plausible innocent explanation.

Filter your link extraction. Skip anchors inside hidden containers, ignore inline styles that remove an element from the layout, and never submit a form field that was not visible. Paths listed under Disallow in robots.txt are frequently bait as well as policy, which gives you two reasons to leave them alone.

Rate limits and robots.txt as engineering constraints

Treating the published rules as free information is the pragmatic position, quite apart from the ethical one. robots.txt tells you which paths the operator considers off limits and often includes a Crawl-delay. A Retry-After header tells you exactly how long to wait. Sitemaps hand you a canonical URL list with modification timestamps, which is cheaper to consume than discovery crawling and reduces your request count outright.

Set a defensible baseline: collect public data only, identify your crawler in the user agent with a contact URL when you are not required to blend in, cache aggressively so you never fetch the same resource twice, use conditional requests on recrawls, and stay well under any published limit. Read the terms of service for the sites you depend on. A crawler that behaves this way generates fewer blocks and costs less to run, which is covered from the billing side in reduce proxy bandwidth costs.

When you have fixed the fingerprints and the pacing and the blocks persist, the remaining variable is the pool itself. Compare your own success rate against a controlled baseline using the method in proxy success rate explained, check whether the target is filtering on usage type at all using residential vs datacenter proxies, and see what a clean pool looks like on the live benchmark table or on individual vendor pages such as NodeMaven. Our sampling procedure is documented in the methodology. If the failures arrive as challenges rather than error codes, the risk-scoring side is covered in why you keep getting CAPTCHAs.