Why You Keep Getting CAPTCHAs and How Proxy Choice Changes It

Bot management systems do not decide whether you are a robot. They estimate it, produce a number, and compare that number against a threshold the site operator configured. Above the line, the request passes silently. Below it, the site blocks outright. In the uncertain band between, the system buys more evidence by making you prove something, and that purchase is the CAPTCHA you keep seeing.

Understanding the challenge as a scoring outcome explains behaviour that otherwise looks arbitrary: why the challenge rate rises gradually rather than flipping on, why one exit passes and the next fails, and why solving the puzzle sometimes just serves you another one.

The score, the threshold and the action

Scoring systems in this space output a confidence value and leave the response policy to the site. Google reCAPTCHA v3 returns a score between 0.0 and 1.0 with no interaction at all, and the site decides what to do with it. Cloudflare and its peers work the same way: a managed challenge is triggered by policy, and the same visitor can be scored differently on two paths of the same domain.

Thresholds are tuned against a cost the operator cares about. Challenging real customers loses conversions, so a product listing might be permissive while the login form and the checkout are strict. This is why your crawler sails through category pages and hits a wall the moment it touches search or a price API.

What feeds the score

The inputs are correlated, which is what makes the system hard to defeat one variable at a time. Fixing the worst input often moves the score less than fixing the contradiction between two of them.

• Address classification. Usage type from commercial IP datasets, the ASN, and whether the address has been seen behind abuse reports or public proxy lists.
• Cross-site history of the exit. A large bot management vendor sits in front of a substantial share of the web and sees your exit IP across all of it. An address burned scraping one retailer arrives pre-scored at the next site using the same vendor.
• Network fingerprint. TLS and HTTP/2 characteristics, and whether they agree with the browser your user agent claims.
• Browser environment. Automation flags, WebGL renderer strings, font and codec availability, screen geometry, timezone and locale, and whether those agree with the exit country.
• Session evidence. Cookie age, whether you have a clearance token, and what the current session has already done on the site.
• Behaviour. Request pacing, navigation order, and on JavaScript-enabled pages the presence of pointer, scroll and keystroke telemetry.

Why the exit IP moves the number so much

The address is the only input available before your client does anything, and it carries history that you did not create. Residential pools are shared. Another customer of the same provider may have spent last week hammering the exact target you are now visiting, from the exact address you were just assigned, and the score reflects that.

Reputation decays, but slowly, and it decays by observation rather than by clock. An address recovers when it is seen behaving normally again, so a heavily reused exit that carries scraping traffic all day never gets the quiet period it needs. Provider behaviour differs sharply here: pools that churn and retire addresses regularly present fewer pre-burned exits than pools that recycle a small set. That difference shows up in fraud score and clean rate, which is why we sample both continuously. The measurement itself is explained in proxy fraud score explained, with current standings in the lowest fraud score rankings and the highest clean rate rankings.

Why one address passes here and fails there

Four things vary per site, and all four are outside your control. The bot management vendor differs, so the reputation data behind the score differs. The threshold differs, because it is tuned to the operator's tolerance for false positives. The action differs, since some sites challenge while others quietly degrade the response. And the baseline traffic differs.

That last one gets overlooked. A site whose customers are overwhelmingly German sees a German consumer ISP as unremarkable and a Brazilian mobile carrier as an outlier, while a global marketplace has no such baseline. Geographic targeting is therefore a scoring decision as much as a content decision: matching your exit country to the audience the site actually serves lowers the challenge rate on its own.

Cookies, clearance tokens and session memory

When you solve a challenge, the site sets a clearance cookie. On most systems that cookie is bound to the address and the fingerprint that solved it. Present it from a different exit and it is rejected, which produces the loop where every solved puzzle is followed by another one. Pinning the session fixes this. Buying a faster solver does not.

1. Assign one sticky exit per logical session and keep it for the session's whole lifetime, using the identifier scheme described in rotating vs sticky proxy sessions.
2. Store the cookie jar keyed to that session identifier, and never share a jar across exits.
3. Warm the session before you go deep: load the entry page, let the scripts run, accept the cookie banner, then request what you actually came for.
4. Retire the cookie jar and the exit together. Reusing a jar on a new address is worse than starting clean.

Where proxy choice stops helping

Be honest about the ceiling. An immaculate residential exit with a clean history, driving a headless browser that reports navigator.webdriver as true and renders through software WebGL, will be challenged. The address raises one input while the browser environment sinks two others, and the score reads all of them.

Volume is the other hard limit. No exit reputation survives a request rate that no household could produce. If a site's median visitor loads twenty pages and your session loads two thousand, the address is going to be scored on that regardless of where it is registered. Mobile pools are the partial exception, since carrier-grade NAT puts thousands of real subscribers behind one address and makes hard bans expensive for the target, which is discussed in mobile proxies explained.

When the address is clean and the challenges continue, the remaining work is on the client. TLS and HTTP/2 fingerprints, header consistency, automation flags and pacing are all covered in how to avoid getting blocked while web scraping.

Solving services against fixing the cause

Third-party solvers accept a challenge, return a token, and charge per solve. They work, within limits, and they have costs that a scraping pipeline feels quickly.

• Unit economics. A per-solve fee is orders of magnitude above the bandwidth cost of the page you were fetching, so a rising challenge rate turns into a rising bill with no ceiling.
• Latency. Solves take seconds to tens of seconds, and each one holds a worker and a concurrency slot for the duration.
• Token binding. A token minted for one IP and fingerprint is frequently rejected when presented from another, so the solver has to be wired into the same session that received the challenge.
• Signal loss. Automated solving hides the challenge rate, which is your best early warning that the pool or the client has degraded.

The defensible arrangement is a hard budget: fix the score inputs first, keep solving as a bounded fallback for the residual, cap spend per domain per day, and alert when the challenge rate crosses a threshold instead of quietly paying more. Check the terms of the sites you rely on before automating challenge responses at all, since some prohibit it explicitly.

Measure your challenge rate per exit

Everything above is testable. Record the outcome of every attempt against the exit that produced it, aggregate by subnet, and the pattern becomes visible within a few thousand requests. Get the exit address from whatever header your provider returns, or fetch an IP echo once per session and cache it.

from collections import Counter

CHALLENGE_MARKERS = (
    '/cdn-cgi/challenge-platform',
    'g-recaptcha',
    'h-captcha',
    'captcha-delivery.com',
    'unusual traffic',
)

seen = Counter()
challenged = Counter()


def record(exit_ip: str, status_code: int, body: str) -> None:
    """Call once per attempt, including the ones that succeeded."""
    subnet = '.'.join(exit_ip.split('.')[:3]) + '.0/24'
    seen[subnet] += 1

    head = body[:20000]
    if status_code in (403, 429) or any(m in head for m in CHALLENGE_MARKERS):
        challenged[subnet] += 1


def report(min_samples: int = 20) -> None:
    for subnet, total in seen.most_common():
        if total >= min_samples:
            print(subnet, round(challenged[subnet] / total, 3), total)

A flat rate across every subnet means the client is the problem, because the addresses are being treated identically. A rate concentrated in a handful of subnets means the pool is handing you a burned segment, and you should either change the targeting that keeps steering you there or change providers. Wire the same counters into the per-request telemetry described in the web scraping proxy setup guide.

One more reason to watch this number: challenge pages are billable traffic. Every interstitial you download is bytes across the residential meter, which is why detecting them in the first chunks of the response matters, as covered in reduce proxy bandwidth costs.

We run the reputation half of this measurement continuously across every provider we track, from US and EU hosts. Current fraud score and clean rate figures sit on the live benchmark table, historical movement is plotted on the performance graphs, and per-vendor detail lives on pages such as SOAX.