Common Scraping Blocks and How to Read Them
Identify what is actually blocking you before you change anything — the header that names the vendor, what each status code really means, and which causes a proxy can and cannot solve.
- Carrier-grade NAT trust — an address a target cannot ban without banning real subscribers.
- Rotation on demand — retire an address that has genuinely soured, when you decide.
- Country-matched exits — so a geo-block is not mistaken for a bot block.
- Dedicated hardware — the reputation on your IP is the one you built.
Carrier IPs shared with real subscribers, so blocking one is expensive.
If a clean mobile IP does not help, the problem was never the IP.
Most time lost to a blocked scraper is spent fixing the wrong thing. The status code alone rarely tells you what happened, and the instinct to rotate IPs and retry treats every symptom as the same disease. This page is a triage order: work out what is refusing you, then pick a fix that matches.
Identify What Is Blocking You
Before changing anything, read the response rather than the status line. The major anti-bot vendors identify themselves, usually without meaning to, and each one implies a different fix:
| Tell-tale | Who it is | What it implies |
|---|---|---|
| cf-ray header, or server: cloudflare | Cloudflare | Body often carries its own code — 1020 for a firewall rule, 1015 for rate limiting. A JavaScript interstitial means it wants a browser. |
| x-datadome header, branded challenge page | DataDome | Heavily fingerprint-driven. A clean IP alone rarely settles it; the client has to look like a browser. |
| _px3 cookie, or a reference to px-captcha | PerimeterX (now HUMAN) | Behavioural scoring is weighted heavily — pacing and interaction matter as much as the address. |
| No vendor marker, plain short body | The site’s own rules | Usually simpler: a User-Agent filter, a required header, or a rate rule you can respect. |
Dump the full response headers and the first few hundred bytes of the body on your first failure and keep them. Almost every question below is answered by that capture, and reproducing a block after you have started changing things is much harder than recording it once.
What Each Status Code Actually Means
Two responses that look equally like failure can have opposite causes. This is the mapping worth internalising:
| Response | Most likely cause | First thing to try |
|---|---|---|
| 403 immediately, tiny or empty body | Anti-bot refusal — fingerprint or headers, not permissions | Fix the client’s TLS profile and headers before touching the proxy |
| 403 after a run of successes | The address earned it — rate or volume | Lower concurrency, then rotate the soured IP |
| 429 with Retry-After | Honest rate limiting | Wait exactly that long. Do not rotate and retry immediately |
| 407 | Your proxy rejected you, not the target | Check credentials or whether this host is allow-listed |
| 503 with a challenge page | Interstitial waiting for JavaScript | A real browser, or an endpoint that returns data without one |
| 200 with no data — a soft block | Challenge or empty shell served as success | Assert on content, not status — this is the one that corrupts datasets silently |
| Redirect loop, or a consent or region wall | Geo or cookie state, not bot detection | Exit from the right country and carry cookies across the session |
The soft block deserves special attention because it is the only failure that does not look like one. A scraper that treats HTTP 200 as success will happily write thousands of empty records and report a green run. Validate that a known field is present before you accept a page.
A Triage Order That Saves a Day
Three questions, in this order. Each one eliminates a whole class of cause, so you change one thing at a time instead of shuffling four:
- Did it fail on the very first request?
Then no reputation has been built yet, and the address is almost certainly innocent. Look at headers, User-Agent and TLS profile. Rotating IPs here changes nothing and wastes them.
- Or after a run of successes?
Then the client shape was fine and something you did earned it. Rate, volume, or pattern. Reduce per-host concurrency first; rotate only once you accept the address itself is spent.
- Does it work in a browser on the same connection?
Then the network path is fine and your client is the difference. Request a fingerprint-echo service such as tls.peet.ws through your client and compare its JA3, JA4 and HTTP/2 fingerprint against a real browser’s. A mismatch here is the whole answer.
Change one variable at a time and keep the failing request as a fixture. The usual failure mode is swapping the proxy, the User-Agent and the client library at once, seeing it work, and never learning which one mattered — so the next target starts from zero.
Fixes That Match the Cause
Sorted by what the evidence pointed at, and honest about which of these a proxy actually addresses:
- IP reputation, rate limits, geo walls — This is genuinely the proxy’s job. A carrier IP behind CGNAT is expensive for a target to ban because thousands of real subscribers sit behind it, and exiting from the right country removes geo-blocks that look like bot blocks.
- TLS or HTTP/2 fingerprint — Not a proxy problem at all. Use a client that impersonates a real browser’s handshake, such as curl_cffi, or drive an actual browser. No address will make urllib3 handshake like Chrome.
- Headless leakage — navigator.webdriver, missing plugin surfaces and a SwiftShader WebGL renderer mark an automated browser regardless of where it exits from.
- Contradictions — A UK exit paired with an America/New_York timezone, or a mobile User-Agent on a datacenter range. Consistency across every layer matters more than the strength of any one of them.
- Pattern — Perfectly even intervals and a rate no person sustains are measured directly. No proxy hides an access pattern; only changing the pattern does.
The useful thing about testing on a clean carrier IP is what it rules out. If a fresh mobile address behaves exactly like the old one, the cause was never the address, and you have just eliminated the most expensive thing on the list. The pillar guide covers the fingerprint layers in more depth, and the best-practices guide covers the rate discipline that stops most of these before they start.
Test Against a Clean Carrier IP
Live PXM2 locations — pick the country your target should see the request coming from, and get a dedicated 4G/5G IP with unlimited bandwidth and rotations:
France
India
Singapore
Frequently Asked Questions
Why am I getting a 403 when scraping?
A bare 403 arriving immediately, with no body or a tiny JSON error, is almost always an anti-bot decision rather than a permissions one. If it happens on the very first request, suspect your headers and TLS fingerprint. If it starts after a run of successes, suspect your rate. The two have completely different fixes, and guessing wrong costs a day.
How do I tell which anti-bot system is blocking me?
Read the response rather than the status line. Cloudflare leaves a cf-ray header and a cloudflare server header, and its own error codes such as 1020 and 1015 appear in the body. DataDome sets x-datadome headers and a branded challenge page. PerimeterX, now HUMAN, sets a _px3 cookie and references px-captcha. Each one implies a different fix.
What is the difference between a 403 and a 429?
A 429 with a Retry-After header is honest rate limiting: the target is telling you the pace is wrong and when to come back. Honour it. A 403 is a refusal to serve you at all, and rotating IPs at it without changing anything else usually just burns addresses.
Why does the page load in my browser but not in Python?
Because your client does not handshake like a browser. The cipher order, TLS extension order and HTTP/2 settings combine into a JA3 or JA4 fingerprint that is compared against known-browser values. You can confirm it by requesting a fingerprint-echo service such as tls.peet.ws through your client and comparing. The fix is a client that impersonates a real TLS profile, or a real browser — not a different proxy.
Can a mobile proxy fix every scraping block?
No, and it is worth being clear about that. A proxy fixes IP reputation, rate limits attached to an address, and geo-blocks. It does nothing for a TLS fingerprint mismatch, a headless browser announcing itself, missing or contradictory headers, or an access pattern no person could produce. If a clean mobile IP does not help, the cause is on that list instead.
Related Mobile Proxy Guides
Once you know what was blocking you, the rest of the cluster covers the fix in detail.
Web scraping guides
Core mobile proxy guides
Rule Out the IP in Five Minutes
Dedicated 4G/5G modems on real carrier ranges — test your target from a clean address and find out whether the problem was ever the proxy.
Try a Clean Carrier IP