Mobile Proxy Web Scraping Guide
The end-to-end workflow for scraping behind a mobile proxy — connecting it, choosing rotation or sticky sessions per job, and diagnosing the blocks that still happen on a perfectly clean IP.
- One endpoint, any client — anything that speaks HTTP(S) or SOCKS5 works unchanged.
- Sticky or rotating — hold an IP for a logged-in job, rotate between independent ones.
- Unlimited bandwidth — no metering, so page weight never changes what a crawl costs.
- Real carrier IPs — requests leave from a mobile network, not a hosting range.
Rotate on demand at job boundaries instead of on a fixed timer.
Keep one exit IP for the whole of a logged-in flow, however long it runs.
Connecting a Mobile Proxy to Your Scraper
A mobile proxy is an ordinary forward proxy that happens to exit through a carrier network. Nothing in your scraper needs to know that. If your client can speak to an HTTP or SOCKS5 proxy — and every mainstream one can — it already supports mobile proxies, and the integration is a connection string rather than an SDK.
- Take the endpoint, port and protocol
You will be given a host, a port, and whether it speaks HTTP(S), SOCKS5, or both. SOCKS5 is the one to pick if you need UDP or non-HTTP traffic; otherwise the difference rarely matters for scraping.
- Choose an authentication model
Either allow-list the public IP your scraper runs from, so no credentials travel at all, or use a username and password. Allow-listing is the more robust choice for a fixed server, and it sidesteps an entire class of browser-automation problems covered later in this cluster.
- Assemble one proxy URL
Almost every client accepts the same shape: http://user:pass@host:port. Keep it in an environment variable rather than in source — it is a credential, and it will end up in a public repository otherwise.
- Verify the exit IP before trusting anything
Request an IP-echo service through the proxy and confirm the address, the country, and that the ASN belongs to a mobile network operator rather than a hosting provider. Skipping this check is how people spend a day debugging blocks that were really a misconfigured proxy.
If the echo service returns your own address, the proxy is not being used at all. Most libraries fail open — a malformed proxy setting is silently ignored and the request goes out directly. Always confirm the exit before you conclude anything about a target.
Once the connection is proven, everything after this point is about behaviour rather than plumbing. The most consequential decision is the next one.
Rotation vs Sticky Sessions: Choosing Per Job
Rotation is not automatically better than a stable IP — they solve opposite problems, and choosing wrong is a common cause of blocks that look inexplicable.
| Pattern | Use it for | It breaks when |
|---|---|---|
| Rotate per request | Anonymous, unauthenticated crawling of many independent URLs | The target uses a session cookie or a cart — the state is tied to the IP that created it |
| Sticky session | Anything behind a login, multi-step flows, paginated results that carry state | You hold one IP so long, and hit it so hard, that the rate itself is the signal |
| Rotate on failure only | Long crawls where most requests succeed and you want to shed a soured IP | You rotate on a 429 and retry immediately — the new IP inherits the same rate problem |
The practical rule: rotate between logical jobs, never in the middle of one. Changing exit IP mid-session on an authenticated site looks exactly like session hijacking, which is a worse signal than the scraping was.
This is also why a fixed rotation timer is a poor default. A timer knows nothing about your job boundaries, so sooner or later it fires between two requests that had to share an address. Rotate on demand when a job completes, and rotate on failure so a soured IP is shed promptly. If you must use an interval, make it longer than your slowest single job.
A 429 deserves its own treatment. It is the target telling you the rate is wrong, and rotating IP to fire again treats a rate problem as an identity problem — which is how a crawl escalates from throttled to blocked. Honour Retry-After, back off exponentially, and add jitter if you are running concurrent workers. The Python guide in this cluster has the retry configuration in code.
Why Scrapers Still Get Blocked on Clean IPs
A good IP is necessary and not sufficient. Anti-bot systems score several independent layers, and the proxy only addresses one of them. If you are still being blocked on clean mobile IPs, the answer is in this list:
- TLS fingerprint (JA3/JA4) — Python’s TLS handshake does not look like Chrome’s — different cipher order, different extensions. A request claiming to be Chrome that handshakes like urllib3 is contradicting itself before a single header is read. Libraries such as curl_cffi impersonate a real browser’s TLS profile.
- HTTP/2 and header order — Browsers send a specific set of headers in a specific order with specific pseudo-header framing. A minimal three-header request is not a browser, whatever the User-Agent says.
- Headless leakage — navigator.webdriver, missing plugin and codec surfaces, and a WebGL renderer that reads as SwiftShader all mark an automated browser. Run headful under a virtual display, or use a stealth plugin, and verify against a fingerprint-testing page rather than assuming.
- Geo inconsistency — This one is squarely the proxy’s business: a UK exit IP paired with a browser reporting America/New_York and en-US is a mismatch checked routinely. Set locale and timezone to match the country you are exiting from.
- A mobile UA on a non-mobile IP — Claiming to be an Android phone from a datacenter range is a contradiction any fingerprinting layer can see. If you present a mobile User-Agent and a mobile viewport, the exit IP needs to be a mobile one — which is the whole argument for this proxy type.
- Behaviour — Perfectly even intervals, no mouse movement, instantaneous form fills, and a request rate no person could sustain are all measured. No proxy hides an access pattern; only changing the pattern does.
Before adding another evasion layer, check the boring explanations: robots.txt may already tell you what is off limits, the site may publish an API that returns the same data as clean JSON, and the page you are parsing may be fetching from an internal endpoint you could call directly. An hour in the network tab regularly replaces a week of anti-bot work.
Rate Limits, robots.txt and Staying on the Right Side
Collecting publicly available data is generally lawful, and a proxy does not change that analysis in either direction. What changes it is everything alongside the collection.
The case worth knowing is hiQ Labs v. LinkedIn. The Ninth Circuit held in 2022 that scraping publicly accessible data does not violate the Computer Fraud and Abuse Act — genuinely significant, and usually where the summary stops. It should not. On remand LinkedIn won on breach of contract: hiQ accepted a 500,000 dollar judgment, a permanent injunction requiring it to stop scraping and destroy the collected data and source code, and the company shut down. Not a criminal computer-intrusion offence, and still fatal.
The lesson is that the access question and the contract question are separate. Public scraping is not hacking; a contract you accepted is still enforceable. If you created an account, clicked through terms, or used an API key, you are in contract territory regardless of how public the data looks.
- robots.txt — Not legally binding in most jurisdictions, but reading it is free and ignoring it is evidence of bad faith. Under recent European guidance it is also treated as a machine-readable rights-reservation signal, so it carries more weight now than it did a few years ago.
- Personal data — Data being public does not put it outside GDPR or CCPA. If what you are collecting identifies people, you need a lawful basis for it, and that obligation is entirely independent of how you fetched it.
- Rate — Degrading the service you are collecting from is the fastest way to turn a technical annoyance into a legal one. Concurrency belongs per target host, not globally: eight parallel requests spread over eight domains is ordinary, eight against one is a burst.
- Terms of service — The real risk surface for most projects. Read what you agreed to, and be honest about whether an account was involved.
None of this is legal advice, and jurisdictions differ. It is the shape of the questions worth asking before a crawl rather than after one.
Get a Dedicated Scraping Proxy
Live PXM2 locations — pick the country your target should see the request coming from, and get a dedicated 4G/5G IP with unlimited bandwidth and rotations:
法国
印度
新加坡
常见问题解答
How do I use a mobile proxy for web scraping?
Point your client at the proxy endpoint and authenticate, either by allow-listing your server’s IP or with a username and password. Every mainstream client takes a proxy the same way — a URL of the form http://user:pass@host:port. Then verify the exit IP before you trust anything: request an IP-echo service through the proxy and confirm the address and country are what you expect.
Should I rotate proxies on every request?
Only for anonymous crawling of independent URLs. The moment a session cookie, a cart, a paginated result set or a login is involved, per-request rotation breaks the job — that state is bound to the IP that created it. The general rule is to rotate between logical jobs and never in the middle of one.
Why am I still getting blocked when using a proxy?
Because the IP is one signal among several. Anti-bot systems also score your TLS handshake, your HTTP/2 header order, whether the browser leaks automation flags, and whether your claimed locale and timezone agree with the exit country. A clean mobile IP paired with a default headless fingerprint is still an obvious bot.
What is a sticky session and when do I need one?
A sticky session holds the same exit IP for as long as you need it rather than rotating underneath you. You need one for anything authenticated, any multi-step flow, and any paginated crawl that carries state. Changing exit IP mid-session on a logged-in site looks like session hijacking, which is a worse signal than the scraping was.
How often should a scraping proxy rotate?
Tie rotation to job boundaries rather than to a clock. A fixed timer will eventually fire in the middle of a multi-step request and break it. If you do want a time-based rule, make the interval longer than your slowest single job, and rotate on failure as well so a soured IP is shed promptly.
相关移动代理指南
This guide is the tool-agnostic overview. The rest of the cluster goes deeper on choosing a proxy tier and on the code for specific stacks.
Web scraping guides
核心移动代理指南
Scrape From a Real Carrier IP
Dedicated 4G/5G modems with unlimited bandwidth and unlimited rotations — rotate at your job boundaries, or hold one IP for as long as a session needs.
Get a Scraping Proxy