Mobile Proxy Python Scraping

requests and httpx: Sessions, Proxy Dicts and the CONNECT Tunnel

If the markup you want is already in the initial response, do not launch a browser. An HTTP client is an order of magnitude cheaper in memory and time. Use one Session so the connection pool and cookie jar are reused across requests rather than renegotiating TLS every call:

import os
import requests

PROXY = os.environ["PXM2_PROXY"]   # http://user:pass@host:port

with requests.Session() as s:
    s.proxies.update({"http": PROXY, "https": PROXY})
    s.headers.update({
        "User-Agent": "Mozilla/5.0 (Linux; Android 14) AppleWebKit/537.36 ...",
        "Accept-Language": "en-GB,en;q=0.9",
    })
    r = s.get("https://example.com/listings", timeout=30)
    r.raise_for_status()

The detail people get wrong is the https entry. It still uses an http:// scheme, because that scheme describes how to reach the proxy — not the protocol of the target. The client opens a CONNECT tunnel through the proxy and TLS then runs end to end inside that tunnel, so the proxy never sees your plaintext. Writing https:// there instead is the single most common reason a Python proxy appears not to work for HTTPS.

The second detail is consistency: a mobile User-Agent should be paired with a mobile IP. Claiming to be an Android phone from a datacenter range is a contradiction any fingerprinting layer can see.

httpx works the same way with a different keyword — pass proxy to the Client constructor. It is worth preferring when you want HTTP/2, since a browser-like protocol version is one fewer discrepancy between what your headers claim and how your client behaves.

Rotating Proxies in Python: Round-Robin, Random and Rotate-on-Failure

There are two quite different situations here, and conflating them causes most of the confusion. If you were given a single rotating endpoint, the rotation happens upstream and your code only decides when to ask for a fresh IP. If you hold a list of endpoints, you own the rotation policy yourself.

For the second case, assign one proxy per Session rather than per request. A Session carries the connection pool and the cookie jar, so swapping its proxy mid-flight defeats the point of having one:

import itertools
import requests

PROXIES = [
    "http://user:pass@host:10001",
    "http://user:pass@host:10002",
    "http://user:pass@host:10003",
]

def sessions_for(proxies):
    "One Session per exit IP, reused for the life of a job."
    out = []
    for p in proxies:
        s = requests.Session()
        s.proxies.update({"http": p, "https": p})
        out.append(s)
    return out

pool = itertools.cycle(sessions_for(PROXIES))

for job in jobs:
    session = next(pool)          # a whole job stays on one IP
    run(session, job)

Note what the loop does not do: it never changes IP inside run(). Rotate between logical jobs, never in the middle of one — an exit IP that changes during an authenticated flow looks like session hijacking, which is a worse signal than the scraping was.

Random selection is a reasonable alternative to round-robin when jobs are independent and you want to avoid a predictable cycle. Rotate-on-failure is the pattern worth adding to either: when a proxy starts returning blocks, drop it from the pool rather than continuing to feed it work.

Handling 429s: Retry-After, Exponential Backoff and Jitter

A 429 is the target telling you the rate is wrong. Rotating IP and firing again treats a rate problem as an identity problem, and it is how a crawl escalates from throttled to blocked. Honour Retry-After and back off exponentially:

from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

retry = Retry(
    total=5,
    backoff_factor=1.5,               # 1.5s, 3s, 6s, 12s, 24s
    status_forcelist=[429, 500, 502, 503, 504],
    respect_retry_after_header=True,
    allowed_methods=["GET", "HEAD"],
)
s.mount("https://", HTTPAdapter(max_retries=retry, pool_maxsize=20))

Add jitter on top if you are running concurrent workers, or they will all wake up and retry in the same instant — a thundering herd that looks far more mechanical than the original traffic did. And concurrency belongs per target host, not globally: eight parallel requests spread over eight domains is ordinary, while eight against one is a burst.

Scrapy and asyncio: Middleware, aiohttp and Concurrency Limits

Scrapy already ships proxy support. Its built-in HttpProxyMiddleware reads request.meta["proxy"], so setting that key in a spider or a small middleware is enough for a single endpoint. For rotation across a list, scrapy-rotating-proxies adds a pool and its own ban tracking:

ROTATING_PROXY_LIST = [
    "http://user:pass@host:10001",
    "http://user:pass@host:10002",
]
# or: ROTATING_PROXY_LIST_PATH = "proxies.txt"

DOWNLOADER_MIDDLEWARES = {
    "rotating_proxies.middlewares.RotatingProxyMiddleware": 610,
    "rotating_proxies.middlewares.BanDetectionMiddleware": 620,
}

CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 0.5
AUTOTHROTTLE_ENABLED = True

Its default ban heuristic treats a non-200 status, an empty body, or an exception as a dead proxy, which is blunt — a target that answers a block with a 200 and a challenge page will slip straight past it. Override ROTATING_PROXY_BAN_POLICY with a class that recognises what your specific target does when it blocks you.

The two settings at the bottom matter more than the middleware for staying unblocked. AutoThrottle adapts the delay to observed latency, and a per-domain concurrency cap keeps a wide crawl from turning into a burst against one host.

With aiohttp the proxy goes on the individual request rather than the session, which makes per-request rotation trivial and per-host politeness entirely your responsibility. Wrap your fetches in a semaphore keyed by target host — asyncio makes it very easy to turn a polite crawl into a denial-of-service you did not intend.

Get a Dedicated Python Scraping Proxy

Live PXM2 locations — pick the country your target should see the request coming from, and get a dedicated 4G/5G IP with unlimited bandwidth and rotations:

Frequently Asked Questions

Related Mobile Proxy Guides

If your target renders its data with JavaScript, the HTTP-client path above will not reach it — the browser guide covers that case.

Web scraping guides

Core mobile proxy guides