Web Scraping Best Practices
The discipline that keeps a scraper alive past week one: rate budgets set per host, fetching less rather than harder, retries that do not compound, and the one metric that warns you before your dataset quietly goes empty.
- Unlimited bandwidth — caching and re-crawls do not change what the month costs.
- Rotation on demand — shed a soured IP at a job boundary instead of on a timer.
- Dedicated hardware — your block rate reflects your behaviour, not a stranger’s.
- Sticky sessions — long jobs finish on the address that started them.
Per-modem billing, so a re-crawl or a cache miss costs nothing extra.
Rotate on demand when block rate climbs, at a boundary you choose.
Getting a scraper working is the easy part. Keeping it working — across a growing target list, past the first rate limit, without a bill that scales faster than the data — is a different discipline, and it is mostly about restraint. This page is the operational half of the cluster; connecting the proxy itself is covered in the mobile proxy web scraping guide.
Rate and Concurrency Budgets That Hold Up
The single most useful change most crawlers can make is to budget per target host rather than globally. Eight parallel requests spread over eight domains is ordinary traffic; eight against one domain is a burst, and the target only ever experiences the second number.
| Control | Sensible starting point | How to tune it |
|---|---|---|
| Concurrent requests per host | 2 to 5 | Raise only while the error rate stays flat; back off the moment it moves |
| Total worker pool | 10 to 20 across many hosts | Grow while throughput keeps climbing; stop when it flattens |
| Delay to the same domain | A few seconds; 10 is genuinely polite | If robots.txt sets Crawl-delay, that is the answer — use it |
| Adaptive throttle | On, from day one | Let observed latency drive the delay rather than a fixed constant |
Adaptivity matters more than any specific number above. A crawler that slows down when the target shows strain survives; one that holds a constant rate through rising latency is announcing that nothing on the other end is being observed.
Headers, Sessions and Fingerprint Consistency
Rate limits and caching answer how often you ask. The other half of not getting flagged is whether the request looks like it came from the client it claims to be — a clean IP does not fix a header set or a TLS handshake that belongs to no real browser.
Match the headers to the identity you claim
A real browser sends User-Agent, Accept, Accept-Language, Accept-Encoding and the Sec-Fetch-* headers in a fixed order, and its TLS handshake — cipher suite order and extensions, visible to the server as a JA3 or JA4 fingerprint — is specific to that browser version. A stock HTTP client such as requests, curl or node-fetch presents none of that; it announces itself as a script regardless of the User-Agent string set on top of it. Pick one real, current browser and OS combination, copy its actual header set and order, and hold that identity fixed rather than randomizing headers per request — a User-Agent that changes every fetch from one IP is its own signal.
Give a session one IP for its whole lifetime
A login, a cart, or a paginated result set is one session in the site’s eyes, and it should stay on one IP until that unit of work is done. Rotating mid-session while the cookie jar carries on unchanged puts one set of cookies arriving from two IPs within minutes — a clearer signal than either fact alone. Mobile proxies with sticky sessions exist for this: hold the session for as long as the task needs, and rotate on completion rather than on a fixed timer that cuts across it.
- User-Agent — One real browser and OS string, fixed for the life of the identity, not reshuffled per request
- Header set and order — Accept, Accept-Language and Sec-Fetch-* headers matching what that browser actually sends
- TLS fingerprint — A client library whose handshake matches a real browser, not the default stack’s own signature
- Cookies and session IP — Accepted and replayed on every request, on the same IP, for as long as the session runs
None of this replaces a real proxy — it decides whether the IP being paid for gets to look like what it is.
Fetch Less: Caching and Conditional Requests
Every request you do not make is one that cannot be blocked, cannot be rate limited, and cannot be wrong. Before tuning concurrency upward, take the cheaper win of fetching less:
- Cache what you already fetched — A local or shared cache turns a re-run into a diff rather than a full crawl. For distributed workers a central store keeps them from independently re-fetching the same pages.
- Use conditional requests — Send If-None-Match or If-Modified-Since where the server offers an ETag or Last-Modified. An unchanged page then costs a 304 instead of a full body, which is cheaper for both sides.
- Crawl incrementally — Re-walking an entire catalogue on a schedule is the classic way to earn a block for data that did not change. Prefer sitemaps, feeds, and change detection over exhaustive sweeps.
- Look for the endpoint behind the page — Rendered pages are often fetching from an internal JSON endpoint you could call directly — smaller, more stable, and easier to parse than the markup wrapped around it.
- Deduplicate before queueing — Canonicalise URLs and drop the ones already seen. Tracking parameters and session identifiers turn one page into dozens of apparently distinct fetches.
With unlimited-bandwidth proxies the argument for caching is not the transfer cost — it is that every avoided request is one fewer chance to be noticed.
Retries, Dead Letters and Adaptive Backoff
Retries are where a polite crawler quietly becomes an aggressive one. Three rules keep them honest:
- Honour Retry-After, then back off exponentially
When the server says how long to wait, that is the answer. Where it does not, grow the interval rather than repeating a fixed one — and add jitter so concurrent workers do not all wake together and re-create the burst.
- Cap attempts and use a dead-letter queue
A URL that has failed five times is telling you something a sixth attempt will not change. Park it for inspection instead of leaving it to circulate forever consuming budget.
- Keep retries idempotent
Retry reads freely; be careful with anything that changes state on the far side. Restricting automatic retries to GET and HEAD is the safe default.
Rotating the IP and retrying immediately is not a backoff strategy. A 429 is a statement about your rate, and answering it by changing identity spends addresses to buy a few more minutes at a pace the target already rejected.
Knowing You’re Blocked Before Your Data Does
Scrapers rarely fail loudly. They start returning challenge pages with a 200 status, or empty result sets that parse cleanly, and the pipeline reports success for days. Instrument for that specific failure:
| Metric | What it tells you | Act when |
|---|---|---|
| Block rate per target | The earliest honest signal you have | It rises at all — this moves before your dataset does |
| Records missing a required field | Catches soft blocks that return HTTP 200 | The share moves away from its normal baseline |
| 95th-percentile response time | Target strain, or a challenge being served | It climbs while throughput does not |
| Queue depth | Whether you are keeping up or accumulating | It grows steadily across a full run |
A useful cheap addition is a canary: one known-good URL with a known-good answer, fetched on a schedule. When the canary starts returning something unexpected you know it is you, not the parser, and you know it before the day’s data is written.
When the block rate does move, common scraping blocks and how to read them covers identifying what is refusing you before you start changing things.
Get a Dedicated Scraping Proxy
Live PXM2 locations — pick the country your target should see the request coming from, and get a dedicated 4G/5G IP with unlimited bandwidth and rotations:
France
India
Poland
Frequently Asked Questions
How many concurrent requests can I make when scraping?
Budget per target host, not globally. Two to five concurrent requests against one host is a defensible starting point, and a worker pool of ten to twenty across many hosts is ordinary. Raise the pool while throughput keeps climbing and the error rate stays flat; the moment errors rise without throughput following, you have found the ceiling.
How long should I wait between requests?
If robots.txt publishes a Crawl-delay, use it — that is the operator telling you the answer. Absent one, a delay of a few seconds to the same domain is conservative and rarely wrong, and ten seconds is genuinely polite for a small crawl. Delay matters far less than adaptivity: slowing down when the target signals stress is what keeps a crawl alive.
How do I scrape without getting banned?
Fetch less and behave consistently. Cache what you already have, use conditional requests so unchanged pages cost nothing, crawl incrementally instead of re-walking a whole catalogue, and keep per-host concurrency low. Most bans are earned by volume and rhythm rather than by any single request.
What should I monitor in a scraping pipeline?
Block rate per target above everything else — it moves before your data does. Then success rate, 95th-percentile response time, queue depth, and a data-quality check such as the share of records missing a required field. A scraper that silently returns challenge pages still reports HTTP 200 and an empty result set.
Is it better to rotate proxies or slow down?
Slow down first. Rotation answers an identity problem; a 429 is a rate problem, and rotating into a new IP to keep the same rate simply spends addresses to buy a few more minutes. Reduce concurrency, honour Retry-After, and rotate when an individual IP has genuinely soured.
Related Mobile Proxy Guides
This page is the operational discipline; the rest of the cluster covers the mechanics it applies to.
Web scraping guides
Core mobile proxy guides
Proxies That Do Not Punish a Re-Crawl
Dedicated 4G/5G modems billed per period rather than per gigabyte — caching, retries and incremental sweeps cost the same as a single pass.
Get a Scraping Proxy