Web Scraping Best Practices
The discipline that keeps a scraper alive past week one: rate budgets set per host, fetching less rather than harder, retries that do not compound, and the one metric that warns you before your dataset quietly goes empty.
- Unlimited bandwidth — caching and re-crawls do not change what the month costs.
- Rotation on demand — shed a soured IP at a job boundary instead of on a timer.
- Dedicated hardware — your block rate reflects your behaviour, not a stranger’s.
- Sticky sessions — long jobs finish on the address that started them.
Per-modem billing, so a re-crawl or a cache miss costs nothing extra.
Rotate on demand when block rate climbs, at a boundary you choose.
Getting a scraper working is the easy part. Keeping it working — across a growing target list, past the first rate limit, without a bill that scales faster than the data — is a different discipline, and it is mostly about restraint. This page is the operational half of the cluster; connecting the proxy itself is covered in the pillar guide.
Rate and Concurrency Budgets That Hold Up
The single most useful change most crawlers can make is to budget per target host rather than globally. Eight parallel requests spread over eight domains is ordinary traffic; eight against one domain is a burst, and the target only ever experiences the second number.
| Control | Sensible starting point | How to tune it |
|---|---|---|
| Concurrent requests per host | 2 to 5 | Raise only while the error rate stays flat; back off the moment it moves |
| Total worker pool | 10 to 20 across many hosts | Grow while throughput keeps climbing; stop when it flattens |
| Delay to the same domain | A few seconds; 10 is genuinely polite | If robots.txt sets Crawl-delay, that is the answer — use it |
| Adaptive throttle | On, from day one | Let observed latency drive the delay rather than a fixed constant |
Adaptivity matters more than any specific number above. A crawler that slows down when the target shows strain survives; one that holds a constant rate through rising latency is announcing that nothing on the other end is being observed.
Fetch Less: Caching and Conditional Requests
Every request you do not make is one that cannot be blocked, cannot be rate limited, and cannot be wrong. Before tuning concurrency upward, take the cheaper win of fetching less:
- Cache what you already fetched — A local or shared cache turns a re-run into a diff rather than a full crawl. For distributed workers a central store keeps them from independently re-fetching the same pages.
- Use conditional requests — Send If-None-Match or If-Modified-Since where the server offers an ETag or Last-Modified. An unchanged page then costs a 304 instead of a full body, which is cheaper for both sides.
- Crawl incrementally — Re-walking an entire catalogue on a schedule is the classic way to earn a block for data that did not change. Prefer sitemaps, feeds, and change detection over exhaustive sweeps.
- Look for the endpoint behind the page — Rendered pages are often fetching from an internal JSON endpoint you could call directly — smaller, more stable, and easier to parse than the markup wrapped around it.
- Deduplicate before queueing — Canonicalise URLs and drop the ones already seen. Tracking parameters and session identifiers turn one page into dozens of apparently distinct fetches.
With unlimited-bandwidth proxies the argument for caching is not the transfer cost — it is that every avoided request is one fewer chance to be noticed.
Retries, Dead Letters and Adaptive Backoff
Retries are where a polite crawler quietly becomes an aggressive one. Three rules keep them honest:
- Honour Retry-After, then back off exponentially
When the server says how long to wait, that is the answer. Where it does not, grow the interval rather than repeating a fixed one — and add jitter so concurrent workers do not all wake together and re-create the burst.
- Cap attempts and use a dead-letter queue
A URL that has failed five times is telling you something a sixth attempt will not change. Park it for inspection instead of leaving it to circulate forever consuming budget.
- Keep retries idempotent
Retry reads freely; be careful with anything that changes state on the far side. Restricting automatic retries to GET and HEAD is the safe default.
Rotating the IP and retrying immediately is not a backoff strategy. A 429 is a statement about your rate, and answering it by changing identity spends addresses to buy a few more minutes at a pace the target already rejected.
Knowing You’re Blocked Before Your Data Does
Scrapers rarely fail loudly. They start returning challenge pages with a 200 status, or empty result sets that parse cleanly, and the pipeline reports success for days. Instrument for that specific failure:
| Показатель | What it tells you | Act when |
|---|---|---|
| Block rate per target | The earliest honest signal you have | It rises at all — this moves before your dataset does |
| Records missing a required field | Catches soft blocks that return HTTP 200 | The share moves away from its normal baseline |
| 95th-percentile response time | Target strain, or a challenge being served | It climbs while throughput does not |
| Queue depth | Whether you are keeping up or accumulating | It grows steadily across a full run |
A useful cheap addition is a canary: one known-good URL with a known-good answer, fetched on a schedule. When the canary starts returning something unexpected you know it is you, not the parser, and you know it before the day’s data is written.
When the block rate does move, the blocks triage guide covers identifying what is refusing you before you start changing things.
Get a Dedicated Scraping Proxy
Live PXM2 locations — pick the country your target should see the request coming from, and get a dedicated 4G/5G IP with unlimited bandwidth and rotations:
Великобритания
Испания
Индия
Часто задаваемые вопросы
How many concurrent requests can I make when scraping?
Budget per target host, not globally. Two to five concurrent requests against one host is a defensible starting point, and a worker pool of ten to twenty across many hosts is ordinary. Raise the pool while throughput keeps climbing and the error rate stays flat; the moment errors rise without throughput following, you have found the ceiling.
How long should I wait between requests?
If robots.txt publishes a Crawl-delay, use it — that is the operator telling you the answer. Absent one, a delay of a few seconds to the same domain is conservative and rarely wrong, and ten seconds is genuinely polite for a small crawl. Delay matters far less than adaptivity: slowing down when the target signals stress is what keeps a crawl alive.
How do I scrape without getting banned?
Fetch less and behave consistently. Cache what you already have, use conditional requests so unchanged pages cost nothing, crawl incrementally instead of re-walking a whole catalogue, and keep per-host concurrency low. Most bans are earned by volume and rhythm rather than by any single request.
What should I monitor in a scraping pipeline?
Block rate per target above everything else — it moves before your data does. Then success rate, 95th-percentile response time, queue depth, and a data-quality check such as the share of records missing a required field. A scraper that silently returns challenge pages still reports HTTP 200 and an empty result set.
Is it better to rotate proxies or slow down?
Slow down first. Rotation answers an identity problem; a 429 is a rate problem, and rotating into a new IP to keep the same rate simply spends addresses to buy a few more minutes. Reduce concurrency, honour Retry-After, and rotate when an individual IP has genuinely soured.
Связанные руководства по мобильным прокси-серверам
This page is the operational discipline; the rest of the cluster covers the mechanics it applies to.
Web scraping guides
Основные руководства по мобильным прокси-серверам
Proxies That Do Not Punish a Re-Crawl
Dedicated 4G/5G modems billed per period rather than per gigabyte — caching, retries and incremental sweeps cost the same as a single pass.
Get a Scraping Proxy