How to Set Web Scraping Concurrency Without Retry Storms

TL;DR

• Concurrency limits in-flight work; rate limits control how quickly new requests start.
• Apply limits to the source and account scope that actually shares capacity, not just to each worker independently.
• Retried requests must consume the same admission budget as first attempts.
• Treat HTTP 429 as a shared cooldown signal and preserve any valid Retry-After instruction.
• Increase concurrency only while accepted data throughput improves without unacceptable latency, errors, or queue growth.
• Keep CAPTCHA tasks and destination requests in separate pools with an explicit handoff between them.

A scraper can launch more requests and collect less usable data. Slow responses keep connections occupied, workers repeat failed requests, and the queue fills with copies of work already in progress. Increasing the worker count can then amplify the same bottleneck. The useful performance measure is accepted data delivered within the job's deadline, not the number of requests started.

This guide explains how to set a scraping concurrency limit for an authorized collection workflow. It covers source-level admission, retry scheduling, browser capacity, and a controlled tuning process. CapSolver fits into a separate supported CAPTCHA task stage when the permitted workflow requires one. Neither a larger worker pool nor a completed challenge changes the source's rate policy, so the scheduler must retain control throughout the run.

Separate concurrency from request rate

Concurrency is the number of operations currently in flight, while request rate measures starts over time. A concurrency limit alone does not guarantee a steady request rate because response time changes how quickly slots become available.

Imagine a permitted test source with four active request slots. If requests finish quickly, those slots can generate many starts in a short period. If responses slow down, the same four slots generate fewer starts while spending more time occupied. Both behaviors obey the concurrency limit. A separate rate controller is needed when starts must remain within a time-based allowance.

The concurrency glossary and rate-limiting glossary describe related concepts. In your scheduler, record them as separate settings. Add a maximum waiting queue and a job deadline so delayed work cannot accumulate indefinitely behind otherwise reasonable limits.

There is no universal best number of concurrent requests. Source policy, payload size, browser rendering, bandwidth, and downstream processing all influence the useful operating point. Start from the source's documented allowance and your own capacity, then measure an explicitly permitted workload.

Step 1: Define the scope that shares a limit

A concurrency limit must cover the workers that compete for the same capacity or permission. A per-process setting is insufficient when several processes, machines, or jobs access the same limited source.

Identify the relevant scope before creating workers. It may be a hostname, an API credential, a source-defined account quota, or a controlled browser session. Use the scope described by the provider rather than assuming that every URL has independent capacity. Subdomains can also share infrastructure or account-level allowances.

Separate global capacity from source capacity. A global ceiling protects your own machines and outbound resources. Per-source ceilings prevent one fast queue from consuming all available slots. If the source is temporarily paused, another independently permitted source can continue without borrowing the paused source's identity or quota.

For browser workflows, decide what occupies a slot: a navigation, an active page, or an entire session. An account-bound session should not be casually shared between concurrent jobs. Its cookies, pending actions, and expected destination need an owner for the duration of the task.

Step 2: Make every request pass through admission

Admission should happen before the request starts and should apply equally to initial attempts, retries, redirects where controlled by your client, and worker restarts. A retry path that sends directly to the transport can defeat the main scheduler's limits.

Use one task identity for the intended observation and separate attempt identities for transport work. This allows the queue to distinguish a legitimate retry from a duplicated scheduled job. If a worker restarts, it should inspect whether the observation already completed before creating another request.

A slot should be released when the relevant operation has actually finished or its transport has been cancelled. Releasing capacity merely because the caller stopped waiting can leave hidden requests running beyond the nominal limit. Treat uncertain or still-running work as occupied capacity until its state is resolved.

Bound the queue as well. When too much work arrives, defer a later collection window, reject excess demand, or reduce the requested scope according to your service contract. Keeping an unlimited backlog shifts failure into memory pressure and stale observations.

Step 3: Schedule retries instead of sleeping everywhere

Retries should return to a controlled queue with a future eligible time, an attempt count, and the remaining task deadline. Scattered sleep calls inside workers make it difficult to coordinate a shared source cooldown.

The HTTP 429 status indicates that too many requests have been sent within a relevant period. When the response includes Retry-After, preserve its delay or date semantics. Do not let every worker independently choose a shorter wait.

Use a shared cooldown for the affected scope. Stop admitting new work there while the cooldown is active, including requests that have never failed. Already-running work may finish, but a successful response from one worker should not automatically erase a valid cooldown observed by another.

For temporary read failures without explicit server guidance, apply a bounded retry policy suited to the operation. A staggered schedule can avoid synchronized bursts after the pause. Do not add retries to a form submission or other state-changing operation unless its effect and repeatability are understood.

Observation	Scheduler response	Evidence to retain
HTTP 429	Pause the affected scope and respect retry guidance	Source scope, response time, cooldown
Temporary read timeout	Requeue only within the attempt and deadline budget	Attempt identity and remaining time
Authentication or access refusal	Stop or route for access review	Redacted response category
Supported CAPTCHA checkpoint	Enter a separate permitted challenge stage	Current observation and challenge context
Invalid extracted record	Investigate parsing or source data	Retained page reference and validation error

Redeem Your CapSolver Bonus Code

Boost your automation budget instantly!
Use bonus code CAP26 when topping up your CapSolver account to get an extra 5% bonus on every recharge — with no limits.
Redeem it now in your CapSolver Dashboard

Step 4: Keep challenge handling from multiplying demand

A CAPTCHA checkpoint should create a controlled handoff, not an extra request loop running beside the scraper. The destination observation remains one task even if it requires a supported challenge step.

The CapSolver createTask documentation defines task submission, while getTaskResult documents result retrieval. Follow the applicable task requirements and keep the returned task identity associated with the current observation. Polling a known task and creating a new task are different operations.

Give this stage its own limit for active work, time, and attempts. The destination's rate allowance still applies when the browser resumes. A challenge result should not grant an immediate exception to a source cooldown or cause all paused workers to restart together.

If a task submission times out before its acceptance is known, retain the uncertainty. Blindly submitting another task may duplicate work. The scheduler should know whether it is waiting for an existing result, investigating an uncertain request, or closing the observation for review.

After the destination proceeds, verify the expected page and record contract. Otherwise, increased challenge throughput can hide the fact that accepted collection output has not improved.

Step 5: Run a controlled concurrency ramp

A concurrency ramp should change one capacity setting while keeping the comparison workload and acceptance criteria stable. Use an owned test source or another environment where load testing is explicitly permitted.

Establish a baseline

Start with a small allowed workload and record completed observations, accepted records, request latency, error categories, and queue age. Separate time spent waiting for admission from time spent on the network or in the browser. A long end-to-end duration can have different causes that require different fixes.

Keep first attempts and retries visible in the same report. If accepted records remain constant while total requests rise, the additional traffic is not producing useful throughput. Check whether retry policy, page readiness, or downstream validation explains the difference before adding workers.

Increase only while the result improves

Raise the limit by a modest predefined step and observe a comparable interval. Stop increasing when accepted throughput flattens, failure frequency rises, or tail latency exceeds the task's needs. Those observations identify a capacity boundary for that workload; they do not prove a permanent provider-wide limit.

Repeat representative cases across payload sizes and page types. A lightweight HTML page and a JavaScript-heavy dashboard can require very different browser resources. Avoid selecting the operating limit from the easiest pages alone.

Keep room for variation

Choose an operating point with room below the failure boundary rather than permanently running at the highest value that briefly passed. Source performance and your own infrastructure can vary. Reserve capacity for scheduled work and prevent exploratory jobs from taking over the entire pool.

Understand what automatic throttling controls

Automatic throttling can adapt request timing to observed latency, but its behavior depends on the implementation and configured limits. Read the consuming framework's documentation before combining it with a second scheduler.

The Scrapy AutoThrottle documentation describes a target concurrency and latency-based delay adjustment while respecting configured concurrency ceilings. Its target is not an unconditional promise that an exact number of requests will always be active. Non-success responses also receive special treatment in the delay calculation.

Use framework-specific controls for their documented scope. A crawler's local throttle does not automatically coordinate separate deployments or impose a total organizational quota. If several jobs share one allowance, place a shared controller above those independent workers.

Record the effective configuration used for each run. A concurrency experiment is hard to reproduce if the framework, transport pool, retry middleware, and external scheduler all change at once. The browser infrastructure overview provides useful context for separating these resources.

Diagnose lower throughput before adding capacity

Lower accepted throughput can come from the network, source, browser, parser, or destination store. More fetch workers only help some of those bottlenecks and can make others worse.

If the queue grows while network slots remain idle, inspect admission and cooldown logic. If browser memory rises, review session lifetime and page cleanup. If requests finish but records are rejected, inspect content readiness, schema changes, and parser behavior. If accepted records wait to be stored, apply backpressure from the downstream writer.

Keep an explicit pause procedure. Operators should be able to stop new starts for one source, preserve in-flight task identities, and resume gradually after the issue is understood. Restarting every worker at once is a poor substitute for a controlled reopening of capacity.

Tune for useful collection output

Set scraping concurrency around source policy, actual resource ownership, and accepted records. Route every attempt through admission, coordinate cooldowns, and increase load only when the complete pipeline benefits.

For workflows requiring supported CAPTCHA handling, use CapSolver within a separate bounded stage and return to the same source controls afterward. A disciplined scheduler makes the system easier to operate as load and page behavior change.

FAQ

Q: Is a concurrency limit the same as a requests-per-second limit?

No. Concurrency limits active operations. Request rate limits starts over time, so you may need both controls.

Q: Should each worker handle HTTP 429 independently?

Workers sharing the same rate-limited scope should coordinate their cooldown. Independent retries can recreate a burst even when each worker appears conservative.

Q: Can I fix slow scraping by increasing concurrency?

Only when additional capacity improves accepted output within the source's permitted limits. First identify whether acquisition, rendering, parsing, or storage is the bottleneck.

Q: Should a CAPTCHA result let a request skip the scheduler?

No. The destination's rate and concurrency rules still apply after challenge handling completes.

Q: What should happen when downstream storage falls behind?

Reduce or pause new acquisition through backpressure. An unlimited queue of fetched pages increases resource use and can leave observations stale before they are accepted.