Corpus acquisition and rate limiting

What it is

Bulk fetching from public APIs with a shared request ceiling, resumable state, and per-document failure isolation.

Why it is needed

Public data providers rate-limit per IP. A run that trips the limit can be blocked for a day; a run that dies at document 200 of 300 without resuming costs hours.

How it works

A token bucket paces requests, Retry-After is honoured, transient failures back off, and each document is checkpointed so a restart resumes rather than restarts.

In this project

662 documents from three sources at a 9 requests/second ceiling. Two implementation details worth knowing: a token bucket that starts full fires a burst of rps requests instantly and only then settles, putting roughly twice the ceiling into the opening second — which is exactly the window a per-second limit measures — so capacity is pinned to 1.0 for strict pacing. And a daily quota is distinguished from a transient 429, because retrying a daily quota inside the same run cannot help and just burns the remaining attempts.

Integrity, not just existence

A resume check that only asks whether a file exists will skip a truncated download forever, and the failure surfaces much later as an inexplicable parser error. Content-Length is verified before the write, files are written through a .part file and renamed atomically, and resume compares the file on disk against the byte count recorded in the manifest.

Trade-offs

Common mistakes

• A token bucket that starts full
• Treating HTTP 200 as success when the body is an error page
• Assuming a file that exists is a file that is complete

When to use it

How it interacts with the rest of the system

Provenance and reproducibilityDeduplication by containment