Batch-process 10,000 documents locally for the cost of electricity

The fifth item in 5 things you can do with a local LLM was automation and batch processing. This is the deep-dive: how to run a local model against a queue of thousands of documents, where this workflow's economics actually break in favor of local, and the honest throughput limits of doing it on a single workstation.

This is the use case where the cost-inverts-at-scale argument stops being theoretical. A cloud API is cheap for ten documents and brutal for ten thousand. A local GPU's cost is fixed the day you buy it. The break-even — the point at which "buy a card" beats "rent an API" — is measured in batches, not years.

The shape of the problem

You have a corpus — support tickets, scanned PDFs, meeting transcripts, a backlog of emails, an image library needing alt text, contracts needing clause extraction — and a transformation you want applied to every item: classify, extract structured fields, summarize, translate, redact, tag. Doing this by hand is infeasible. Doing it through a cloud API at scale is expensive and, for sensitive corpora, non-compliant. Doing it locally is the third option, and it's often the right one.

The architecture is simple: a queue, a worker loop, a local model, and an output store.

1. The queue — a folder of input files, a database table of pending rows, or a real queue (Redis, SQS) for larger setups.
2. The worker loop — pull an item, build the prompt, call the local model API (Ollama exposes one), parse the output, write the result, mark the item done, repeat.
3. The model — same local model server as everything else. Often a smaller model is the right pick here, because throughput matters more than per-item brilliance.
4. The output store — wherever you want the results: a CSV, a database, JSON files alongside the originals.

That's the whole thing. The sophistication is in the prompt and the parsing, not the plumbing.

The throughput math, honestly

Here's the part to read carefully, because local batch processing has one real constraint and it's not cost — it's that a single local GPU processes one stream at a time at usable speed.

Single-stream generation on a 24GB card runs roughly 30–100 tok/s depending on model size and backend (our benchmarks have the real numbers). For a typical extraction task — say, a 150-token prompt in, a 100-token structured answer out — that's somewhere around 1–3 seconds per item, plus overhead. Call it ~2 seconds per item conservatively.

• 1,000 items → ~30–60 minutes
• 10,000 items → ~5–10 hours
• 100,000 items → ~2–4 days

That's the honest shape. For a one-time backfill or a nightly batch, it's fine — start it, walk away, come back to finished work. For "I need a million items by tomorrow morning," it's not — you need a server farm, not a workstation, and you should be honest with yourself about that before you start.

The lever you do have: for batch work, pick the smallest model that does the job adequately. A 7B–8B model runs 2–5× faster than a 30B and is often perfectly good at classify/extract/summarize tasks where the bar is "good enough" not "frontier." Throughput is a model-size decision more than anything else. See the MoE shift for why a small-active MoE is often the right batch pick.

The lever you don't have on a single workstation: meaningful concurrency. You can run multiple worker processes, but they share one GPU and the throughput doesn't multiply — you're slicing the same pie. Real horizontal scaling means more GPUs, which is a different cost conversation. (And if you go there, vLLM/SGLang for continuous batching becomes the right serving layer, not Ollama.)

The economics: where local actually wins

Cloud API pricing is per-token, and it's reasonably cheap per-call — until volume hits. A rough illustration (use your actual provider's pricing for real math):

• A typical extraction call: ~250 tokens in, ~100 tokens out. At cloud per-token pricing in the $1–$10 per million-token range, that's somewhere on the order of $0.001–$0.003 per call, depending on provider and tier.
• At 10,000 calls: ~$10–$30. Annoying but not crazy.
• At 100,000 calls: ~$100–$300. Starting to hurt.
• At 1,000,000 calls (a real batch job, run repeatedly): ~$1,000–$3,000 per run. Now you're paying for a card every few runs.

(The exact numbers depend on your provider, your token counts, and your pricing tier — don't quote these, redo the math with your real figures. The shape is what matters: per-token pricing scales linearly forever; a local GPU's cost is fixed.)

Compare to a $300–$700 card: it's paid for itself somewhere in the 100K–1M-call range, and after that every call is the cost of electricity. For a recurring batch job — nightly ticket classification, weekly report summarization, monthly contract review — the break-even comes fast, often within the first run or two.

And that's before the compliance argument. If the corpus is sensitive (customer PII, internal communications, medical records), the cloud option may not be available at any price. Local isn't just cheaper; it's the only compliant path.

What it's good at

• Classification and routing. "Is this support ticket a refund request, a bug, or a how-to question?" Run it across the whole inbox; route accordingly.
• Structured extraction. "Pull the parties, dates, and termination clause from each contract into this JSON schema." Reliable for well-structured documents; needs review for edge cases.
• Summarization at scale. A year of meeting transcripts → one-sentence summaries each, in a few hours. Suddenly searchable.
• Alt text generation. Point a multimodal local model at an image library; get draft alt text for every image. (Heavier lift — requires a vision-capable model.)
• Translation, redaction, tagging, formatting normalization. Any repetitive text transformation where "good enough at scale" beats "perfect but manual."

Where it falls down

• Throughput on truly large batches. Single-workstation throughput caps out as described above. Million-item-by-tomorrow is server-farm territory.
• Quality variance at scale. A 2% error rate on 100 items is 2 wrong. On 10,000 items it's 200 wrong. Build in sampling-based QA — spot-check a random N from every batch before trusting the output.
• Prompt drift. If the input distribution shifts (new ticket types, new contract templates), the prompt that worked yesterday starts producing garbage. Batch jobs need monitoring, not fire-and-forget.
• Parsing failures. Models don't always return clean structured output. Wrap your parsing defensively (JSON repair, retries with a stricter prompt, fallback to a flag-for-human-review bucket).
• Idempotency. If a batch crashes at item 6,000 of 10,000, you want to resume, not restart. Make the worker loop checkpoint its progress.

A reasonable starting config

• Hardware: whatever you have — a 12GB+ card. Batch work is forgiving on hardware because you're not latency-bound.
• Runtime: Ollama for ease. If you're running multiple worker processes or hitting real concurrency, that's the cue to evaluate vLLM/SGLang.
• Model: the smallest that does the job. For classify/extract/summarize, start at 7B–8B and only step up if quality demands it.
• Queue: a folder of files or a database table for small batches; Redis for larger ones.
• Worker pattern: pull → prompt → parse → write → checkpoint. Idempotent. Sampled QA every run.

The takeaway

Batch processing is the local-AI use case where the economics aren't even close. Per-token cloud pricing scales linearly forever; a local GPU's cost is fixed the day you buy it, and for any recurring batch job the break-even arrives within a run or two. The binding constraint isn't cost — it's single-stream throughput, which caps you at "tens of thousands of items per overnight run" rather than "millions by morning." For the former, which is most real batch work, local is the obvious answer; for the latter, you need a server farm, and you should know that going in.

The model server is the same as the self-host starter; the cost-at-scale framing is in why local AI matters in 2026; and the throughput numbers behind the model-size choice are in the open benchmarks.