Local vs cloud AI: the full comparison

"Should I run AI locally or use the cloud?" gets answered with "it depends" or, worse, with a tribal slogan from someone who picked a side. Neither helps you decide. This is the honest four-axis comparison — cost, privacy, latency, capability — with a concrete decision rule at the end. We run local hardware and we use cloud APIs; the truthful answer is that each wins on different axes for different people, and pretending one is universally better is how you waste money or give up capabilities you needed.

If you want the deep-dive on any single axis, this piece links out: privacy, cost math, and the media-generation-specific version for image/video work. This is the general decision framework that sits above all of them.

Axis 1 — Cost: it depends entirely on volume

This is the axis people get most wrong, because the right answer flips based on one variable.

Cloud wins at low volume. A pay-per-token API is genuinely cheap for occasional use. A few hundred chat messages a month, a handful of coding queries, the odd document summary — you'll spend dollars, not tens of dollars, and you'll never amortize hardware. If this is your usage, buying a GPU to "save money" is a category error. Stay on the API.

Local wins at high volume. A local GPU is a fixed cost. You pay once, and the marginal cost of inference becomes the electricity (~$0.001 per thousand generations once you own the hardware). The moment your usage is high — agentic loops that call the model forty times per task, batch evaluations over thousands of items, RAG over large corpora, a team sharing the box — the cloud API bill climbs fast and the local break-even arrives in weeks, not years. We worked the full math here, including the crossover point.

The honest summary: cost is a volume play. Light users should stay on the cloud and keep their money. Heavy users, or anyone who values the other three axes below, amortize hardware fast. The crossover is knowable — calculate it rather than guess.

Axis 2 — Privacy: local wins structurally, cloud mitigates contractually

This is the axis with the widest gap between marketing and reality, and it's the one where local has a categorical advantage.

**Cloud can promise privacy; only local can provide it as architecture.** "We don't train on your data" is a real and meaningful promise, but it's a contract layered on top of a structural fact: your data left your machine to reach the model. That means transit logging, retention windows, abuse-detection queues, human-review pathways, subprocessor routing, subpoena compliance, and the ever-present possibility of terms-of-service drift. Every one of those is a mitigation, not an elimination. The threat model is here.

Local removes the provider entirely. Inference on hardware you control means no transit, no provider retention, no provider breach surface, no provider subpoena target. For casual use the cloud's promises are usually adequate. The moment your data has legal, medical, financial, or source-protection sensitivity, "we won't look at it" stops being an acceptable substitute for "they can't." The privacy case in full.

The honest summary: for most casual use, cloud privacy is fine and you shouldn't over-index on it. For sensitive work, local isn't a preference — it's the only configuration where the privacy posture is architectural rather than contractual. Build the genuinely-private version.

Axis 3 — Latency: local wins for anything agentic

Cloud round-trips add 200–800ms of network latency on top of generation time. For a one-shot question you type and wait on, that's invisible. The cloud "feels" instant for chat.

Local wins the moment the model is called in a loop. An agent that calls the model forty times to finish a task eats that network latency forty times. A coding agent iterating on a file. A RAG pipeline retrieving and re-querying. A batch job generating a thousand outputs. At those scales, network latency is the difference between "feels instant" and "feels broken," and local collapses it to the model's own generation speed. We measured our fleet's generation speeds — a 30B model on a 24GB card runs at genuinely real-time tok/s for interactive use.

The honest summary: for interactive one-shot chat, latency is a wash and the cloud's bigger models may win your attention. For anything agentic, batched, or latency-sensitive, local is categorically better because there's no network in the loop.

Axis 4 — Capability: cloud still wins the ceiling, local has closed most of the gap

This is the axis where the cloud's honest remaining advantage lives, and pretending otherwise is dishonest.

Cloud wins on raw frontier capability. The largest models — the genuinely biggest, trained at the most compute — still don't fit on consumer hardware, full stop. If your workload genuinely requires frontier-class reasoning on hard problems (the 10% case), no local setup competes. This is the cloud's real, durable argument.

Local has closed the gap on almost everything else. A 30B-parameter model quantized to Q4_K_M now lands within striking distance of last year's frontier cloud models on real workloads, and it runs on a single $300 GPU. For the 90% of daily work — chat, writing, coding assistance, document Q&A, structured output — the local answer is now good enough that "but the cloud has a bigger model" is often a theoretical rather than practical advantage. The model landscape is here.

The honest summary: if you need the absolute biggest, smartest model for genuinely hard problems, the cloud is still the answer and likely will be. For daily-driver work, local is competitive, and its advantages on the other three axes often outweigh the capability delta you're giving up.

The decision rule

Four axes, one rule. Work through these in order:

1. Is your data sensitive (legal, medical, financial, source-protective)? → Local. Privacy is architectural and the other axes barely matter. Start here.
2. Is your usage high-volume (agents, batch, team, RAG at scale)? → Local. Cost inverts in your favor and latency compounds. The cost math.
3. Do you need frontier-class reasoning on hard problems? → Cloud. This is the cloud's honest remaining monopoly. Accept the tradeoffs.
4. Is your usage light and your data non-sensitive? → Cloud. You'll save money and skip setup. Don't buy hardware to solve a problem you don't have.

Notice that 1 and 2 both point local, 3 points cloud, and 4 points cloud. The largest group of people who should be on local are the ones in 1 and 2 who are currently paying a cloud API because they haven't done the math. The largest group who shouldn't buy hardware are the people in 4 who'd never amortize it.

The honest hybrid

Most serious users end up running both. Local for the daily-driver work where it wins on cost, privacy, and latency; cloud APIs for the occasional frontier-reasoning task where capability is the only axis that matters. This isn't a compromise — it's the correct architecture for a lot of people. The local box handles 90% of your volume cheaply and privately; the cloud handles the 10% that genuinely needs the biggest model. That's the position we actually hold: local is a reasonable default, not a religion, and the cloud's remaining argument is "we have the biggest model" — which most days, you don't need.

Next steps: the VRAM guide if you're leaning local and need to pick hardware; the cost breakdown if you want the actual crossover number for your usage; the $300 starter if you want to try local this weekend before committing.