Is local AI actually private? An honest threat model

"Local AI is private" is true the way "a locked door is secure" is true — correct as far as it goes, useless as a complete answer. A locked door stops a casual walker-through; it doesn't stop a determined thief, a copied key, or a window. Local inference stops one specific class of adversary (anyone who'd see your data via the provider you're no longer using) and does approximately nothing against several others.

If you're going to rely on local for privacy — especially for regulated or sensitive work — you owe yourself a real threat model rather than a slogan. Here it is: who local protects you from, who it doesn't, and the five questions that decide whether your specific setup actually earns the word.

The mental model: local removes an adversary, not all adversaries

Run a model on hardware you control and you've removed the provider from your threat surface. That's not trivial — it's the largest single adversary most people face, because the provider is the one entity guaranteed to see every request in a cloud setup. No provider means no provider training pipeline, no provider retention window, no provider human-review pathway, no provider subpoena compliance, no provider terms-of-service drift.

But every other adversary is still there. Local is necessary for the privacy posture most sensitive use cases require. It is not sufficient on its own. The gap between those two words is where this article lives.

What local actually protects against

Be specific. Against these adversaries, local inference is a structural defense — not a policy promise, an architecture:

• The provider's training pipeline. No data leaves your machine to reach a model, so there's nothing to train on.
• The provider's retention and logging. No transit, no transient storage, no abuse-detection logs, no safety-review queue.
• Provider-side data breach. Data that was never sent can't be stolen from where it was never stored.
• Subpoena / lawful access to the provider. A provider can't produce data it never held.
• Terms-of-service drift. Today's enterprise DPA is a contract; tomorrow's is renegotiable. An architecture doesn't renegotiate.

Notice the recurring verb: never sent. That's the property. If your setup actually keeps inference on your hardware (and we'll check below), these threats are gone by construction, not by promise.

What local does NOT protect against

This is the part that gets omitted, and it matters more than the list above.

• A compromised host. If your machine has malware, a keylogger, or a remote-access trojan, the model's privacy is irrelevant — the adversary reads the input before it reaches the model and the output before it reaches you. Host security is upstream of model privacy. Patched OS, sane browsing, disk encryption, no random software with network access.
• You, pasting secrets elsewhere. Local inference is private; the cloud tab you also have open is not. If you copy a sensitive output and paste it into a cloud tool because you feel safe about your local setup, the architecture didn't fail — you did. Humans are the bypass.
• A networked model with tool access. This is the modern trap. If you give a local agent tools — file access, shell, web fetch — and network egress, a malicious or hijacked prompt can exfiltrate data through your local model. Local keeps the inference private; it does not keep a tooled agent from phoning home. The fix is egress control and sandboxing, covered below.
• Physical access to the machine. A seized, stolen, or borrowed laptop with an unencrypted disk gives up everything, local model included. Full-disk encryption is the relevant control, not the model.
• The model itself behaving badly. Open-weight models have biases, failure modes, and hallucinations. Privacy is about data flow; model behavior is a separate axis you still own. A private model that confidently lies to you is still a problem.

The five questions that decide if your setup is actually private

Run your setup — actual or planned — through these. Each one is a potential hole.

1. Where does the inference compute happen? If the answer is "a server I control, on hardware I own, in a location I can physically point to," good. If it's "dedicated capacity on a provider's cloud," that's mitigated privacy, not private — the provider still sees the data in transit and at rest. Private cloud endpoints are not local.
2. Does any request leave your network to reach the model? If yes — even a "private" relay — the data transits infrastructure you don't control. The honest local setup has the model and the client on the same side of your boundary.
3. Is the model a pinned local file, or a version someone else can change? A pinned GGUF with a known sha256 can't be silently swapped for an instrumented version. We publish hashes for exactly this reason — the file you vetted is the file that runs. A cloud model behind an API can change overnight.
4. Where do transcripts and logs live? Open WebUI and similar store history in a local SQLite file. That's your data on your disk. If your chat history syncs to a vendor backend "for convenience," that convenience is a privacy hole.
5. What can the model reach on the network? A pure chat model with no tools is a closed box. A local agent with file access and internet is a potential exfiltration path. Match the model's permissions to the sensitivity of what you're feeding it.

If you can answer all five the "private" way, your setup earns the word for the threats local can address. If any one is a "well, sort of," you have a mitigated posture — which may be fine for your threat model, but name it honestly.

Three concrete setups, rated

To make this useful rather than abstract:

The $300 starter — single machine, Ollama + Open WebUI, no tools. Scores clean on all five. Inference is local, nothing leaves the network, the model is a pinned local file, transcripts are in local SQLite, the chat model has no network tools. Genuinely private for chat — protects against every provider-class adversary. Vulnerable to a compromised host (control: disk encryption + sane hygiene) and to you pasting secrets elsewhere (control: you).

The offline RAG knowledge base — local model pointed at your documents. Also clean, with one addition: the documents are now on the same machine as the model. That's fine — they were already on a machine you control; now they're being reasoned about locally instead of emailed to a cloud API. Genuinely private, and a strict improvement over the cloud-RAG alternative for sensitive corpora. Same host-security caveat applies.

A local coding agent with file access and shell — the private coding agent. Clean on questions 1–4. Question 5 is the risk. A coding agent that can read your files and run commands can, if hijacked by a malicious prompt, exfiltrate those files over the network you've given it. This is the modern local-privacy trap. The controls: sandbox the agent, restrict egress, don't feed it material you wouldn't want leaving the machine if the agent misbehaved. Private for inference, requires care for agency.

The honest summary

Local AI is not a privacy magic spell. It's a precise architectural choice that removes the provider from your threat model — the largest, most reliable adversary in a cloud setup — while leaving every other adversary exactly where it was. For most readers, removing the provider is the single highest-value privacy move available, and it's the precondition for the regulated-industry cases (legal, medical, finance, journalism) where the architecture isn't optional.

But "I run it locally" is the start of the privacy argument, not the end. Host security, egress control for tooled agents, disk encryption, and your own discipline about where else you paste things are what turn a local model into an actually-private system. Build for the adversary you're actually facing — not for the word "private" on a feature list.

If you haven't already: the privacy pillar makes the architectural case this threat model rests on, and the $300 starter is the minimum setup that scores clean on all five questions.