All product reviews

llms · research

Local vs hosted LLMs for private documents

How to choose a local or hosted model for sensitive documents by threat model, retention, capability and operating cost—not by slogans.

2026-08-07 · 2 min read

Evidence: Vendor data-control documentation reviewed 7 August 2026; no security audit performed

“Private” is not a model feature. It is a path your document takes, the systems that can retain it, the people who can administer those systems, and the contract that governs the whole chain. Start by naming the risk, then choose the deployment.

This is a research comparison based on vendor documentation checked on 7 August 2026. It is not a penetration test, a legal opinion or proof that any deployment is safe.

The verdict

Use a local runtime when the strongest useful rule is “the document does not leave this machine or network.” Ollama can run models locally and supports schema-constrained JSON output, which makes it viable for many extraction jobs without sending the source to a model provider.

Use a hosted API when its capability or operational simplicity is worth the transfer and the document is allowed under the provider, account and endpoint controls you actually configured. OpenAI states that API data is not used to train models by default, while default abuse-monitoring logs can be retained for up to 30 days and some endpoints store application state unless configured otherwise. Anthropic says commercial inputs and outputs are normally deleted within 30 days, with documented exceptions and zero-data-retention arrangements for eligible customers. Those are not the same as “nothing is stored.”

Decide from the document backwards

Classify a representative document before choosing a product:

1. Public or low sensitivity. A hosted API is usually the least operational work.
2. Confidential business data. Check contracts, region, retention, logs and endpoint behaviour.
3. Regulated, privileged or identity-rich data. Require a formal review; local is not automatically compliant if the laptop, backups or telemetry are uncontrolled.
4. Material that must never cross a boundary. Keep inference, indexes, prompts and outputs inside that boundary and verify outbound traffic rather than assuming it.

Then run an accuracy evaluation on redacted examples. A private system that extracts the wrong party, amount or date is still unsafe for the decision built on top of it.

The costs people omit

Local inference has no per-token invoice, but it has hardware, electricity, setup, patching and support costs. Hosted inference has a visible unit price, plus possible engineering work to redact, route and audit requests. Compare cost per accepted result, including human correction time.

What would change this review

We would upgrade the evidence label after testing the same document set locally and through hosted APIs, publishing accuracy, latency, energy, network traces and the exact retention configuration.

Sources: Ollama structured outputs, OpenAI API data controls, and Anthropic commercial data retention.