Local LLM vs Cloud LLM: Privacy, Cost and Performance Compared

Open-weight language models, whose weights can be downloaded and run on your own hardware, have improved quickly. That has turned a once academic question into a practical one for many businesses: should we keep calling a cloud AI provider, or run our own model? The honest answer depends on your data, your volume and your team, and it is often a mix of both.

Definitions first

A cloud LLM is accessed through a provider's API: you send a prompt, the provider runs the model and returns the answer. A local or self-hosted LLM runs on infrastructure you control. 'Local' can mean a developer laptop, an on-premise server or a GPU instance in your own cloud account; what matters is that the weights are served by your software and data does not go to the model provider.

Privacy: separate the myth from the real reasons

Consumer chat apps and business APIs are not the same thing. OpenAI states that data submitted through its API and business products is not used to train its models by default (OpenAI enterprise privacy), and other major providers make similar commitments for commercial use. For many companies, a business API with a data processing agreement is acceptable.

Self-hosting becomes necessary, rather than preferable, in specific situations:

• Contracts forbid it. Client agreements, government work or defence projects that prohibit sending data to third-party processors.
• Data residency. Requirements that data stay in a particular country or facility, where no suitable hosted option exists.
• Highly sensitive categories. Patient records, legal privilege, source code or trade secrets where your risk appetite for third parties is zero.
• Air-gapped environments. Factories, labs or field equipment with no reliable internet connection.
• On-device experiences. Apps that must work offline or promise users that nothing leaves the phone; see on-device AI and local-first apps.

Quality: what open-weight models do well

The largest hosted models continue to lead on the hardest tasks: complex multi-step reasoning, difficult coding and long agentic workflows. Open-weight models in the small and medium range are often good enough for summarisation, extraction into structured fields, classification, drafting and answering questions from retrieved documents. Rankings change every few months, so test candidate models on your own examples rather than relying on leaderboards alone.

Retrieval narrows the gap. When a model answers from passages you supply, it needs less built-in knowledge, so a smaller private model can perform well. Our guide to RAG vs fine-tuning explains how to adapt open-weight models further.

Hardware: a sizing rule you can use

Memory is the first constraint. Model weights take roughly parameters multiplied by bytes per parameter: 2 bytes at 16-bit precision, about 1 byte at 8-bit and about half a byte at 4-bit quantisation. Add headroom for the key-value cache, which grows with context length and the number of simultaneous users.

Serving software matters as much as hardware. Production inference servers batch requests and manage memory efficiently; the vLLM project, for example, reported 2 to 4 times higher throughput than earlier serving systems at similar latency through its PagedAttention memory management (Kwon et al., 2023). For a single machine or prototyping, tools such as Ollama make running a model straightforward.

Cost: do the maths for your workload

Cloud APIs charge per token, so cost scales with use and is close to zero when idle. Self-hosting has a fixed cost whether the GPU is busy or not, plus the people who run it. A simple comparison:

1. 01Estimate monthly tokens: requests per day × (input tokens + output tokens) × 30.
2. 02Multiply by the API's published input and output prices for the model you would use.
3. 03Price the self-hosted alternative: GPU instances or hardware amortised over its life, at enough capacity for your peak and your latency target.
4. 04Add operations: monitoring, upgrades, on-call and security work, as a share of an engineer's time.
5. 05Compare at today's volume and at the volume you expect in 12 months.

Self-hosting tends to win for steady, high-volume, narrow workloads such as classifying every incoming document. APIs tend to win for low or spiky volume, for tasks that need the strongest models, and for teams without spare operations capacity.

Licences and model choice

Open-weight does not always mean open source. Licences vary: some are permissive, while others restrict commercial use above a size threshold or for certain purposes. Meta's Llama licences, for example, include conditions for very large services. Have the licence reviewed before building a product on a model.

The hybrid pattern

Many organisations end up with both. A routing layer sends sensitive or high-volume requests to a private model and everything else to a hosted frontier model, behind one internal API. Because most inference servers expose OpenAI-compatible endpoints, applications can switch between them with little change. This keeps sensitive data in-house without giving up the best models where they matter, and it reduces dependence on any single provider.

A decision guide

If you are weighing these options, our team runs side-by-side tests of private and hosted models on your own tasks, sizes the hardware and sets up secure serving. See private and local LLM deployment, or read the broader AI automation guide for business.

Sources and further reading

• OpenAI, Enterprise privacy
• Kwon et al. (2023), Efficient Memory Management for Large Language Model Serving with PagedAttention
• Ollama
• Dettmers et al. (2023), QLoRA: Efficient Finetuning of Quantized LLMs
• European Commission, Data protection in the EU