Local AI models: when they are worth it for your business and when they are not

With ChatGPT, Gemini, Claude or Grok, every input goes to the provider's servers. With a local model, the model file sits on your device and the computation happens there. Programs such as Ollama or LM Studio make this accessible, LM Studio with a graphical interface. Ollama describes itself as a way to run open models on your own computer.

The models themselves are called open-weight models: their weights can be downloaded. Examples include gpt-oss from OpenAI, Gemma from Google and Qwen, which Ollama lists in its overview. Important: downloadable does not automatically mean free for every use. Licences differ. Read them before using a model commercially.

Confidential documents: contracts, personnel files, design documents, calculations, conversation notes with customers. If you do not want such material on someone else's servers, local is the most consistent answer. The data does not leave the machine, and the question of what a provider does with it goes away.

Clearly defined tasks: summarising, sorting, simple drafts, speech recognition. In my experience, smaller models that run on ordinary hardware are often enough for these.

Recurring volume: with cloud services you usually pay by usage. Locally there are no fees per request. In return you pay with hardware, electricity and time. So local is not automatically cheaper. It pays off only when use is regular and substantial.

Independence: a local model also runs without internet, and no provider changes your plans or terms overnight.

Hardware: the model has to fit into memory. The larger and more capable the model, the more memory and computing power it needs. Small models run on a current laptop or Mac, large ones need special hardware. Which size runs for you is stated in the details of each model.

Quality: in my assessment, the strongest cloud models are still better on demanding tasks, such as long multi-step jobs or when a lot of knowledge is needed. On simple tasks the difference is often smaller. Do not just take my word for it, compare with your own tasks.

Upkeep: someone has to set up the model, update it, secure it and make it available to the team. Who may access it? Where is the data, who makes backups? With a cloud service the provider takes over part of this, locally you do.

Data protection remains a topic: local does not automatically mean compliant without effort. You still need a clear purpose, access rights, documentation and team rules. The German data protection conference has published guidance on AI and data protection. It does not replace advice for your individual case.

Mistakes remain mistakes: local models also invent things. Results are checked just as with cloud models.

A simple rule of thumb: what is sensitive and clearly defined can run locally. What needs top quality and is not critical can be handled by a cloud model on a business plan. In between, the individual case decides. How the large cloud models differ is covered in my comparison of ChatGPT, Gemini, Claude and Grok.

Many businesses start with the cloud because it works immediately, and move individual tasks to local later. That is perfectly fine. What matters is that the team knows which data may go where.

I built Ratschn, a Mac app that turns speech into text locally. I mention it here only as an example, not as a recommendation. Speech recognition is a task where local fits well: it is clearly defined, a comparatively small model is often enough, and speech is frequently confidential, such as dictation, customer conversations or notes on the go. The recognition runs on the device. The principle can be implemented with other tools as well.

First: roughly sort your data into non-critical, internal and confidential. Second: pick a single task that involves confidential data, for example summarising conversation notes. Third: test small, on existing hardware, and compare the result with a cloud model. Fourth: decide whether the quality is sufficient and who takes over upkeep. Fifth: write down the rules so that everyone knows what may go where.

Start with the data question, not with hardware. What must not leave the business, and which task should be handled with it? Only then is it worth looking at models and machines.