# Running AI Locally: What You Need and When It Is Worth It

> How to run an AI model on your own machine with Ollama, LM Studio or llama.cpp: commands, memory needs, file formats and when local beats a paid API.

*Tags: ai, models, privacy, tools*

---


> [!info] Quick Definition
> **Running AI locally** means executing a language model on your own computer or server, with its weights downloaded, instead of sending each request to a provider's API. It is viable with open-weight models and gives more control over data and cost, in exchange for hardware and maintenance.


## Which tools exist and how do they differ?

To run a model locally you need three things: downloadable weights (see [[open-vs-closed-models|open vs. closed models]]), a program that runs them and enough memory. Three programs are common:

| Tool | What it is | Platforms and license | How you use it |
|---|---|---|---|
| Ollama | Model manager with a local server | macOS, Windows, Linux and Docker; MIT license | Terminal and REST API at `http://localhost:11434` |
| LM Studio | Desktop application | macOS (Apple Silicon), Windows and Linux; free at home and at work since July 8, 2025 | Graphical interface, `lms` CLI and a local server compatible with the OpenAI API |
| llama.cpp | C/C++ inference library | MIT license; backends for CUDA, Metal, Vulkan, HIP, SYCL and CPU, among others | Command line and its own HTTP server |

All three can use the same file format: **GGUF**, a binary format designed for loading and saving models quickly. It was developed by Georgi Gerganov, the author of llama.cpp. LM Studio runs GGUF models with llama.cpp and, on Apple Silicon, MLX models as well. LM Studio's terms can change: check them before using it at a company.

## How do you install and test it?

The shortest path is Ollama:

1. Download the installer from `ollama.com/download` (macOS, Windows or Linux). On Linux, the documentation also offers `curl -fsSL https://ollama.com/install.sh | sh`. If you prefer to know what you are running, read the script in your browser first.
2. In the terminal, run `ollama run gemma4:e2b`. The command downloads the model and opens a chat; type `/bye` to leave. To only download it, use `ollama pull gemma4:e2b`.
3. Check where it runs with `ollama ps`: the PROCESSOR column shows "100% GPU", "100% CPU" or a split between the two.
4. Free the memory with `ollama stop gemma4:e2b`. By default, a model stays loaded for five minutes after its last use.

Ollama listens only on `127.0.0.1`, port 11434. A test request from another terminal:

```shell
curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e2b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": false
  }'
```

The answer comes back in `message.content`. According to Ollama's FAQ, the default context window is 4096 tokens; you change it with `OLLAMA_CONTEXT_LENGTH`. If you set `OLLAMA_HOST=0.0.0.0:11434`, the server becomes reachable from other machines: do that only on a trusted network.

In LM Studio you download the app, search for a model, load it and, if you need an API, switch on "Start server" in the Developer tab or run `lms server start`. For llama.cpp, the project README describes installation by script, Docker, prebuilt binaries or building from source, and an `-hf` option to fetch a model straight from Hugging Face. The server documentation shows `./llama-server -m models/7B/ggml-model.gguf -c 2048`. Command names vary between versions: check the README of the one you install.

## How much memory and what hardware do you need?

The model loads into system RAM if it runs on CPU, or into the graphics card's VRAM if it runs on GPU. Two examples of file sizes in Ollama's library: `gemma4:e2b` takes 4.6 to 7.5 GB depending on the variant, and `gemma4:31b` takes 19 to 20 GB. That size is a minimum: the conversation context uses extra memory, and according to Ollama it grows with `OLLAMA_NUM_PARALLEL` multiplied by `OLLAMA_CONTEXT_LENGTH`.

**Quantization** stores the weights with fewer bits so the model takes less space and runs faster, at the cost of some precision. In Hugging Face's table, the Q4_K type uses 4.5 bits per weight; an F16 model uses 16. That is why many downloadable models already come quantized.

Requirements published by each tool:

- **LM Studio:** on Mac, Apple Silicon with macOS 14 or later and 16 GB of RAM recommended; on Windows, x64 with AVX2 or ARM, 16 GB of RAM and at least 4 GB of VRAM recommended; on Linux, Ubuntu 20.04 or later.
- **Ollama on GPU:** NVIDIA with compute capability 5.0 or higher and driver 550 or newer; AMD through ROCm v7; Apple through Metal; and Vulkan on Windows and Linux.
- **Ollama on Windows:** Windows 10 22H2 or newer and at least 4 GB of disk for the program. Models can add up to tens or hundreds of GB.

## Why does this matter to a designer?

Because running locally changes the product's constraints:

- **Privacy.** For sensitive data, such as clinical notes or contracts, keeping the information on the machine can be a requirement, not a preference.
- **Cost and availability.** There is no per-token fee and no dependence on the network: LM Studio, for example, works offline once models are downloaded. In return, speed depends on your hardware (see [[llm-latency|LLM latency]]).
- **Quality.** A small local model performs differently from a large cloud one: the interface should communicate its limits.
- **Installation.** If you ask users to install something, onboarding changes entirely: operating system, disk space and memory become part of the experience.

There is a naming trap: Ollama also offers models in its own cloud, with tags such as `gemma4:cloud`. They require signing in and the requests are processed on Ollama's servers, which according to its documentation does not use them to train models. If you need nothing to leave your machine, use only local tags and disable the cloud features.

## When should you NOT do it?

When volume is low and the data is not sensitive: an API is usually cheaper and simpler. Nor when nobody will maintain the system, update the models and watch its security. And when the product needs the capability of a large model, a small local one may fall short.

## How do you choose between local and an API?

With a short test:

1. Pick one concrete task and gather 30 real cases (anonymized if needed).
2. Try two or three local models of different sizes, using their quantized tags.
3. Measure quality, response time and memory used (`ollama ps` helps with the last one).
4. Run the same cases against an API.
5. Decide with the data: privacy, total cost and quality.

If local quality is enough, the rest of the design can stay the same; only the infrastructure changes.

## Example

Illustrative example: a design studio wants to summarize user interviews that contain names and health information. By internal policy, the transcripts cannot leave the office. The team tests a local model on a machine with 32 GB of memory: the summaries are good for short interviews and lose detail in long ones. The team then splits each transcript by topic before summarizing and shows the researcher the original sentence next to each point. Privacy is preserved and verification stays with a person.

## Common mistakes

- Assuming local means private. Review logs, integrations, `-cloud` tags and who has access to the machine.
- Choosing a model that does not fit in memory and blaming the tool. Look at `ollama ps`: if the split is "CPU/GPU", part of the model runs in RAM and is slower.
- Forgetting the context: the model file is a minimum, not the total consumption.
- Exposing the server on the network (`OLLAMA_HOST=0.0.0.0`) without needing to.
- Pasting install commands without reading them. Prefer the installer from the official page.

## Sources

- [Ollama: quickstart](https://docs.ollama.com/quickstart)
- [Ollama: FAQ](https://docs.ollama.com/faq)
- [Ollama: GPU support](https://docs.ollama.com/gpu)
- [Ollama: Windows](https://docs.ollama.com/windows)
- [Ollama: Linux](https://docs.ollama.com/linux)
- [Ollama: cloud models](https://docs.ollama.com/cloud)
- [Ollama: GitHub repository](https://github.com/ollama/ollama)
- [Ollama: library, gemma4](https://ollama.com/library/gemma4)
- [llama.cpp: GitHub repository](https://github.com/ggml-org/llama.cpp)
- [llama.cpp: server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md)
- [LM Studio: system requirements](https://lmstudio.ai/docs/app/system-requirements)
- [LM Studio: lms CLI](https://lmstudio.ai/docs/cli)
- [LM Studio: free to use at work (July 8, 2025)](https://www.lmstudio.ai/blog/free-for-work)
- [Hugging Face: GGUF format](https://huggingface.co/docs/hub/gguf)


---

Source: https://www.fernandoux.com/en/wiki/ai/local-ai/