Running AI Locally: What You Need and When It Is Worth It

How to run an AI model on your own machine with Ollama, LM Studio or llama.cpp: commands, memory needs, file formats and when local beats a paid API.

info Quick Definition
Running AI locally means executing a language model on your own computer or server, with its weights downloaded, instead of sending each request to a provider’s API. It is viable with open-weight models and gives more control over data and cost, in exchange for hardware and maintenance.

Which tools exist and how do they differ?

To run a model locally you need three things: downloadable weights (see open vs. closed models), a program that runs them and enough memory. Three programs are common:

ToolWhat it isPlatforms and licenseHow you use it
OllamaModel manager with a local servermacOS, Windows, Linux and Docker; MIT licenseTerminal and REST API at http://localhost:11434
LM StudioDesktop applicationmacOS (Apple Silicon), Windows and Linux; free at home and at work since July 8, 2025Graphical interface, lms CLI and a local server compatible with the OpenAI API
llama.cppC/C++ inference libraryMIT license; backends for CUDA, Metal, Vulkan, HIP, SYCL and CPU, among othersCommand line and its own HTTP server

All three can use the same file format: GGUF, a binary format designed for loading and saving models quickly. It was developed by Georgi Gerganov, the author of llama.cpp. LM Studio runs GGUF models with llama.cpp and, on Apple Silicon, MLX models as well. LM Studio’s terms can change: check them before using it at a company.

How do you install and test it?

The shortest path is Ollama:

  1. Download the installer from ollama.com/download (macOS, Windows or Linux). On Linux, the documentation also offers curl -fsSL https://ollama.com/install.sh | sh. If you prefer to know what you are running, read the script in your browser first.
  2. In the terminal, run ollama run gemma4:e2b. The command downloads the model and opens a chat; type /bye to leave. To only download it, use ollama pull gemma4:e2b.
  3. Check where it runs with ollama ps: the PROCESSOR column shows “100% GPU”, “100% CPU” or a split between the two.
  4. Free the memory with ollama stop gemma4:e2b. By default, a model stays loaded for five minutes after its last use.

Ollama listens only on 127.0.0.1, port 11434. A test request from another terminal:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e2b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": false
  }'

The answer comes back in message.content. According to Ollama’s FAQ, the default context window is 4096 tokens; you change it with OLLAMA_CONTEXT_LENGTH. If you set OLLAMA_HOST=0.0.0.0:11434, the server becomes reachable from other machines: do that only on a trusted network.

In LM Studio you download the app, search for a model, load it and, if you need an API, switch on “Start server” in the Developer tab or run lms server start. For llama.cpp, the project README describes installation by script, Docker, prebuilt binaries or building from source, and an -hf option to fetch a model straight from Hugging Face. The server documentation shows ./llama-server -m models/7B/ggml-model.gguf -c 2048. Command names vary between versions: check the README of the one you install.

How much memory and what hardware do you need?

The model loads into system RAM if it runs on CPU, or into the graphics card’s VRAM if it runs on GPU. Two examples of file sizes in Ollama’s library: gemma4:e2b takes 4.6 to 7.5 GB depending on the variant, and gemma4:31b takes 19 to 20 GB. That size is a minimum: the conversation context uses extra memory, and according to Ollama it grows with OLLAMA_NUM_PARALLEL multiplied by OLLAMA_CONTEXT_LENGTH.

Quantization stores the weights with fewer bits so the model takes less space and runs faster, at the cost of some precision. In Hugging Face’s table, the Q4_K type uses 4.5 bits per weight; an F16 model uses 16. That is why many downloadable models already come quantized.

Requirements published by each tool:

  • LM Studio: on Mac, Apple Silicon with macOS 14 or later and 16 GB of RAM recommended; on Windows, x64 with AVX2 or ARM, 16 GB of RAM and at least 4 GB of VRAM recommended; on Linux, Ubuntu 20.04 or later.
  • Ollama on GPU: NVIDIA with compute capability 5.0 or higher and driver 550 or newer; AMD through ROCm v7; Apple through Metal; and Vulkan on Windows and Linux.
  • Ollama on Windows: Windows 10 22H2 or newer and at least 4 GB of disk for the program. Models can add up to tens or hundreds of GB.

Why does this matter to a designer?

Because running locally changes the product’s constraints:

  • Privacy. For sensitive data, such as clinical notes or contracts, keeping the information on the machine can be a requirement, not a preference.
  • Cost and availability. There is no per-token fee and no dependence on the network: LM Studio, for example, works offline once models are downloaded. In return, speed depends on your hardware (see LLM latency).
  • Quality. A small local model performs differently from a large cloud one: the interface should communicate its limits.
  • Installation. If you ask users to install something, onboarding changes entirely: operating system, disk space and memory become part of the experience.

There is a naming trap: Ollama also offers models in its own cloud, with tags such as gemma4:cloud. They require signing in and the requests are processed on Ollama’s servers, which according to its documentation does not use them to train models. If you need nothing to leave your machine, use only local tags and disable the cloud features.

When should you NOT do it?

When volume is low and the data is not sensitive: an API is usually cheaper and simpler. Nor when nobody will maintain the system, update the models and watch its security. And when the product needs the capability of a large model, a small local one may fall short.

How do you choose between local and an API?

With a short test:

  1. Pick one concrete task and gather 30 real cases (anonymized if needed).
  2. Try two or three local models of different sizes, using their quantized tags.
  3. Measure quality, response time and memory used (ollama ps helps with the last one).
  4. Run the same cases against an API.
  5. Decide with the data: privacy, total cost and quality.

If local quality is enough, the rest of the design can stay the same; only the infrastructure changes.

Example

Illustrative example: a design studio wants to summarize user interviews that contain names and health information. By internal policy, the transcripts cannot leave the office. The team tests a local model on a machine with 32 GB of memory: the summaries are good for short interviews and lose detail in long ones. The team then splits each transcript by topic before summarizing and shows the researcher the original sentence next to each point. Privacy is preserved and verification stays with a person.

Common mistakes

  • Assuming local means private. Review logs, integrations, -cloud tags and who has access to the machine.
  • Choosing a model that does not fit in memory and blaming the tool. Look at ollama ps: if the split is “CPU/GPU”, part of the model runs in RAM and is slower.
  • Forgetting the context: the model file is a minimum, not the total consumption.
  • Exposing the server on the network (OLLAMA_HOST=0.0.0.0) without needing to.
  • Pasting install commands without reading them. Prefer the installer from the official page.

Sources

Frequently asked questions

What does running AI locally mean?
It means executing a language model on your own computer or server, with its weights downloaded, instead of sending every request to an external provider’s API.
How do I try a local model in five minutes?
Install Ollama from ollama.com/download and run, for example, ollama run gemma4:e2b. The command downloads the model and opens a chat in your terminal. Type /bye to leave.
How much memory do I need?
At least the size of the model file, plus room for the context. In Ollama’s library, gemma4:e2b is 4.6 to 7.5 GB and gemma4:31b is 19 to 20 GB. The memory is RAM when running on CPU or VRAM when running on GPU.
Is a local model more private?
It can be, because data stays on your machine. But Ollama’s -cloud tags, such as gemma4:cloud, run in Ollama’s cloud, and logs and integrations matter too.