Run a private AI model at home

Is it for you?

A local AI model is a file on your computer. A free program runs it. You chat with it like an online chatbot, but nothing leaves your machine.

Good for:

• Questions you don’t want stored on someone else’s server.
• Reading and summing up your own files.
• Writing and coding help without an account.
• Making images on your own machine.
• Working with no internet at all.

Not for:

• Matching the biggest online models on a cheap machine. Small local models are less sharp.
• Old or low-memory machines. Below 8 GB it gets slow and dumb fast.
• Facts you can’t check. Local models make things up too.

Your prompts stay at home. No account, no logs on someone else’s server.

Model The file with the AI’s “brain”. You download it once, like a big app.

Memory: how much, and how fast

Two numbers decide what you can run.

Capacity decides which models fit. The model has to fit in memory, plus a few GB for your chat and the system.

Bandwidth decides how fast it writes. Each new word reads the model from memory again. So faster memory means more words per second. A DGX Spark has 273 GB/s. A Mac Studio M5 Ultra has about 1.2 TB/s. Both have lots of memory, but the Mac writes faster on the same model (Mia, 7 Oct).

Model files come in sizes. Fewer bits per number means a smaller file and a little less quality. Here is one model, Qwen3.8 27B, in Ollama’s library:

Rough rule for Q4: about 0.6 GB per billion parameters. More examples:

Sizes are from the Ollama model library on 19 Oct 2026.

On a graphics card, only the card’s own memory (VRAM) counts at full speed. On a Mac or a shared-memory box, most of the total memory counts.

Q4, Q8, GGUF Q4 and Q8 are 4-bit and 8-bit versions of a model. GGUF is the file format Ollama and llama.cpp use. Other engines use other formats, like EXL3.

Parameters How big the model is. 27B means 27 billion. Bigger is smarter and slower.

Which machine should I buy?

Start with the computer you have. Buy only when you know what you want to run. Speeds are in tokens per second (tok/s) for one chat. They come from people who own the machines. Your numbers will differ.

Prices: RTX 5090, Strix Halo and DGX Spark from our tracker on 19 Oct 2026. The Mac price is Mia’s figure. Used card prices vary a lot.

Pick by what you’ll do:

• Chat and writing: any row works. Bigger memory means a smarter model.
• Coding: speed matters, because code answers are long. A 24–32 GB NVIDIA card running a 27B model is the sweet spot.
• Images: an NVIDIA card is fastest. Shared-memory boxes work, just slower. On Strix Halo, one 1-megapixel image took 16–18 seconds (Jean-Claude). Mia names the Mac and the DGX Spark for media work too (Mia).

These boxes keep selling out and going up in price. Memory is scarce this year. Check stock and prices before you buy.

We track who has them, in the US, Europe and Asia: AI hardware tracker.

Tokens per second How fast the model writes. A token is about three quarters of a word. About 6 tok/s keeps up with your reading. Faster helps for long code answers.

Install Ollama

Ollama is a free, open-source program (MIT license) that downloads and runs models.

curl -fsSL https://ollama.com/install.sh | sh

Prefer to read a script before you run it? Open ollama.com/install.sh in your browser first.

Run your first model

1. Open a terminal (Mac: Terminal. Windows: PowerShell).
2. Download a model and start a chat:

ollama run gemma4

Bigger machine? Use the starter model from step 3 instead, for example ollama run qwen3.8:27b.

1. The first time, it downloads the file. That can take a while.
2. Type a question and press Enter.
3. Type /bye to quit.

Handy commands:

ollama list        # what you have
ollama ps          # what is loaded right now
ollama pull gpt-oss:20b
ollama rm gemma4   # delete a model

Skip any model tagged “cloud”. Those run on Ollama’s servers, so your prompts leave your machine. Local tags look like gemma4:12b.

Add a chat window: Open WebUI

Open WebUI gives you a ChatGPT-style page in your browser that talks to Ollama on your machine.

pip install open-webui
open-webui serve

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

1. Create the first account. It’s a login on your own machine, and it becomes the admin.
2. Pick your model at the top and chat.

Open WebUI is source-available under its own license, with a rule to keep its branding. The code is public. It is not a standard open-source license.

Faster engines: llama.cpp and more

Ollama is the easy start. It’s not the fastest. The engine that runs the model matters almost as much as the model (Mia). One RTX 3090 owner went from 40 to 100 tok/s on the same model by moving from Ollama to exllamav3 (source).

llama.cpp is the engine many tools are built on. It’s open source (MIT) and has its own simple web chat built in. It runs on NVIDIA, AMD, Apple and plain CPUs. On AMD it also has a Vulkan build, so you can skip AMD’s ROCm drivers.

1. Install it. On Mac or Linux:

brew install llama.cpp

Or use the official script, or winget install llama.cpp on Windows:

curl -LsSf https://llama.app/install.sh | sh

1. Start a small model and the built-in web chat:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

1. Open the address it prints in your browser. The default port is 9931.

Command not found? Older installs call it llama-server instead of llama serve.

Have a model file already? Point at it with -m:

llama-server -m ~/models/model.gguf --port 8080

Once you know your machine, these are worth a look:

• NVIDIA card, 16–32 GB: exllamav3 (MIT) with EXL3 files. Mia keeps one-click recipes for Qwen3.8 27B on 16–24 GB cards (Mia).
• DGX Spark: vLLM or SGLang (both Apache 2.0), or TensorFold (Apache 2.0).
• Strix Halo: Jean-Claude’s OmaRed scripts (MIT) set up Linux, kernel flags and the Halogen Flash engine for Qwen3.8 Flash (Jean-Claude). We couldn’t confirm Halogen’s own license, so check it first.
• Mac: MLX builds, which Ollama also offers as “-mlx” tags.

Prefer an app? The llama.cpp team and Hugging Face make Llama, a small open-source app for Mac and Windows. Get it from llama.app.

Run it fully offline

1. Download the models you want while you’re online.
2. Turn off Ollama’s cloud features. Create ~/.ollama/server.json with:

{
  "disable_ollama_cloud": true
}

Or set OLLAMA_NO_CLOUD=1. Restart Ollama. The log then says Ollama cloud disabled: true.

1. Open WebUI: start it with OFFLINE_MODE=true. That stops update checks and model downloads. Set HF_HUB_OFFLINE=1 too.

OFFLINE_MODE=true HF_HUB_OFFLINE=1 open-webui serve

1. llama.cpp: add --offline. It then only uses models already on disk.
2. Pull the network cable or turn off Wi-Fi. Ask something. If you get an answer, it’s all local.

Ollama only listens on your own machine (127.0.0.1) by default. Leave it that way unless you know why you’d change it.

Privacy tips

• Use open-source tools. Ollama, llama.cpp and exllamav3 are MIT. Open WebUI’s code is public under its own license.
• Know which apps are closed. LM Studio is a popular free app, but the app itself is closed source. Only its lms command-line tool is open (MIT).
• Know the closed parts. Most model files are “open weights”: free to download and run, but you can’t see how they were trained. NVIDIA’s CUDA drivers are closed source too.
• Skip sign-ins. You don’t need an Ollama account to run models locally. Signing in is only for its cloud models.
• Turn off phoning home. Disable Ollama cloud and set Open WebUI’s offline mode, as in the last step. The Ollama Mac and Windows apps download updates on their own.
• Download models from the source. Use the Ollama library or the maker’s Hugging Face page. Avoid random re-uploads.
• Keep it on localhost. Don’t open the chat to the internet. If you want it on your phone, use your home network only.

Models still make things up. Check anything that matters, like health, money or law.

Common mistakes

1. Buying for memory size alone. Check how fast the memory is too, its bandwidth. That sets the speed (Mia).
2. Judging a machine by Ollama alone. The same box can be 2x faster with another engine (source).
3. Filling memory to the brim. Long chats and documents need room too. Leave a few GB free, more for long context.
4. Squeezing in a big model with a tiny file. Below Q4, most files lose a lot. Start at Q4_K_M or Q8. Some carefully made 3-bit files hold up, like Mia’s EAQ (Mia), but test them.
5. Copying tuning tips from forums. Jean-Claude tested two popular Strix Halo tweaks and saw no change (1, 2). Keep your kernel, drivers and engine up to date instead. That gave him real gains (Jean-Claude).
6. Thinking it got slow when several jobs run at once. Coding agents can start helpers. Four chats at once split the speed, 60 tok/s became 15–20 each (Jean-Claude).
7. Pulling a “cloud” model by accident. In Ollama, a tag with “cloud” runs on their servers. Check the tag.

Models change every few weeks. The starter models here are a starting point. Check the library for newer ones.

Keep it tidy

1. Update now and then. Ollama on Mac and Windows updates itself. On Linux, run the install script again.
2. Delete models you don’t use with ollama rm. They take a lot of disk space.
3. Models live in ~/.ollama/models on Mac, /usr/share/ollama/.ollama/models on Linux, and in your user folder under .ollama\models on Windows.
4. Don’t want Ollama running all the time? Turn it off in Login Items (Mac) or Startup apps (Windows).

Want a phone that leaks less too? Install GrapheneOS on a Pixel.