# Run a private AI model at home

**Short answer:** Install Ollama, download an open model, and chat with it on your own computer. Your prompts stay at home. No account, no logs.

> Your prompts stay on your machine. No account, no logs.

- Level: Beginner
- Time: About 1 hour
- Web version: https://few.guide/guides/private-ai-at-home.html

## What you need

- **A computer with 16 GB of memory or more**: 8 GB works for small models. More memory means bigger, smarter models.
- **20 GB or more of free disk space**: One model file is often 5 to 20 GB. Big ones are 40 GB or more.
- **Internet, once**: To download the tool and a model. After that it can run offline.
- **About an hour**: Most of it is waiting for the download.

## Step 1. Is it for you?

A local AI model is a file on your computer. A free program runs it. You chat with it like an online chatbot, but nothing leaves your machine.

Good for:

- Questions you don’t want stored on someone else’s server.
- Reading and summing up your own files.
- Writing and coding help without an account.
- Making images on your own machine.
- Working with no internet at all.

Not for:

- Matching the biggest online models on a cheap machine. Small local models are less sharp.
- Old or low-memory machines. Below 8 GB it gets slow and dumb fast.
- Facts you can’t check. Local models make things up too.

Your prompts stay at home. No account, no logs on someone else’s server.

**Model** The file with the AI’s “brain”. You download it once, like a big app.

## Step 2. Memory: how much, and how fast

Two numbers decide what you can run.

**Capacity** decides which models fit. The model has to fit in memory, plus a few GB for your chat and the system.

**Bandwidth** decides how fast it writes. Each new word reads the model from memory again. So faster memory means more words per second. A DGX Spark has 273 GB/s. A Mac Studio M5 Ultra has about 1.2 TB/s. Both have lots of memory, but the Mac writes faster on the same model ([Mia, 7 Oct](https://x.com/MiaAI_lab/status/2107954065275842906)).

Model files come in sizes. Fewer bits per number means a smaller file and a little less quality. Here is one model, Qwen3.8 27B, in Ollama’s library:

| Version | File size | Use it when |
|---|---|---|
| BF16 (full) | 56 GB | You have memory to spare. Rarely worth it at home. |
| Q8 | 30 GB | Close to full quality. Good if it fits. |
| Q4_K_M | 18 GB | The usual pick. Small quality loss. |

Rough rule for Q4: about 0.6 GB per billion parameters. More examples:

| Your memory | Fits (Q4) | Example |
|---|---|---|
| 8 GB | Up to about 4B | **gemma4:e2b**, 4.6 GB. Tight. |
| 16 GB | Up to about 12B | **gemma4:12b**, 7.7 GB. |
| 24–32 GB | Up to about 30B | **qwen3.8:27b**, 18 GB. **gemma4:31b**, 19 GB. |
| 64 GB | Up to about 70B | **llama3:70b**, about 42 GB. |
| 128 GB | Up to about 120B | **gpt-oss:120b**, 65 GB, with room for long chats. |

Sizes are from the Ollama model library on 19 Oct 2026.

On a graphics card, only the card’s own memory (VRAM) counts at full speed. On a Mac or a shared-memory box, most of the total memory counts.

**Q4, Q8, GGUF** Q4 and Q8 are 4-bit and 8-bit versions of a model. GGUF is the file format Ollama and llama.cpp use. Other engines use other formats, like EXL3.

**Parameters** How big the model is. 27B means 27 billion. Bigger is smarter and slower.

## Step 3. Which machine should I buy?

Start with the computer you have. Buy only when you know what you want to run. Speeds are in tokens per second (tok/s) for one chat. They come from people who own the machines. Your numbers will differ.

- **The computer you have, 16 GB or more** ($0)
  - Good for: Chat, light coding
  - Starter model: **gemma4:12b**
  - Speed seen: Depends on the machine
  - Downsides: Old laptops are slow. On a Mac, the shared memory all counts.

- **NVIDIA RTX 3090, 24 GB** (Used, prices vary)
  - Good for: Chat, coding, images
  - Starter model: **qwen3.8:27b**
  - Speed seen: 40 tok/s in Ollama, 100 tok/s with exllamav3 ([one owner](https://x.com/ChafikMoalem/status/2101306380003233945))
  - Downsides: Bought used. Big, hot, needs a desktop PC. 24 GB caps model size.

- **NVIDIA RTX 5090, 32 GB** (About $5,000)
  - Good for: Fastest for 27–32B models, coding, images
  - Starter model: **qwen3.8:27b**
  - Speed seen: 113 tok/s prose, 275 tok/s code with exllamav3 ([Mia](https://x.com/MiaAI_lab/status/2108925156601053255))
  - Downsides: Expensive and often sold out. Needs a desktop PC. 32 GB caps model size.

- **Mini PC with AMD Ryzen AI Max+ 395 (“Strix Halo”), 128 GB** (From about $3,650)
  - Good for: Big models for chat and coding. Images, slower
  - Starter model: **gpt-oss:120b**
  - Speed seen: 42–45 tok/s on long chats, 60+ normally, Qwen3.8 Flash on Halogen ([Jean-Claude](https://x.com/jauneclaude/status/2104552077133533517), [14 Oct](https://x.com/jauneclaude/status/2110311178438430770))
  - Downsides: Best on Linux. More setup. Slower than a big card for images ([Jean-Claude](https://x.com/jauneclaude/status/2111172649435238844)).

- **NVIDIA DGX Spark, 128 GB** (About €6,000)
  - Good for: Big models, long context, several users
  - Starter model: **nemotron-3.5-bolt**
  - Speed seen: 71 tok/s Nemotron 3.5 Bolt, 46–55 tok/s Qwen3.8 Flash Next ([Mia](https://x.com/MiaAI_lab/status/2111103752824385843), [17 Oct](https://x.com/MiaAI_lab/status/2111540267840901174))
  - Downsides: Lower bandwidth, so single-chat speed is modest. Best numbers need special engines.

- **Mac Studio M5 Ultra, 256 GB** (About $10,000)
  - Good for: The biggest models on one quiet box
  - Starter model: **qwen3.8-flash-next**
  - Speed seen: 28 tok/s on GLM 5.3 Flash, a very large model ([MacStories via Mia](https://x.com/MiaAI_lab/status/2102098618035356156))
  - Downsides: Price. Slower than two Sparks at reading long prompts on that test.
Prices: RTX 5090, Strix Halo and DGX Spark from our tracker on 19 Oct 2026. The Mac price is [Mia’s figure](https://x.com/MiaAI_lab/status/2106457257160417386). Used card prices vary a lot.

**Pick by what you’ll do:**

- **Chat and writing:** any row works. Bigger memory means a smarter model.
- **Coding:** speed matters, because code answers are long. A 24–32 GB NVIDIA card running a 27B model is the sweet spot.
- **Images:** an NVIDIA card is fastest. Shared-memory boxes work, just slower. On Strix Halo, one 1-megapixel image took 16–18 seconds ([Jean-Claude](https://x.com/jauneclaude/status/2111738095125315812)). Mia names the Mac and the DGX Spark for media work too ([Mia](https://x.com/MiaAI_lab/status/2111148596649185375)).

**These boxes keep selling out and going up in price.** Memory is scarce this year. Check stock and prices before you buy.

We track who has them, in the US, Europe and Asia: [AI hardware tracker](/ai-parts.html).

**Tokens per second** How fast the model writes. A token is about three quarters of a word. About 6 tok/s keeps up with your reading. Faster helps for long code answers.

## Step 4. Install Ollama

Ollama is a free, open-source program (MIT license) that downloads and runs models.

**Mac**
- Go to ollama.com/download and download the Mac app.
- Open the file and drag Ollama to Applications.
- Open Ollama. It asks to add the `ollama` command. Say yes.

**Windows**
- Go to ollama.com/download and download the Windows installer.
- Run it. Keep the default options.
- Open Ollama from the Start menu.

**Linux**
- Open a terminal.
- Run the official install script:

```
curl -fsSL https://ollama.com/install.sh | sh
```
It installs Ollama as a service that starts on boot.

Prefer to read a script before you run it? Open ollama.com/install.sh in your browser first.

## Step 5. Run your first model

- Open a terminal (Mac: Terminal. Windows: PowerShell).
- Download a model and start a chat:

```
ollama run gemma4
```
Bigger machine? Use the starter model from step 3 instead, for example `ollama run qwen3.8:27b`.

- The first time, it downloads the file. That can take a while.
- Type a question and press Enter.
- Type `/bye` to quit.

Handy commands:

```
ollama list        # what you have
ollama ps          # what is loaded right now
ollama pull gpt-oss:20b
ollama rm gemma4   # delete a model
```
**Skip any model tagged “cloud”.** Those run on Ollama’s servers, so your prompts leave your machine. Local tags look like `gemma4:12b`.

## Step 6. Add a chat window: Open WebUI

Open WebUI gives you a ChatGPT-style page in your browser that talks to Ollama on your machine.

**pip**
Needs Python 3.11.

```
pip install open-webui
open-webui serve
```
Then open http://localhost:8080.

**Docker**

```
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main
```
Then open http://localhost:3000.

- Create the first account. It’s a login on your own machine, and it becomes the admin.
- Pick your model at the top and chat.

Open WebUI is source-available under its own license, with a rule to keep its branding. The code is public. It is not a standard open-source license.

## Step 7. Faster engines: llama.cpp and more

Ollama is the easy start. It’s not the fastest. The engine that runs the model matters almost as much as the model ([Mia](https://x.com/MiaAI_lab/status/2111820702160736747)). One RTX 3090 owner went from 40 to 100 tok/s on the same model by moving from Ollama to exllamav3 ([source](https://x.com/ChafikMoalem/status/2101306380003233945)).

**llama.cpp** is the engine many tools are built on. It’s open source (MIT) and has its own simple web chat built in. It runs on NVIDIA, AMD, Apple and plain CPUs. On AMD it also has a Vulkan build, so you can skip AMD’s ROCm drivers.

- Install it. On Mac or Linux:

```
brew install llama.cpp
```
Or use the official script, or `winget install llama.cpp` on Windows:

```
curl -LsSf https://llama.app/install.sh | sh
```
- Start a small model and the built-in web chat:

```
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
```
- Open the address it prints in your browser. The default port is 9931.

Command not found? Older installs call it `llama-server` instead of `llama serve`.

Have a model file already? Point at it with `-m`:

```
llama-server -m ~/models/model.gguf --port 8080
```
**Once you know your machine,** these are worth a look:

- **NVIDIA card, 16–32 GB:** exllamav3 (MIT) with EXL3 files. Mia keeps one-click recipes for Qwen3.8 27B on 16–24 GB cards ([Mia](https://x.com/MiaAI_lab/status/2101259705154736639)).
- **DGX Spark:** vLLM or SGLang (both Apache 2.0), or TensorFold (Apache 2.0).
- **Strix Halo:** Jean-Claude’s OmaRed scripts (MIT) set up Linux, kernel flags and the Halogen Flash engine for Qwen3.8 Flash ([Jean-Claude](https://x.com/jauneclaude/status/2105700798328107211)). We couldn’t confirm Halogen’s own license, so check it first.
- **Mac:** MLX builds, which Ollama also offers as “-mlx” tags.

Prefer an app? The llama.cpp team and Hugging Face make Llama, a small open-source app for Mac and Windows. Get it from llama.app.

## Step 8. Run it fully offline

- Download the models you want while you’re online.
- Turn off Ollama’s cloud features. Create `~/.ollama/server.json` with:

```
{
  "disable_ollama_cloud": true
}
```
Or set `OLLAMA_NO_CLOUD=1`. Restart Ollama. The log then says `Ollama cloud disabled: true`.

- Open WebUI: start it with `OFFLINE_MODE=true`. That stops update checks and model downloads. Set `HF_HUB_OFFLINE=1` too.

```
OFFLINE_MODE=true HF_HUB_OFFLINE=1 open-webui serve
```
- llama.cpp: add `--offline`. It then only uses models already on disk.
- Pull the network cable or turn off Wi-Fi. Ask something. If you get an answer, it’s all local.

Ollama only listens on your own machine (127.0.0.1) by default. Leave it that way unless you know why you’d change it.

## Step 9. Privacy tips

- **Use open-source tools.** Ollama, llama.cpp and exllamav3 are MIT. Open WebUI’s code is public under its own license.
- **Know which apps are closed.** LM Studio is a popular free app, but the app itself is closed source. Only its lms command-line tool is open (MIT).
- **Know the closed parts.** Most model files are “open weights”: free to download and run, but you can’t see how they were trained. NVIDIA’s CUDA drivers are closed source too.
- **Skip sign-ins.** You don’t need an Ollama account to run models locally. Signing in is only for its cloud models.
- **Turn off phoning home.** Disable Ollama cloud and set Open WebUI’s offline mode, as in the last step. The Ollama Mac and Windows apps download updates on their own.
- **Download models from the source.** Use the Ollama library or the maker’s Hugging Face page. Avoid random re-uploads.
- **Keep it on localhost.** Don’t open the chat to the internet. If you want it on your phone, use your home network only.

**Models still make things up.** Check anything that matters, like health, money or law.

## Step 10. Common mistakes

- **Buying for memory size alone.** Check how fast the memory is too, its bandwidth. That sets the speed ([Mia](https://x.com/MiaAI_lab/status/2107954065275842906)).
- **Judging a machine by Ollama alone.** The same box can be 2x faster with another engine ([source](https://x.com/ChafikMoalem/status/2101306380003233945)).
- **Filling memory to the brim.** Long chats and documents need room too. Leave a few GB free, more for long context.
- **Squeezing in a big model with a tiny file.** Below Q4, most files lose a lot. Start at Q4_K_M or Q8. Some carefully made 3-bit files hold up, like Mia’s EAQ ([Mia](https://x.com/MiaAI_lab/status/2110757642461126947)), but test them.
- **Copying tuning tips from forums.** Jean-Claude tested two popular Strix Halo tweaks and saw no change ([1](https://x.com/jauneclaude/status/2102035623976870332), [2](https://x.com/jauneclaude/status/2102035626782855319)). Keep your kernel, drivers and engine up to date instead. That gave him real gains ([Jean-Claude](https://x.com/jauneclaude/status/2103436459223253330)).
- **Thinking it got slow when several jobs run at once.** Coding agents can start helpers. Four chats at once split the speed, 60 tok/s became 15–20 each ([Jean-Claude](https://x.com/jauneclaude/status/2110311178438430770)).
- **Pulling a “cloud” model by accident.** In Ollama, a tag with “cloud” runs on their servers. Check the tag.

Models change every few weeks. The starter models here are a starting point. Check the library for newer ones.

## Step 11. Keep it tidy

- Update now and then. Ollama on Mac and Windows updates itself. On Linux, run the install script again.
- Delete models you don’t use with `ollama rm`. They take a lot of disk space.
- Models live in `~/.ollama/models` on Mac, `/usr/share/ollama/.ollama/models` on Linux, and in your user folder under `.ollama\models` on Windows.
- Don’t want Ollama running all the time? Turn it off in Login Items (Mac) or Startup apps (Windows).

Want a phone that leaks less too? [Install GrapheneOS on a Pixel](grapheneos-pixel.html).

## Sources

- [Ollama FAQ](https://docs.ollama.com/faq): cloud off (server.json, OLLAMA_NO_CLOUD), localhost 127.0.0.1:11434, model folders, updates, login items
- [Ollama CLI](https://docs.ollama.com/cli): run, pull, list, ps, rm
- [Ollama on macOS](https://docs.ollama.com/macos): macOS 14 or newer, Apple silicon
- [Ollama on Windows](https://docs.ollama.com/windows): Windows 10 22H2, NVIDIA and AMD
- [Ollama on GitHub](https://github.com/ollama/ollama): install commands, MIT license
- [Ollama library](https://ollama.com/library/gemma4/tags): gemma4 and gpt-oss file sizes
- [llama.cpp on GitHub](https://github.com/ggml-org/llama.cpp): install script, llama serve -hf, MIT license
- [llama.cpp server](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md): -m, --offline, --port, default port 9931, built-in web UI
- [llama.cpp install](https://github.com/ggml-org/llama.cpp/blob/master/docs/install.md): Homebrew, Winget
- [llama.app](https://llama.app/): the Llama app by the llama.cpp team and Hugging Face, open source, works offline
- [Open WebUI on GitHub](https://github.com/open-webui/open-webui): pip and Docker install, Python 3.11, HF_HUB_OFFLINE, license
- [Open WebUI env config](https://docs.openwebui.com/reference/env-configuration/): OFFLINE_MODE, update check
- [exllamav3 on GitHub](https://github.com/turboderp-org/exllamav3): MIT license
- [TensorFold on GitHub](https://github.com/ashhart/TensorFold): Apache 2.0 license
- [SGLang on GitHub](https://github.com/sgl-project/sglang): Apache 2.0 license
- [vLLM on GitHub](https://github.com/vllm-project/vllm): Apache 2.0 license
- [OmaRed Strix Halo scripts on GitHub](https://github.com/claudejaune/Qwen3.8-Halogen-Flash-Strix-Halo): MIT license
- [LM Studio app terms](https://lmstudio.ai/app-terms): closed source. The lms CLI is MIT (github.com/lmstudio-ai/lms)
- [Ollama library](https://ollama.com/library/qwen3.8/tags): qwen3.8:27b sizes, BF16 56 GB, Q8 30 GB, Q4_K_M 18 GB
- [Mia on X, 7 Oct](https://x.com/MiaAI_lab/status/2107954065275842906): DGX Spark 273 GB/s vs M5 Ultra 1.2 TB/s
- [Mia on X, 10 Oct](https://x.com/MiaAI_lab/status/2108925156601053255): Qwen3.8 27B on RTX 5090 with exllamav3, 113 tok/s prose, 275 code
- [RTX 3090 owner on X, 19 Sep, reposted by Mia](https://x.com/ChafikMoalem/status/2101306380003233945): Ollama Q4_K_M 40 tok/s, EXL3 100 tok/s
- [Mia on X, 16 Oct](https://x.com/MiaAI_lab/status/2111103752824385843): Nemotron 3.5 Bolt on one DGX Spark, 71 tok/s, about 52 GB
- [Mia on X, 17 Oct](https://x.com/MiaAI_lab/status/2111540267840901174): Qwen3.8 Flash Next on one DGX Spark, 46-55 tok/s
- [Mia on X, 21 Sep](https://x.com/MiaAI_lab/status/2102098618035356156): M5 Ultra vs 2x DGX Spark on GLM 5.3 Flash, from MacStories
- [Mia on X, 3 Oct](https://x.com/MiaAI_lab/status/2106361199416029253): the 256 GB M5 Ultra as the best buy, about $10k
- [Mia on X, 18 Oct](https://x.com/MiaAI_lab/status/2111820702160736747): the engine matters almost as much as the model
- [Mia on X, 15 Oct](https://x.com/MiaAI_lab/status/2110757642461126947): 3-bit EAQ quant of Qwen3.8 27B, 12 GB
- [Jean-Claude on X, 28 Sep](https://x.com/jauneclaude/status/2104552077133533517): Strix Halo, 42-45 tok/s at 262k context
- [Jean-Claude on X, 14 Oct](https://x.com/jauneclaude/status/2110311178438430770): 60+ tok/s, split across 4 parallel jobs
- [Jean-Claude on X, 18 Oct](https://x.com/jauneclaude/status/2111738095125315812): 1 MP image in 16-18 s on Strix Halo
- [Jean-Claude on X, 21 Sep](https://x.com/jauneclaude/status/2102035623976870332): tuning tweaks that made no difference
- [Jean-Claude on X, 25 Sep](https://x.com/jauneclaude/status/2103436459223253330): kernel and llama.cpp updates raised speed
- [Jean-Claude on X, 1 Oct](https://x.com/jauneclaude/status/2105700798328107211): OmaRed scripts for Strix Halo

Speeds and buying advice are from public posts on X by Mia (MiaAI_lab) and Jean-Claude (jauneclaude), and one owner report, linked in each step. They don’t review or endorse this guide. Tools checked against docs.ollama.com (FAQ, CLI, macOS, Windows, Linux), ollama.com/library, the Ollama and llama.cpp READMEs on GitHub, the llama.cpp server README, llama.app, and the Open WebUI README and docs.openwebui.com, on 19 Oct 2026. Tools and commands change. Check the official docs if something looks different.