Ollama from scratch, part 1: installing it and running your first model

This is part 1 of a five-part guide to running AI models on your own computer with Ollama. It assumes you have never done this before. Later parts cover choosing and sizing models, adding models Ollama does not ship, running it as a server, and using your own data.

If you have heard that you can run something like ChatGPT on your own laptop, with no account and no internet, that is true, and Ollama is the shortest way to get there. This guide takes you from nothing to a model answering questions on your machine, and then through everything you will actually want to do after that.

I wrote a more advanced piece on Ollama a while back, about quantization and working out whether a model fits in memory. This series is the on-ramp to it.

What Ollama actually is

Ollama is a free program that downloads AI models and runs them on your own computer. The model files sit on your disk. Once they are there, nothing you type goes anywhere.

Under the surface it is two pieces:

A command you type into, called ollama.
A small background service that does the real work. It loads models into memory, runs them, and also listens on http://localhost:11434 so other programs can talk to it. Part 4 is about that part.

When you install Ollama you get both. The menu-bar icon on macOS, or the system-tray icon on Windows, is just a convenient way to start and stop that background service.

Why run a model locally

Privacy. The text never leaves your machine. That matters for a config file, a contract, a medical letter, anything you would hesitate to paste into a website.
Cost. No subscription and no per-use fee.
Offline. It works on a plane or a bad hotel connection.
Learning. You can see the moving parts instead of a chat box.

One honest caveat before you start. A model you can run at home is smaller than the ones behind ChatGPT or Claude, and the gap on hard reasoning is real. Local models are very good at a large number of small, well-defined jobs, and noticeably weaker at the hard questions. Keep that in mind and you will not be disappointed.

Installing on macOS

The app. Go to ollama.com/download, choose macOS, and download the file. Open it, drag Ollama to your Applications folder, and launch it once. A small llama icon appears in the menu bar at the top of the screen. That is the background service running.

Homebrew, if you prefer:

brew install ollama
ollama serve

Either way, open Terminal and check it:

$ ollama --version
ollama version is 0.32.6

The app needs macOS 14 (Sonoma) or later.

The ollama.com download page showing macOS, Linux and Windows options — The download page detects your operating system. All three installers put the same ollama command on your PATH.

Installing on Windows

Go to ollama.com/download, choose Windows, and run OllamaSetup.exe. It does not ask for administrator rights. It installs into your user folder, starts straight away, and puts a llama icon in the system tray at the bottom-right of the screen. You may have to click the small arrow to see it.

The installer adds ollama to your PATH, so open a new PowerShell or Command Prompt window and check:

> ollama --version
ollama version is 0.32.6

Two notes. You do not need WSL; there was a time you did, and that time has passed. And Windows SmartScreen or your antivirus may pause on first launch. The installer is signed by Ollama.

Installing on Linux

One line, from the official instructions:

curl -fsSL https://ollama.com/install.sh | sh

That script downloads a single binary to /usr/local/bin/ollama, creates a system user called ollama, and installs a systemd service that starts on boot. If you have an NVIDIA or AMD graphics card with drivers already installed, it detects that too.

Check it:

$ ollama --version
ollama version is 0.32.6

$ systemctl status ollama
● ollama.service - Ollama Service
     Active: active (running)

If you would rather install by hand, or you need specific CUDA or ROCm notes, the Linux document linked above covers both.

Running your first model

One command:

ollama run qwen2.5:0.5b

qwen2.5:0.5b is a deliberately tiny model, about 400 MB, that runs on anything. The first time, Ollama downloads it. Downloading a model is called a pull, and you will watch it happen:

pulling manifest
pulling c5396e06af29: 100% ▕████████████████▏ 397 MB
verifying sha256 digest
writing manifest
success

>>> Send a message (/? for help)

At the >>> prompt, type a question and press Enter:

>>> In one sentence, what is a large language model?
A large language model can generate human-like text based on specific
rules and patterns in the data it receives.

>>> /bye

/bye, or Ctrl+D, leaves the chat. The model stays in memory for a few minutes afterwards in case you come back to it. Part 4 explains why, and how to change that.

When you are ready for something more capable, ollama run llama3.2 pulls a 3-billion-parameter model, about 2 GB, which is the usual real starting point. Anything you run that is not already on disk gets pulled first.

Where everything lives now

ollama list shows what you have downloaded:

$ ollama list
NAME              ID              SIZE      MODIFIED
qwen2.5:0.5b      a8b0c5157701    397 MB    2 minutes ago
llama3.2:latest   a80c4f17acd5    2.0 GB    5 minutes ago

ollama ps shows what is loaded into memory right now:

$ ollama ps
NAME            ID              SIZE      PROCESSOR    CONTEXT    UNTIL
qwen2.5:0.5b    a8b0c5157701    479 MB    100% GPU     4096       4 minutes from now

100% GPU under PROCESSOR is the result you want. If it says CPU, or splits like 40%/60% CPU/GPU, the model did not fit in graphics memory and fell back to the slower path. Part 2 is about avoiding that.

The model files themselves are in a hidden .ollama folder:

System	Location
macOS	`~/.ollama/models`
Linux	`/usr/share/ollama/.ollama/models`
Windows	`C:\Users\<you>\.ollama\models`

On Linux the path is under /usr/share/ollama because the background service runs as that ollama user, not as you.

Inside are two folders. blobs holds the actual weights, in large files named by a hash, shared between models that use the same layers. manifests records which blobs make up which model. You never touch these directly. To remove a model, use:

ollama rm qwen2.5:0.5b

which also clears the blobs nothing else needs.

What people get wrong on day one

Treating the app and the command as two separate things. They are one install. The tray or menu-bar app and the ollama command both talk to the same background service.
Expecting a chat window. Ollama is a command line and an API first. The desktop app is a launcher and a settings panel. If you want a graphical chat, part 4 shows how to connect one. Ollama has been adding a built-in chat window; if your version has one, good, but the command line is still where this series lives.
Pulling the biggest model first. A 70-billion-parameter model is a 40 GB download that then will not fit in memory on a normal machine. Start at 1 to 3 billion, work up, and read part 2 before the download.
Seeing Error: listen tcp 127.0.0.1:11434: bind: address already in use and thinking something broke. It means the background service is already running. That is the normal state.

Next in the series

Part 2 takes a model name apart. llama3.1:8b-instruct-q4_K_M has four separate pieces of information in it. It also explains why an 8-billion-parameter model is a 4.7 GB download and not a 16 GB one, and gives you a way to know, before you pull anything, whether it will run on your machine.