Ollama from scratch, part 4: using Ollama as a server

This is part 4 of a five-part guide to running AI models locally with Ollama. Parts 1, 2 and 3 covered installing, choosing and adding models. This part is about the background service, which is where Ollama gets genuinely useful.

It was a server the whole time

Back in part 1 there were two pieces: the ollama command, and a background service. That service is an HTTP server listening on port 11434. Everything ollama run does, it does by talking to that server. So can your own scripts, a chat app, an editor plugin, or another computer on your network.

Check it is up:

curl http://localhost:11434

Ollama is running

  ollama run   ─┐
  your script  ─┼──►  Ollama server  ──►  model in memory  ──►  GPU or CPU
  a chat app   ─┘      (port 11434)

Three ways to call it

curl and the native API

The main endpoint is /api/chat. Set "stream": false so you get one JSON object back instead of a stream of them:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{ "role": "user", "content": "Say hello in one word." }],
  "stream": false
}'

There is also /api/generate for a single prompt with no conversation, and /api/embed for turning text into vectors (part 5 uses that one).

The OpenAI-compatible endpoint

Ollama also answers on /v1, in the same shape as the OpenAI API. Most code and tools written for OpenAI work by changing the base URL and passing any string as the key:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")

reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Say hello in one word."}],
)
print(reply.choices[0].message.content)

The official Python package

pip install ollama

import ollama

reply = ollama.chat(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Say hello in one word."}],
)
print(reply["message"]["content"])

All three hit the same server and the same model in memory.

Keeping a model loaded

The first request after a quiet spell is slow, because the model has to be read into memory first. After that, replies are fast. By default Ollama keeps a model resident for 5 minutes after the last request, then frees the memory.

To change that, set OLLAMA_KEEP_ALIVE on the server (see the environment section below), or pass keep_alive per request:

curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{ "role": "user", "content": "hi" }],
  "stream": false,
  "keep_alive": "30m"
}'

"30m" keeps it for thirty minutes, -1 keeps it until you unload it, 0 unloads it immediately. If you call Ollama in bursts through the day, raising this removes the reload wait. If you are short on memory, lower it.

Running several models at once

Ollama loads a model the first time a request names it, and can keep more than one resident at the same time. Two environment variables shape this:

• OLLAMA_MAX_LOADED_MODELS — how many models may be in memory together.
• OLLAMA_NUM_PARALLEL — how many requests one model handles at the same time.

The defaults adapt to your free memory, and for a single user they are usually fine. Set them explicitly when you want a predictable number.

The real limit is memory. Two 8-billion-parameter models resident is roughly 12 to 14 GB, because you are paying for both. If they do not both fit, Ollama unloads the least recently used one to make room, and that request pays the reload.

The common useful case is cheap: a chat model and a small embeddings model (nomic-embed-text is about 270 MB) loaded together, so a retrieval app can embed and generate without swapping.

Exposing it on your network

By default Ollama only listens on localhost, meaning the same machine. To let other devices reach it, bind to all interfaces:

OLLAMA_HOST=0.0.0.0

Ollama has no authentication of any kind. Anyone who can reach the port can use your models and read your prompts. Only do this on a home or otherwise trusted network. If you need it reachable more widely, put a reverse proxy with a password in front of it.

Setting environment variables, per OS

This is the step people get stuck on, because a variable set in your shell is not seen by the background service the app runs.

macOS (the app)

launchctl setenv OLLAMA_HOST "0.0.0.0"
launchctl setenv OLLAMA_KEEP_ALIVE "30m"

Then quit Ollama from the menu bar and open it again.

macOS or Linux (running ollama serve yourself)

Set the variable in the same shell before starting it, or prefix the command:

OLLAMA_KEEP_ALIVE=30m OLLAMA_HOST=0.0.0.0 ollama serve

Linux (the systemd service)

sudo systemctl edit ollama.service

Add this block, save, then restart:

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_KEEP_ALIVE=30m"

sudo systemctl restart ollama

Windows

1. Quit Ollama from the system tray.
2. Search the Start menu for Edit environment variables for your account.
3. Add a new variable, name OLLAMA_HOST, value 0.0.0.0. Add others the same way.
4. Start Ollama again. Open a new terminal window for the change to be visible there.

Endpoints worth knowing

The full request and response fields are in the API documentation.

What people get wrong

• Running ollama serve when the service is already up. The address already in use error means it is running, not broken.
• Forgetting "stream": false and being surprised by a stream of partial JSON objects.
• Setting a variable in a terminal and expecting the app's service to see it. The app reads its environment at launch. Use launchctl setenv, systemctl edit, or the Windows dialog, then restart Ollama.
• Binding to 0.0.0.0 on a shared network. There is no password. Localhost only, unless you trust everyone on the network or you added a proxy.

Next in the series

Part 5 is the one the search terms keep asking for: getting your own data into a model. It walks through the three approaches, from a one-line system prompt to retrieval to actual fine-tuning, and when each is the right one.