Ollama from scratch, part 3: adding models that are not in the library

This is part 3 of a five-part guide to running AI models locally with Ollama. Part 1 was install and first run, part 2 was choosing from the library. This part is about everything outside the library.

The Ollama library is curated and fairly small. The wider world of open models, most of it on Hugging Face, is much larger: fine-tuned variants, models for specific languages, brand-new releases that have not been added yet, and eventually the model you build yourself in part 5. All of it can run in Ollama. You just have to point Ollama at it.

What "compatible" means: GGUF

Ollama runs models stored in a file format called GGUF, which comes from the llama.cpp project. The rule of thumb:

• If a model is published as GGUF, Ollama can almost certainly run it.
• If it is only published as safetensors (the raw format most models are released in first), it needs converting before Ollama can use it.

On Hugging Face you can tell at a glance. Look for GGUF in the repository name or on its tags, or open the Files tab and look for .gguf files. Many popular models have a separate community GGUF repository, often from users like bartowski or TheBloke.

Method 1: pull straight from Hugging Face

This is the easy path and it needs nothing extra. Ollama can pull GGUF models directly from Hugging Face by putting hf.co/ in front of the repository name:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF

That pulls a default quantization. To ask for a specific one, add it after a colon, matching the quant name from the file list:

ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q4_K_M

This is an official feature, documented by both Ollama and Hugging Face. It works with any public GGUF repository.

Which quantization file do I pick?

A GGUF repo often has a dozen files. For a first choice:

• Q4_K_M — the safe default. Small, and the quality loss is hard to notice.
• Q5_K_M or Q6_K — a little larger, a little better, if you have memory to spare.
• Q8_0 — nearly the full-quality size. Rarely worth it over a bigger model at Q4.
• Q3_K_* or Q2_K — only if you are desperate for space. Below Q4 the model stays fluent but gets unreliable about specifics, which is a hard failure to spot. The detailed post explains why.
• IQ4_XS, IQ3_M and other IQ names — "i-quants". Similar size to the plain quants, slightly better quality, slightly slower. Fine to use.

Part 2's sizing still applies: the file size is roughly the memory it needs.

Method 2: import a GGUF file with a Modelfile

If you already have a .gguf file, or you want to keep it somewhere specific, you import it with a Modelfile. A Modelfile is a short text file, a bit like a Dockerfile, that describes how to build an Ollama model.

1. Download one .gguf file from a repository (the download icon next to the file).

2. In the same folder, create a file named Modelfile, with no extension, one line:

   FROM ./Llama-3.2-3B-Instruct-Q4_K_M.gguf

3. Build it:

   ollama create my-llama -f Modelfile

4. Run it like any other model:

   ollama run my-llama

You can add settings in the same file, which is the subject of part 5:

FROM ./Llama-3.2-3B-Instruct-Q4_K_M.gguf

PARAMETER temperature 0.3
SYSTEM """
You are a concise assistant. Prefer short answers.
"""

The chat template gotcha

A model needs a prompt template: the exact wrapping of special tokens that tells it where your message starts and stops. Most GGUF files carry their own template and Ollama uses it automatically.

Sometimes one does not, or carries a wrong one. The symptom is a model that replies with visible tokens like <|im_start|> or [INST], or one that never stops talking. The fix is to add a TEMPLATE line to your Modelfile, copied from the model's card on Hugging Face or from a known-good model of the same family. To see what a working library model uses as a reference:

ollama show --modelfile llama3.1:8b

That prints its full Modelfile, TEMPLATE included, which you can adapt.

What does not work, and how to read the error

A safetensors-only model. ollama create can import a folder of safetensors directly for common architectures:

FROM /path/to/model-directory

It is slower than GGUF and not every architecture is supported. If yours is not, you will see:

Error: unsupported architecture

The path then is to convert it to GGUF with llama.cpp's convert_hf_to_gguf.py, or to find a community GGUF version, which for any popular model usually exists within days.

A GGUF built for a newer llama.cpp than your Ollama. New model architectures land in llama.cpp first and reach Ollama a little later. The symptom is a pull that succeeds and a load that fails, often mentioning a version or an unknown key. The fix is to update Ollama: the macOS and Windows apps update themselves; on Linux re-run the install script from part 1.

A model too big for your machine. It pulls fine, then ollama run errors or crawls. Imported models are sized exactly like library ones, so check ollama ps and re-read part 2.

Check it worked

ollama list                  # your imported model is in here
ollama show my-llama         # architecture, parameters, quantization
ollama ps                    # after a run: GPU or CPU

What people get wrong

• Downloading every quant file. You need one. The rest are alternatives.
• Cloning the whole repository. git clone of a GGUF repo pulls every quantization, which can be 40 GB. Download the single file instead.
• A wrong FROM ./ path. The path is relative to where you run ollama create, not to the Modelfile. Run the command from the folder that holds the .gguf.
• Expecting safetensors to behave like GGUF. GGUF is the smooth path. Anything else is a conversion step first.

Next in the series

Part 4 is about the background service: calling Ollama's HTTP API from curl, Python and other tools, keeping a model loaded so the first request is not slow, running several models at once, and setting the environment variables that control all of it on each operating system.