Ollama from scratch, part 5: putting your own data into the model

This is the last of a five-part guide to running AI models locally with Ollama. Parts 1 to 4 covered installing, choosing, adding and serving models. This part is about making a model work with your own information.

The honest headline

Ollama does not train models. It runs them. Searching for "train Ollama on my data" is common, and the useful answer is that there are three different things people mean by it, at very different levels of effort. Try them in this order.

  What you need                          Approach            Effort
  ─────────────────────────────────────  ──────────────────  ────────────
  follow rules, answer in a set tone     Tier 1  Modelfile   seconds
  answer from your own documents         Tier 2  retrieval   an afternoon
  learn a narrow skill or rigid format   Tier 3  fine-tune   a GPU session

Tier 1: a Modelfile

The cheapest option, and enough surprisingly often. A Modelfile (introduced in part 3) can bake a system prompt and settings into a named model.

Create a file called Modelfile:

FROM llama3.1:8b

PARAMETER temperature 0.2

SYSTEM """
You are our support assistant. Answer only from company policy.
Be brief. If you are not sure, say so and suggest contacting a human.
"""

Build and run it:

ollama create support -f Modelfile
ollama run support

Now every conversation with support starts with those instructions. This handles "always reply in our voice", "follow these rules", "format answers like this". What it cannot do is carry a large body of knowledge: the system prompt is small, and a 200-page handbook will not fit in it. That is the next tier.

Tier 2: retrieval (RAG)

RAG, retrieval-augmented generation, means: when a question comes in, find the few relevant pieces of your data and paste them into the prompt alongside the question. The model answers from what it was handed. This is what most people who say "train on my data" actually want, and nothing gets trained.

The moving parts:

1. Split your documents into small chunks.
2. Turn each chunk into a vector (a list of numbers) with an embedding model.
3. Store the vectors.
4. Per question: embed the question, find the closest chunks, put them in the prompt.

Get an embedding model:

ollama pull nomic-embed-text

Here is the whole idea as a runnable script. It needs pip install ollama numpy and the two models pulled. No vector database, just a list and some arithmetic, so the mechanism is visible:

import ollama
import numpy as np

DOCS = [
    "Our refund window is 30 days from delivery.",
    "Support hours are 9am to 6pm IST, Monday to Friday.",
    "Enterprise plans include a dedicated account manager.",
    "Passwords must be at least 12 characters long.",
]

def embed(texts):
    resp = ollama.embed(model="nomic-embed-text", input=texts)
    return np.array(resp["embeddings"])

doc_vectors = embed(DOCS)

def answer(question):
    q = embed([question])[0]
    sims = doc_vectors @ q / (
        np.linalg.norm(doc_vectors, axis=1) * np.linalg.norm(q)
    )
    top = [DOCS[i] for i in sims.argsort()[::-1][:2]]
    context = "\n".join(f"- {line}" for line in top)
    prompt = (
        "Answer using only the context below.\n\n"
        f"Context:\n{context}\n\n"
        f"Question: {question}"
    )
    return ollama.generate(model="llama3.1:8b", prompt=prompt)["response"]

print(answer("How long do I have to send something back?"))

The question never uses the words "refund" or "30 days", but its vector lands near the refund sentence, so that chunk goes into the prompt and the model answers correctly.

For real use you would replace the list with your own files, chunked, and the arithmetic with a proper vector store. You rarely write that yourself: tools like Open WebUI, AnythingLLM, LlamaIndex and LangChain all do RAG and all can point at Ollama for both the embedding and the answer, so your documents never leave the machine.

Tier 3: actual fine-tuning

Fine-tuning changes the model's weights on your examples. Reach for it when the first two tiers cannot get a consistent enough result: a narrow skill, a rigid output format every single time, a very specific style. It needs hundreds to thousands of example pairs, and a GPU for the training run.

Ollama does not do this step. The common tools are:

• Unsloth — the beginner-friendly option, with free Colab notebooks you fill in and run.
• Axolotl — driven by a YAML config file, popular for repeatable runs.
• MLX-LM — fine-tunes on Apple Silicon directly.
• llama.cpp — has its own training utilities.

Most of these produce a LoRA adapter: a small file, tens to a few hundred megabytes, that layers on top of the base model rather than replacing it. A small LoRA run on a 7-to-8-billion model is often under an hour on a single rented or Colab GPU.

The path back into Ollama:

1. Fine-tune. You get a LoRA adapter, or a full set of merged weights.

2. Convert to GGUF with llama.cpp's convert_hf_to_gguf.py (adapters have a matching converter).

3. Write a Modelfile:

   FROM llama3.1:8b
   ADAPTER ./my-lora-adapter.gguf

   or, for merged weights:

   FROM ./my-merged-model.gguf

4. Build and run it like anything else:

   ollama create my-tuned-model -f Modelfile
   ollama run my-tuned-model

The fine-tuned model still has to fit in memory by the rules in part 2. A LoRA adapter adds almost nothing to the size.

Which tier for which problem

Start at the top. Most people never need tier 3, and trying it first is how weekends disappear.

What people get wrong

• Fine-tuning to add facts. It is bad at that and expensive. Facts are RAG.
• A tiny dataset. Ten or twenty examples change nothing. Fine-tuning wants hundreds at least.
• Skipping tier 1 and 2. A system prompt plus retrieval solves most of what "train on my data" is really asking for, in an afternoon, with no GPU.
• Forgetting the size rules. A tuned model is still a model and still has to fit in memory.

The series, in one place

1. Install Ollama and run your first model
2. Model names, pulling, and what will run
3. Adding models that are not in the library
4. Using Ollama as a server
5. Putting your own data into the model (this part)

For the deeper material on quantization, the KV cache, and working out exactly what fits, the longer post picks up where this series stops.