Agent memory vs RAG vs fine-tuning: what your team actually needs

Most teams that want AI to "know the company" need retrieval plus a small amount of reviewed memory, not fine-tuning. Fine-tuning changes how a model behaves. Retrieval (RAG) and memory change what it knows at the moment it answers, and that is what most work questions depend on.

The three terms get used loosely, and vendors blur them. This post explains what each one does, where each one breaks, and how to pick for a real team.

The short answer

	What it changes	Good for	Bad for
Agent memory	What the agent carries between sessions	Preferences, standing rules, short team conventions	Large sets of documents, facts that change often
RAG (retrieval)	What the model reads right before it answers	Docs, policies, tickets, past decisions	Tone and format, tasks that need every document at once
Fine-tuning	The model's weights, so its default behavior	Consistent format, a narrow task repeated at high volume	Facts that change, citing sources, permissions

If you remember one line: put facts in retrieval, put behavior in prompts first and fine-tuning last, and keep memory small and reviewed.

What agent memory is

Agent memory is information an AI tool keeps between conversations so you don't repeat yourself. In chat products it often looks like a list of saved facts: your role, your writing preferences, the project you're on. In coding agents it is often a file the agent reads at the start of each session. Claude Code, for example, loads CLAUDE.md files at the start of every session and treats them as context, not enforced configuration (Claude Code docs).

Memory works for things that are short, stable and apply everywhere:

"We write in US English and never use em dashes."
"Prices are quoted in USD, excluding tax."
"Agents have read-only access to the staging database."

Memory gets risky as it grows. Each tool keeps its own, so what ChatGPT remembers about your pricing is not what Cursor or your custom agent knows. Memories also go stale quietly. A saved note that says "the Pro plan is $49" keeps showing up after you change the price, and nothing in the tool flags it. We cover how the main chat tools handle this in AI memory explained.

What RAG is

RAG stands for retrieval-augmented generation. The term comes from a 2020 research paper by Lewis et al. In practice it means: before the model answers, a system searches your sources, pulls the most relevant passages, and puts them in the prompt. The model then answers from what it just read.

For a team, this is the workhorse. Ask "what did we decide about annual discounts?" and a retrieval system can find the pricing doc, the Slack thread and the decision record, then answer with a link to each.

RAG works well when:

The facts change often (prices, policies, roadmaps, who owns what).
You need sources, so a person can check the answer.
Different people should see different things. Retrieval can filter by permission before anything reaches the model.

RAG fails in predictable ways. If the source is wrong, the answer is wrong, with a citation. If two documents disagree, the model may pick either one. If the search step misses the right passage, the model falls back on general knowledge. In my experience, many "the AI made it up" complaints about retrieval systems trace back to bad or conflicting sources, not the model.

What fine-tuning is

Fine-tuning trains an existing model further on your own examples, so its default behavior shifts. OpenAI's guide describes it as a way to get consistent formatting, handle a wider range of inputs than fits in a prompt, cut token costs with shorter prompts, and let a smaller, cheaper model do well on a specific task (OpenAI model optimization guide). The same guide recommends writing evals and working on prompts first, and notes that prompting may be all you need.

Fine-tuning is a poor fit for company knowledge for three reasons:

It goes stale. When your refund policy changes, the model keeps the old one until you retrain.
It can't cite. The model can't point to the doc a fact came from, so nobody can check it.
It ignores permissions. Once salary bands are in the weights, anyone who uses the model can get at them.

Where fine-tuning does fit: a support team sending thousands of replies a week in a strict format, or a product that classifies the same kind of ticket over and over. Those are narrow, repeated tasks with stable rules. Few internal teams have one.

How to choose for a real team

Start from the question people ask, not the technique. Common requests and what usually fits:

"Write release notes in our style." A prompt with two or three past examples. If several people use it, turn it into a shared skill. No fine-tuning needed.
"What's our current price for annual plans?" Retrieval over the pricing doc, with a link. Not memory, because it changes.
"Remember that I prefer bullet points." Personal memory.
"Why did we drop the free tier?" Retrieval over a decision log. If you don't have one, no technique will help.
"Tag 20,000 support tickets a month into 12 categories." A candidate for fine-tuning, after you've tried prompting and measured the results.

A quick test: if the answer would change when someone edits a doc, it belongs in retrieval. If it's a rule about how to behave that rarely changes, it belongs in a prompt or memory. If it's a high-volume task where prompting works but costs too much, consider fine-tuning.

The part most teams miss: the source

All three techniques depend on the same thing: a clean, current record of what your team knows and has decided. Memory copies bits of it into one tool. RAG reads from it at answer time. Fine-tuning freezes a snapshot of it.

So before you pick a technique, check the source:

Is there one place where current decisions live, with dates and owners?
When something changes, does the old version get marked as replaced?
Do permissions on the source match who should see the answer?
Can every AI tool your team uses reach it, or only one?

The last question matters more each year. The average company now runs 27 AI-powered SaaS apps (BetterCloud). If each one keeps its own memory, you get many slightly different versions of your company. We go deeper on this in context engineering for teams.

FAQ

Is RAG better than fine-tuning?

For company knowledge, usually yes. Retrieval stays current, cites sources and can respect permissions. Fine-tuning is better for changing a model's format or behavior on a narrow, repeated task.

Can I use memory and RAG together?

Yes, and most good setups do. Keep memory for short standing rules and preferences. Use retrieval for anything that comes from a document and might change.

Does fine-tuning teach a model new facts?

It can change what a model says, but it's a weak way to store facts. You can't easily update one fact, you can't see where an answer came from, and you can't limit who sees it. Use retrieval for facts.

What should a small team do first?

Write down current decisions, policies and how-tos in one place, keep them dated, and make them reachable by the AI tools you already use. That does more than any technique choice.

The bottom line

Memory, RAG and fine-tuning solve different problems. Most teams need retrieval over a well-kept source, a little reviewed memory and good prompts. Few need fine-tuning. The hard part is the source, not the technique. We're building Shane to be that source: one place for your team's context and decisions, handed to every AI tool with sources and permissions. If that's a problem your team has, you can join the waitlist at tryshane.ai.