RAG vs Fine-Tuning vs Custom LLM

Most business use cases are served best by retrieval-augmented generation (RAG) on top of a commercial or open-weight model, because the usual problem is that the model has never seen your documents, not that it lacks ability. Fine-tuning an LLM makes sense when you need consistent style, format or narrow specialist behavior that prompting cannot deliver, and you have enough reviewed examples to teach it. Training a custom LLM from scratch is rarely justified outside large organizations with unusual data and budgets to match. This guide defines the three options, compares them in one table, and walks through a decision path based on accuracy, data, privacy and cost.

What RAG, fine-tuning and custom LLMs actually change

Start with the baseline every option is measured against: a general-purpose model with a well-written prompt and the right context included. Many requests that arrive as “we need to fine-tune a model” are solved by better instructions and a few worked examples. Measure that first, then pay for anything below.

Retrieval-augmented generation (RAG)

RAG keeps the model unchanged and changes what it reads. When a user asks a question, a retrieval step searches your documents, database records or tickets, pulls the most relevant passages, and places them in the prompt. The model answers from that material and can cite it.

A RAG implementation has four moving parts: an ingestion pipeline that cleans and splits documents into chunks, an index (vector embeddings, keyword search or both), a retriever that ranks chunks for each query, and the generation step with instructions on how to use the retrieved text. Quality lives mostly in the first three. Poor chunking, stale indexes and weak ranking cause more wrong answers than the model does.

Fine-tuning an LLM

Fine-tuning adjusts the model’s weights by training it further on your examples: pairs of inputs and the outputs you want. Parameter-efficient methods such as LoRA make this practical on open-weight models, and most commercial providers offer hosted fine-tuning for their own models.

What fine-tuning changes well: tone, format, terminology, the handling of a narrow task, and how reliably the model follows a structure such as a JSON schema, a report layout or a fixed label set. What it changes poorly: facts. A fine-tuned model does not become a reliable store of your knowledge base, and whatever it learned is frozen until you retrain. LLM fine-tuning is a behavior tool, not a knowledge tool.

Custom LLM

A custom LLM means pretraining a model from scratch, or continuing pretraining on a large domain corpus. It requires data at a scale few organizations hold, a specialist team, dedicated compute, and an evaluation program to prove the result beats an off-the-shelf model with RAG. It makes sense in a handful of cases: languages or domains that public models cover badly, rules that forbid any external weights, or products where the model itself is the business. For everyone else, “custom LLM” in practice means a fine-tuned open-weight model that you host yourself.

Combinations are normal

The options stack. A common production pattern is RAG for knowledge plus a lightly fine-tuned small model for format and tone, wrapped in LLM integration work: authentication, logging, permission checks, and connections to the systems where the answers are used. Most of the engineering effort in LLM development goes into that wrapper, not into the model.

Comparison table: RAG vs fine-tuning vs custom LLM

Dimension	RAG	Fine-tuning	Custom LLM
What it changes	What the model reads at query time	How the model behaves	What the model is
Best for	Questions over your documents and data; material that changes often; answers that must cite sources	Consistent format, tone or labels; narrow repeated tasks; moving a task onto a smaller, lower-cost model	Domains or languages public models cover badly; strict no-external-weights rules; the model as the product
Data needed	Clean, permissioned documents; no labels	Hundreds to a few thousand reviewed input-output examples, plus a held-out test set	A very large curated corpus, plus everything fine-tuning needs
Keeps up with change	Yes, re-index the source	No, retrain	No, retrain at far higher cost
Traceability	High; answers can cite retrieved passages	Low; knowledge sits inside the weights	Low
Privacy exposure	Documents pass through the model at query time; controlled by hosting choice and retrieval permissions	Training data is embedded in the weights; the model file becomes a sensitive asset	Full control and full responsibility
Cost profile	Engineering up front, then per-query usage that grows with context size	Labeling, training runs and evaluation, then hosting a dedicated model	Compute, data, team and ongoing operations
Time to first useful result	Days to weeks	Weeks, mostly spent on data	Months or longer
Typical failure	Retrieval misses the right passage and the model answers anyway	Overfits to the examples; behavior drifts on inputs unlike the training set	Underperforms a public model after a large spend

Decision guide: six questions in order

1. Does better prompting fix it? Write the instructions properly, add three to five worked examples, give the model the full relevant context, and test on fifty real inputs. If the result is good enough, stop here.
2. Is the gap knowledge or behavior? If the model gives wrong facts about your products, policies or customers, it lacks knowledge: use RAG. If it knows enough but answers in the wrong shape, tone or label set, it lacks behavior: consider fine-tuning.
3. How often does the source material change? Price lists, policies, tickets and inventory change weekly or daily. RAG re-indexes; a fine-tuned model goes stale.
4. Must the answer show its sources? Legal, clinical, financial and compliance uses usually need citations a reviewer can check. That is RAG territory.
5. Do you need lower latency or cost at volume? A small model fine-tuned on one narrow task can replace a large general model for that task, cutting cost and response time. This is the strongest business case for fine-tuning.
6. Do you need something nobody else has? Only then does a custom LLM enter the conversation, and it should go through the same proof-of-concept discipline as any other AI investment.

Worked examples

• Staff assistant that answers questions from policies and procedures: RAG, with retrieval filtered by each user’s permissions.
• Customer support drafts in your brand voice: RAG for product facts plus a fine-tuned model for tone, or prompting with a style guide if volume is modest.
• Extracting fields from contracts into a fixed schema: start with prompting and schema-constrained output; fine-tune a small model once you have a few hundred reviewed extractions and need to cut cost.
• Ticket classification into twenty categories: fine-tune a small model, or use classic machine learning; a large general model is more than the task needs.
• An AI agent that books, updates and reports: RAG for context, tools for actions, and governance before autonomy. See AI Agents for Business: Use Cases and How to Build One Safely.
• A model for a specialty language with little public text: continued pretraining or a custom LLM, after a candid assessment of how much data you hold.

Privacy, data and cost: what each option exposes and what it consumes

Privacy and data governance

With RAG, your documents leave your systems at query time if the model is hosted elsewhere. Three controls matter: where the model runs (an API under terms that exclude training on your data, a private cloud deployment, or an open-weight model on infrastructure you control), what the retriever may return for each user (apply document permissions at retrieval, not afterwards), and what you index in the first place (do not index what you cannot protect).

With fine-tuning, your data becomes part of the model. You cannot easily remove one customer’s records later, memorized text can resurface in outputs, and the model file now needs the same protection as the data that trained it. Review the training set the way you would review a data export: classify it, minimize it, and record who approved it. A custom LLM carries all of this at larger scale.

Regulated data raises the bar. Health information needs a business associate agreement with any provider that touches it, and de-identification where the use case allows. The policy side is covered in Generative AI Governance: Policies, Controls and Guardrails and the data side in Data Governance for AI: The Prerequisite Nobody Wants to Fund. This is general information, not legal advice.

Cost

RAG costs are mostly engineering up front (ingestion, indexing, evaluation, integration) followed by per-query model usage that rises with the amount of retrieved text. Control it by retrieving fewer, better chunks, caching frequent answers, and routing simple questions to a smaller model.

Fine-tuning costs sit in data preparation and evaluation more than in the training run itself. Someone has to produce and review the examples, build a test set, and repeat the cycle when requirements change. Hosting a dedicated fine-tuned model adds a fixed monthly cost that only pays back at volume. A custom LLM multiplies every line of this and adds a specialist team.

Whatever you choose, budget for evaluation. A test set of real inputs with known good answers is the one asset that transfers between approaches, and it is the only way to know whether the change you paid for improved anything.

Where PS3G fits

PS3G builds LLM-based features for organizations of every size through our AI development service: RAG implementations over document stores, databases and ticket systems; fine-tuning of open-weight and hosted models with proper evaluation sets; and the integration work that connects a model to the applications your team already uses. We start with a short assessment that answers the questions above with your data, then a fixed-scope first version. Because we build and run our HIPAA-compliant EHR and telehealth platform, DocVilla, we design for permissioned retrieval and regulated data from the first sprint, and we add AI governance and data governance work when the use case calls for it.

Related reading

• AI Agents for Business: Use Cases and How to Build One Safely
• Generative AI Governance: Policies, Controls and Guardrails
• Data Governance for AI: The Prerequisite Nobody Wants to Fund
• AI development services

Frequently asked questions

Is fine-tuning an LLM the same as training one?

No. Pretraining builds a model from scratch on a very large corpus and is what model providers do. Fine-tuning takes an existing trained model and adjusts it on a comparatively small set of your examples so it behaves differently on your task. Fine-tuning is measured in hours or days of compute; pretraining in months.

Can RAG and fine-tuning be combined?

Yes, and in production they often are. RAG supplies current facts with citations, while a fine-tuned model improves how those facts are presented: the format, tone, length and label set. Build RAG first, measure it, and add fine-tuning only when the remaining errors are about behavior rather than missing information.

How much data do you need to fine-tune an LLM?

Less than most people expect for behavior changes, and more than they expect for quality. A few hundred carefully reviewed examples can shift format and tone on a modern model. The hidden requirement is a separate test set of real inputs with known good outputs, because without it you cannot tell whether the fine-tuned model is better or only different.

Does RAG stop hallucinations?

It reduces them when retrieval works, because the model answers from supplied text and can be instructed to say “not found” otherwise. It does not remove them. Retrieval can miss the right passage, chunks can lose context, and the model can still blend sources. Evaluation on real questions, with groundedness checks, is what keeps a RAG system reliable over time.

Which approach is safest for confidential data?

RAG on infrastructure you control, with retrieval filtered by user permissions, gives the most control for the least effort, and nothing is baked into the model. Fine-tuning embeds your data in weights that then need the same protection as the data itself. A custom LLM gives full control but places every safeguard on your team. Hosting choice and contract terms matter in all three.

Do we need a vector database for a RAG implementation?

Not always. Many RAG implementations run well on the full-text search your database or search engine already provides, or on a hybrid of keyword and vector search. A dedicated vector database earns its place when the corpus is large, semantic matching matters more than exact terms, or you need metadata filtering at scale. Decide from a test on your own documents, not from a product category.