Fine-Tuning vs. Prompt Engineering vs. RAG: A Decision Framework from Production

"Should we fine-tune a model or just use prompt engineering?"

I get this question every week. And every week, the answer is the same: it depends on three things. Not ten things. Not "it's complicated." Three things.

I've deployed all three approaches in production — pure prompt engineering, retrieval-augmented generation, and fine-tuned models — for different clients with different needs. Here's the framework I use to decide, and the real-world trade-offs that tutorials don't mention.

The three questions

Before you write any code, answer these:

1. Is your task about knowledge or about behavior?

If your model needs to know specific facts — your product catalog, your documentation, your company's policies — that's a knowledge problem. RAG solves knowledge problems. You retrieve the relevant information and include it in the prompt. The model doesn't need to "know" anything; it just needs to reason over the context you provide.

If your model needs to behave a certain way — write in your brand voice, follow a specific output format, apply domain-specific reasoning patterns, classify things according to your taxonomy — that's a behavior problem. Fine-tuning solves behavior problems. You're teaching the model a new skill, not giving it new information.

Prompt engineering can handle both, up to a point. Simple knowledge tasks work with few-shot examples in the prompt. Simple behavior tasks work with careful instructions. But as complexity grows, prompt engineering hits a ceiling and you need RAG, fine-tuning, or both.

2. How often does the underlying information change?

If the information changes daily or weekly — product inventory, news, documentation updates — you need RAG. A fine-tuned model bakes knowledge into its weights at training time. If that knowledge changes after training, the model is wrong until you retrain. RAG sidesteps this entirely by retrieving current information at inference time.

If the information is relatively stable — classification taxonomies, output formats, reasoning patterns — fine-tuning works well because you're training on patterns that won't change frequently.

3. What's your latency and cost budget?

RAG adds latency. Every request requires a retrieval step (searching your vector database) before the generation step. In my experience, this adds 200-500ms to response time depending on your retrieval infrastructure. If your latency budget is tight, this matters.

Fine-tuning adds upfront cost but reduces per-request cost. A fine-tuned smaller model can often match a larger model's performance on your specific task, which means cheaper and faster inference. But the training cost — both compute and the engineering time to prepare training data — is significant.

Prompt engineering is the cheapest to start but the most expensive to scale. Those long system prompts with examples and instructions consume tokens on every single request. At high volume, the token cost of a 2000-token system prompt adds up fast.

When I use each approach

Prompt engineering only

For prototyping and validation. Every project starts here. If prompt engineering solves your problem well enough, ship it and move on. Don't over-engineer.

I also use prompt engineering as the permanent solution when the task is simple enough, the volume is low enough, and the maintenance burden of RAG or fine-tuning isn't justified. A startup processing 100 requests per day with a straightforward classification task doesn't need RAG or fine-tuning. A well-crafted prompt handles it.

The ceiling: when your prompt exceeds ~1500 tokens of instructions and examples, or when you find yourself constantly tweaking the prompt to handle edge cases, or when the model's behavior is inconsistent despite clear instructions — you've outgrown prompt engineering.

RAG

When the model needs access to information that changes, is too large to fit in a prompt, or is specific to a customer. The canonical use cases: question-answering over documentation, customer support with access to knowledge bases, any system where the model needs to reference specific documents.

The details that matter in production: your retrieval quality determines your system quality. I've seen teams spend weeks fine-tuning their generation model while their retrieval system returns irrelevant chunks. Fix retrieval first. If the right information isn't in the context window, no amount of generation quality will save you.

Chunking strategy matters more than embedding model choice. How you split your documents into chunks — by paragraph, by section, by semantic boundary — has a larger impact on retrieval quality than which embedding model you use. I typically spend more time on chunking strategy than on any other component of a RAG system.

Hybrid search (combining keyword search with vector search) outperforms pure vector search in almost every production system I've built. Vectors are great for semantic similarity, but sometimes the user's query contains a specific term that should match exactly, and vector search might not surface that exact match. A hybrid approach catches both.

Fine-tuning

When you need the model to consistently exhibit a specific behavior that it can't learn from instructions alone. The cases where I've seen the biggest wins:

Output format consistency. When your system needs to produce structured output in a specific schema, every time, without fail. Prompt engineering gets you to 90% consistency. Fine-tuning gets you to 99%.

Domain-specific reasoning. When the model needs to apply reasoning patterns that are specific to your industry — medical coding, legal analysis, financial classification — and the general model keeps falling back to generic reasoning.

Tone and voice. When "write in our brand voice" isn't specific enough as a prompt instruction, and the model keeps drifting to a generic tone. Fine-tuning on examples of your desired voice produces remarkably consistent results.

Cost optimization at scale. When you're processing high volume and can fine-tune a smaller model to match a larger model's performance on your specific task. The per-request savings compound quickly. This is the approach we used for an enterprise client — fine-tuned LLMs that delivered engagement and conversion lifts the business expected. Off-the-shelf models couldn't match the fine-tuned performance because the task was specific enough that general capability wasn't sufficient.

The combination that works best

In most production systems I build, the answer isn't one approach — it's a combination:

RAG provides the knowledge layer. The system retrieves relevant context for each request.

Fine-tuning provides the behavior layer. A fine-tuned model processes the retrieved context in a way that's consistent, domain-appropriate, and formatted correctly.

Prompt engineering provides the control layer. Even with RAG and fine-tuning, you still use a prompt to coordinate the system, handle edge cases, and provide request-specific instructions.

The three approaches aren't competing. They solve different problems. The decision framework isn't "which one" — it's "which combination, in what order, given your current constraints."

Start simple, add complexity when the metrics demand it

If I could tattoo one principle on every ML engineer's forehead, it would be this: start with the simplest approach that could work, measure whether it does, and add complexity only when the measurements tell you to.

Prompt engineering first. If the metrics aren't good enough, add RAG. If the behavior still isn't right, fine-tune. Each step adds cost and complexity. Only take the step when you have evidence that the current approach is insufficient.

The startups that ship the fastest are the ones that resist the urge to fine-tune before they've tried prompting, and resist the urge to build RAG before they've checked whether the model already knows the answer.

Manmeet Singh

Founder & CEO, AIshar Labs · Ex-Apple, Ex-Instacart · 15 AI Patents

Built ML systems at Apple (Search: Maps, Safari, Spotlight) and Instacart (Search, Recommendations, Ranking). Writes about production AI tradeoffs and system design.

Follow on LinkedIn →