"Should we fine-tune?" is one of the most common questions we hear from engineering teams evaluating LLMs. The answer is usually "not yet" — but when fine-tuning is the right call, it can dramatically improve performance for domain-specific tasks.
Here's a practical guide based on our experience fine-tuning models for legal document analysis, recruitment screening, and enterprise knowledge systems.
When Fine-Tuning Is Not the Answer
Fine-tuning is expensive in time, data, and compute. Before you go there, try these in order:
1. Better Prompts
You'd be surprised how far good prompt engineering goes. Before fine-tuning:
- Write detailed system prompts with examples
- Use few-shot examples that cover your edge cases
- Structure your output format explicitly
- Add chain-of-thought reasoning for complex tasks
We estimate that 60-70% of the "fine-tuning" requests we get are actually prompt engineering problems.
2. RAG (Retrieval-Augmented Generation)
If the model doesn't know domain-specific facts, don't train them into the weights — retrieve them at inference time. RAG is cheaper, easier to update, and doesn't require retraining when information changes.
See our RAG architecture guide for details.
3. Tool Use / Function Calling
If the model needs to interact with systems (databases, APIs, calculators), use function calling rather than training the model to output system-specific commands.
When Fine-Tuning Is the Right Call
Fine-tune when you need:
Consistent output format. If the model needs to produce structured output in a specific schema every time — not 95% of the time — fine-tuning makes it reliable.
Domain-specific language understanding. When your domain uses terminology, abbreviations, or concepts that the base model frequently misinterprets. Legal language, medical terminology, and financial jargon are common examples.
Specific reasoning patterns. When the model needs to follow a domain-specific decision framework consistently. For LegalMind's document analysis, we fine-tuned to follow their specific risk classification framework — something prompt engineering couldn't make reliable.
Latency constraints. A fine-tuned smaller model can match a larger model's performance on a specific task while being 5-10x faster and cheaper to run. If you're processing thousands of requests per minute, this matters.
Cost at scale. Long system prompts with many few-shot examples get expensive at high volume. Fine-tuning moves that knowledge into the weights, reducing token usage per request.
The Fine-Tuning Process
Step 1: Curate Training Data
This is where most projects succeed or fail. You need:
- 50-200 high-quality examples as a minimum starting point
- Input-output pairs that represent the exact task you want the model to perform
- Edge cases — the examples that break prompt engineering are the most valuable training data
- Consistent labeling — if different annotators would produce different outputs for the same input, your model will too
Where to get training data:
- Expert annotations (expensive but highest quality)
- Historical human decisions (available in most businesses)
- LLM-generated examples reviewed by domain experts (efficient but requires careful quality control)
Step 2: Choose Your Approach
Full fine-tuning — Updates all model weights. Most expensive, most flexible. Use for major behavior changes.
LoRA / QLoRA — Updates a small number of additional parameters. 10-100x cheaper than full fine-tuning. Works well for most domain adaptation tasks. This is our default recommendation.
Prompt tuning — Learns a soft prompt prefix. Cheapest, but least powerful. Good for simple style/format adjustments.
For most business applications, LoRA on a 7B-13B parameter model hits the sweet spot of capability, cost, and inference speed.
Step 3: Train and Evaluate
The training loop:
- Split your data: 80% train, 10% validation, 10% test
- Train with LoRA on the training set
- Evaluate on the validation set after each epoch
- Stop when validation performance plateaus (typically 3-5 epochs for LoRA)
- Final evaluation on the held-out test set
Evaluation metrics matter. For classification tasks, use precision/recall/F1. For generation tasks, use a combination of:
- Automated metrics (ROUGE, BERTScore) for rough quality
- LLM-as-judge for nuanced quality assessment
- Domain expert evaluation on a sample (gold standard)
Never ship a model evaluated only on automated metrics. Human evaluation catches issues that metrics miss.
Step 4: Deploy and Monitor
Fine-tuned models need the same production infrastructure as any ML system:
- A/B testing against the base model + prompts to prove the fine-tuned version is actually better
- Monitoring for drift — if the input distribution changes, performance will degrade
- Retraining pipeline — as you collect more examples and corrections, retrain periodically
- Rollback capability — if the new model performs worse on a subset, you need to switch back quickly
What to Watch Out For
Catastrophic forgetting. Fine-tuning on a narrow task can make the model worse at general tasks. Use LoRA (which preserves base model capabilities) or include general-purpose examples in your training mix.
Overfitting to training format. If all your training examples follow the exact same structure, the model may struggle with inputs that deviate slightly. Include variation in your training data.
Evaluation contamination. If your test examples are too similar to your training examples, your metrics will be misleadingly good. Ensure test examples represent genuinely new inputs.
The "good enough" trap. Fine-tuning is additive — each improvement has diminishing returns. Define your accuracy target upfront and stop when you hit it. The jump from 92% to 95% often costs more than the jump from 80% to 92%.
Our Recommendation
For most businesses exploring LLM fine-tuning:
- Start with prompt engineering + RAG. This handles 70% of use cases.
- If prompt engineering plateaus, collect 100+ examples of where it fails.
- Fine-tune with LoRA on a 7B-13B model using those failure cases + general examples.
- Evaluate rigorously with domain experts, not just automated metrics.
- Deploy with A/B testing and monitoring.
- Retrain quarterly as you accumulate more examples.
The goal isn't the most sophisticated model — it's the most cost-effective solution that meets your accuracy requirements.
Considering fine-tuning for your use case? Let's evaluate the options →