Training vs Inference vs Fine-Tuning: What Changes, What Doesn’t

Most confusion about “how AI works” comes from mixing up three very different phases of a model’s life:

1. Training: how the model’s behavior is created

2. Inference: how that behavior is used at runtime

3. Fine-tuning: how you change that behavior for a specific job

If you are a Product Manager, this is not academic. These phases map to different teams, budgets, failure modes, timelines, and risk profiles. They also map to different kinds of decisions you will be asked to make. Decisions like “Should we fine-tune?” or “Why is the model slow?” or “Why did accuracy improve but trust got worse?”

Let’s build a practical mental model first, then we will unpack what happens during pretraining vs inference, and where fine-tuning, instruction tuning, RLHF, distillation, and overfitting actually fit.

A simple mental model

Think of a model as a very large compressed library of patterns. Training is how the library gets written. Inference is how you consult it. Fine-tuning is how you add a specialized appendix and adjust what the librarian prioritizes.

That analogy is useful only if you keep one important constraint in mind: the model does not “look up facts” in a database during inference. It generates outputs by continuing patterns from the input. Sometimes those patterns align with reality. Sometimes they do not.

Once you separate “writing the library” from “consulting the library,” the rest becomes easier.

Part 1: What happens during pretraining (training) vs inference (runtime)

Pretraining: building general behavior from lots of text (and other data)

Pretraining is the initial large-scale training stage that makes a model broadly capable. In plain language, the model is exposed to vast amounts of data and learns to predict what comes next. The task is often described as “next token prediction,” but the product-relevant intuition is simpler:

• The model learns patterns of language, reasoning-like structures, and associations.

• It also learns good-enough heuristics for many tasks because many tasks are latent in the data (summarizing, translating, explaining, writing code, answering questions).

• It does not learn your product, your policies, or your customers unless those are in its training data.

From a PM perspective, pretraining is where the model learns its “general competence.” It is also where many limitations get baked in: gaps in knowledge, biases in data, tendency to produce fluent nonsense, and uneven performance across domains.

A key point that matters later: pretraining changes the model’s weights. The weights are the internal parameters that encode what the model has learned. Once pretraining is done, those weights are fixed unless you train again.

Pretraining is expensive and slow. Most product teams do not do it. They consume a pretrained model from a provider, either via API or by hosting an open model.

So why talk about it? Because many runtime issues are not runtime issues. They are “you are using a general model for a specialized job” issues. You need to know what pretraining gives you by default so you do not try to solve the wrong problem with fine-tuning.

Inference: using the model at runtime without changing its weights

Inference is what happens when your product sends input to the model and gets output back. No learning occurs (in the basic setup). The weights stay the same.

Inference is where product reality bites:

• Latency: How long the user waits.

• Cost: How much each request costs (money, compute, quotas).

• Reliability: Timeouts, rate limits, variance in response quality.

• Safety and compliance: How the system behaves with edge cases and adversarial prompts.

• Evaluation: How you measure that the experience is actually better.

Inference quality depends on more than the model. It depends on the whole system around it:

• The prompt and instructions

• The input context you supply (including retrieved documents)

• The tool calls you allow (search, calculators, databases)

• The output constraints you enforce (schemas, formats, refusal behavior)

• Post-processing and guardrails

• Your UI and the user’s expectations

This is why teams can take the same base model and ship wildly different products. Inference is not just “call model.” It is “design an interaction loop.”

What changes, what doesn’t

Here is the clean separation:

• Training and fine-tuning change the weights. They change the model itself.

• Inference does not change the weights. It changes the inputs, the context, and the runtime scaffolding.

If you remember only one thing: many problems that look like “the model is wrong” are actually “the system did not give it the right context, constraints, or tools at inference time.”

Part 2: Fine-tuning, instruction tuning, and RLHF, explained like you ship software

“Fine-tuning” is a loaded term. People use it to mean several distinct techniques. As a PM, you want crisp definitions so you can ask the right questions.

Fine-tuning: adapting a pretrained model to behave differently

Fine-tuning means taking a pretrained model and training it further on a narrower dataset, so it behaves better for a particular domain, task, style, or policy.

You can fine-tune for many reasons:

• Make outputs follow a strict format consistently

• Teach domain-specific jargon and conventions

• Improve performance on a narrow task (classification, extraction, structured summarization)

• Align the model with company policies and refusal rules (with caveats, discussed later)

• Reduce prompt length by baking instructions into the model

But fine-tuning is not magic. It can improve some behaviors and degrade others. It can also make your system harder to maintain.

Fine-tuning is a family. Two members matter most for PM conversations: supervised fine-tuning and preference-based methods like RLHF.

Supervised fine-tuning (SFT): learning from labeled examples

Supervised fine-tuning (SFT) means you provide example inputs and desired outputs. The model is trained to imitate those outputs.

Plain-language intuition: you are showing the model “when you see X, respond like Y.”

SFT is good for:

• Consistent tone and structure

• Task-specific transformations (extract fields from messy text, classify support tickets, generate templates)

• Learning domain conventions (legal clause style, medical note structure, internal taxonomy)

• Reducing prompt engineering complexity when the behavior is stable

SFT is not good for:

• Teaching new factual knowledge reliably (it can, but it is fragile and can cause unwanted side effects)

• Enforcing hard rules in adversarial settings (it helps, but it is not a security boundary)

• Fixing missing context (if the model needs data that changes daily, SFT is the wrong tool)

SFT also creates a common trap: teams fine-tune to force the model to “know” things that should live in a database or retrieval system. That usually ends in stale knowledge, brittleness, and expensive retraining cycles.

Instruction tuning: SFT focused on following instructions well

Instruction tuning is a type of supervised fine-tuning where training examples are framed as instructions and responses. The goal is to make the model better at following natural language directions.

Many modern “assistant-like” models have been instruction tuned. This is why they respond helpfully to prompts like “Summarize this” or “Write in a professional tone.”

From a PM standpoint, instruction tuning is what makes a base model usable as a product-facing assistant, instead of a raw text completer.

RLHF: aligning behavior using human preferences, not just “correct answers”

RLHF stands for Reinforcement Learning from Human Feedback.

The simplest product-level explanation:

1. Humans rank multiple model outputs for the same prompt (or label them as better or worse).

2. A “preference model” is trained to predict those rankings.

3. The main model is then trained to produce outputs that score better under that preference model.

You can think of RLHF as optimizing for “what humans prefer” under a chosen rubric, rather than pure imitation of a single target answer.

RLHF is often used to improve:

• Helpfulness and politeness

• Refusal behavior and safety constraints

• Reduction of obviously harmful outputs

• Less rambling, better conversational norms

It can also create tradeoffs:

• Over-refusal (the model declines benign requests)

• Overconfidence in a preferred tone (sounds right, not necessarily is right)

• A tendency toward generic, safe outputs when the reward signal punishes risk

If you have ever seen a model respond with cautious, HR-like language even when you wanted a decisive technical answer, you have met a reward-shaping artifact.

Important nuance: RLHF is not one thing. Implementations vary. There are also related methods that use preference optimization without full reinforcement learning. For PM purposes, the key is that preference methods shape behavior toward a policy objective, not just accuracy.

Part 3: Distillation, and why it often shows up in product roadmaps

Distillation is when you train a smaller model (the student) to mimic a larger model (the teacher).

Why it matters to product:

• Smaller models can be cheaper and faster at inference.

• They can run on more constrained hardware.

• They can be easier to host, scale, or deploy on-device.

The typical workflow looks like this:

• Use a large model to generate high-quality outputs (sometimes with chain-of-thought-like reasoning internally, sometimes with tool use).

• Create a dataset of inputs and those outputs.

• Train the smaller model with SFT to imitate them.

This is attractive when you already know what “good” looks like and you need to hit a latency or cost target.

Distillation tradeoff: you can compress behavior, but you can also compress mistakes and blind spots. If the teacher hallucinates in a systematic way, the student will learn that too unless your dataset and evaluation catch it.

Part 4: Overfitting, explained in a way that matters for shipping

Overfitting is when a model learns your fine-tuning dataset too specifically and fails to generalize.

In product terms, overfitting looks like this:

• It performs great in your test set, then disappoints in production.

• It becomes overly rigid, repeating phrases and templates even when they do not fit.

• It “memorizes” examples and struggles with slightly different inputs.

• It may leak sensitive patterns if your data is not handled correctly (this is a risk area to treat seriously).

Overfitting is more likely when:

• The fine-tuning dataset is small or repetitive.

• The task is narrow and the outputs are highly formulaic.

• You fine-tune too aggressively (too many steps, too high learning rate, too many epochs, depending on the method).

• Your evaluation is too close to your training distribution.

As a PM, you do not need the math. You need the discipline:

• Separate training, validation, and test sets properly.

• Evaluate on realistic, messy inputs.

• Include adversarial and edge cases.

• Watch for regressions in general capabilities that your product still relies on.

Part 5: When fine-tuning makes sense, and when it is just complexity

This is the decision point most teams struggle with. Fine-tuning can be the right move. It can also become a maintenance tax that never pays off.

The strongest reasons to fine-tune

Fine-tuning is most justified when you have a stable, repeated task where you can define “good” clearly.

Here are practical patterns where it often makes sense:

1) Format reliability is a product requirement
If your downstream systems require structured output (JSON, fields, tool call arguments), prompting alone can get you far, but it may still fail unpredictably. Fine-tuning can improve consistency, especially if you have many examples of tricky formatting cases.

2) The task is narrow and repeated at high volume
High volume changes the economics. If you are doing millions of similar requests, shaving tokens, reducing prompt length, or using a smaller distilled model can matter. Fine-tuning plus distillation sometimes becomes a cost and latency lever.

3) You have proprietary style or domain conventions
For example: insurance claim notes, clinical documentation patterns, internal compliance phrasing, or a company-specific taxonomy. Retrieval can supply facts, but it does not necessarily teach the model how to write in the correct “house style” while staying concise and consistent.

4) You need better performance on a well-defined transformation
Extraction, classification, routing, normalization, and templated writing are classic SFT wins, provided you can label examples reliably.

5) You can build and maintain a real dataset and evaluation harness
Fine-tuning without evaluation is gambling. If your organization can sustain data pipelines, labeling, red-teaming, and regression tracking, fine-tuning becomes a controllable engineering investment instead of a one-time experiment.

The weakest reasons to fine-tune

These are the common “sounds reasonable” arguments that often backfire:

1) “We want the model to know our latest policies and docs.”
If the information changes, putting it into weights is usually the wrong storage layer. Use retrieval (RAG) or tool calls to fetch the current truth. Then use prompting and guardrails to make the model cite and stay grounded.

2) “Prompting is annoying.”
Prompting can be messy, but it is also flexible and cheap to iterate. Fine-tuning trades iteration speed for baked-in behavior. If your understanding of the task is still evolving, you want flexibility.

3) “We need to reduce hallucinations.”
Fine-tuning can reduce hallucinations in some narrow settings. It can also introduce new hallucinations or increase confidence. The most reliable anti-hallucination strategy is often system design: retrieval, tool use, constraints, and UX that sets expectations.

4) “We want it to follow rules perfectly.”
Fine-tuning improves compliance, but it is not a hard guarantee. If a rule must never be violated, you need external enforcement (policy filters, permission checks, sandboxing, deterministic validators).

5) “We want the model to be safer.”
Fine-tuning can help align behavior, but safety is layered. You still need monitoring, abuse detection, refusal policies, and careful tool permissions. Treat fine-tuning as one ingredient, not the boundary.

A simple heuristic that works in practice

Before you fine-tune, ask:

• Can I solve this with better context at inference time?

• Can I solve this with better constraints (schemas, validators, tool gating)?

• Can I solve this with better UX (ask follow-ups, show sources, allow corrections)?

• Can I solve this with retrieval or tools instead of weights?

If the answer is “yes” to any of those, fine-tuning should be a later step, not the first.

Fine-tuning is best when your problem is not “the model lacks information.” It is “the model’s default behavior is not the behavior we want, even with good context.”

Part 6: A practical comparison across the axes PMs actually manage

When you are deciding between prompting, retrieval, fine-tuning, and distillation, it helps to name the axes explicitly.

Here is a compact tradeoff view. It is generalized, so you should validate with your specific model and traffic patterns.

This table hides a crucial reality: most successful products use a hybrid. Prompting plus retrieval plus validators is a strong baseline. Fine-tuning is a targeted optimization when the baseline is stable and still not good enough.

Part 7: Concrete examples (and the “right tool for the job” instinct)

Let’s make this less abstract with a few product scenarios.

Scenario A: Customer support assistant that must follow policy and cite sources

You want an assistant that answers customer questions and references the correct policy section.

Most teams should start with:

• Retrieval from the policy knowledge base

• Prompting that requires citations from retrieved passages

• Output formatting that includes quotes or links (or internal document IDs)

• A fallback behavior when retrieval confidence is low (ask clarifying questions or escalate)

Fine-tuning might help later for:

• More consistent tone and structure

• Better refusal patterns

• Better summarization of policy language

But if the policy changes weekly, you do not want to bake policy content into weights. You want the model to consult the latest source during inference.

Scenario B: Extracting fields from messy forms into a strict schema

You need name, address, product SKU, and issue category from messy user text. Errors break downstream automation.

Start with prompting plus strict schema validation. Add a retry loop that asks the model to fix invalid JSON.

If you still see a long tail of formatting errors or mis-mappings, SFT can be a good move because:

• The task is narrow and repeated.

• “Good” is definable.

• You can generate many labeled examples from historical data (with privacy review).

This is where fine-tuning tends to pay for itself.

Scenario C: A writing assistant that must match a brand voice

You want consistent tone and structure. Facts come from the user, not the model.

This is a good candidate for SFT if:

• You have many examples of brand-compliant writing.

• The style is stable.

• You need consistency at scale.

It is a weak candidate if:

• The brand voice is still evolving.

• You only have a handful of examples.

• You can get 80 percent of the way there with prompting and a style guide.

Scenario D: Internal analyst assistant that needs up-to-date numbers

If the assistant must answer with current metrics, fine-tuning is the wrong lever. You need tool use.

The product approach is:

• Permissioned connectors to data sources

• Tool calls that fetch and compute

• A response format that includes “data as of” timestamps and sources

• Guardrails that prevent guessing when data is missing

You might fine-tune to improve tool calling reliability and summarization, but the truth lives in the data systems, not in the model weights.

Part 8: What to ask your team when fine-tuning comes up

Fine-tuning proposals often arrive as “We should fine-tune to improve quality.” That is not a plan. Here are questions that force clarity without requiring ML math.

1) What exact behavior is failing today?

Ask for a failure taxonomy. Not just “hallucinations,” but concrete categories. For example: wrong field mapping, missed constraint, over-refusal, unsafe completion, tone mismatch, incorrect citations, tool misuse.

2) Can we reproduce it reliably?

If you cannot reproduce it, you cannot evaluate improvement. Push for a small suite of representative examples.

3) What is the baseline with prompting and retrieval?

Many “fine-tuning needed” claims evaporate after a week of better prompts, better retrieval, and validators.

4) What dataset will we train on, and how is it labeled?

If the dataset is “we will collect some,” you are looking at schedule risk. If the dataset is small, you are looking at overfitting risk. If the labels are inconsistent, you are looking at weird model behavior.

5) How will we evaluate, and what will we monitor in production?

Ask for:

• Offline test set results on realistic inputs

• A plan for regression testing

• Production monitoring signals (error rates, refusal rates, user corrections, escalations)

• A rollback plan

6) What will break when we update the base model?

This one is huge. If you fine-tune, you are now coupling yourself to a base model version. Provider updates might change behavior. Your fine-tune might need retraining. Your evals must catch drift.

Part 9: A grounded mental model for RLHF and “alignment” in products

RLHF is sometimes discussed like it magically makes models safe and truthful. In practice, it shapes behavior toward a preference objective. That objective can include truthfulness, but it is mediated through what labelers prefer and what the reward model learns.

Product implications:

• RLHF can make the assistant feel more helpful and consistent.

• It can also make the assistant more cautious, more generic, or more prone to refusing.

• It may reduce some classes of harmful outputs, but you still need external safeguards.

A useful way to frame it internally:

• SFT teaches “do this.”

• RLHF teaches “do what we like.”

Both can be valuable, but neither replaces product controls, monitoring, or clear UX boundaries.

Part 10: Putting it together (a pragmatic decision sequence)

If you are building an LLM feature and wondering where to invest, a practical sequence is:

1. Start with inference design
   Nail the user intent, the UX, and the workflow. Decide where the model adds value and where deterministic software should stay in charge.

2. Add grounding and constraints
   Retrieval for facts. Tools for dynamic data. Schemas and validators for structured outputs. Clear refusal rules enforced outside the model when needed.

3. Build evaluation early
   A small, curated test suite beats vibes. Add edge cases as you learn.

4. Only then consider fine-tuning
   Use it when you have stable targets, enough data, and a clear ROI on consistency, cost, or latency.

5. Consider distillation when economics demand it
   If a smaller model unlocks scale or latency, distillation can be the bridge, but only if your evaluation is strong.

This sequence is not dogma. It is a way to avoid paying the fine-tuning tax before you have earned the right to.

Closing thought: the question behind the question

When someone says “Should we fine-tune?”, they are often really asking one of these:

• “Can we make the model more predictable?”

• “Can we stop it from embarrassing us?”

• “Can we reduce cost and latency?”

• “Can we make it behave like our product, not like a generic chatbot?”

Fine-tuning is one answer. Often it is not the first.

If you keep the separation clear, training builds general behavior, inference applies it, fine-tuning reshapes it, you can reason about the tradeoffs like any other product system. Quality, latency, cost, risk, maintenance. The same five axes. The same discipline.

And that is the point. Models are new. Shipping is not.