I Fine-Tuned the Model. The Baseline Still Won.

I trained three adapters and kept the system that did not train at all.

The constrained retrieval baseline scored 0.5647 on the acceptance evaluation. LoRA scored 0.4522. DoRA scored 0.4638. A revised retrieval-augmented fine-tune reached 0.4803.

That was enough to make the deployment decision. It was not the most important result.

The important result was that changing the serving path moved the same untrained model by 0.103 composite points. If I had ignored that, I could have credited training for a runtime effect or blamed training for a broken comparison.

The decision was written before the run

The goal was not “make fine-tuning look useful.” The goal was to decide whether an adapter should replace an existing constrained retrieval system.

That distinction changes the experiment. A training loss can improve while the product gets worse. A composite score can improve while the failure you care about becomes unacceptable. The adapter does not ship because it learned something. It ships only if it clears the deployment gates.

The evaluation compared:

Every trained variant missed the baseline.

The complete public aggregate is in the DGX Spark evidence repository. The source corpus, held-out examples, prompts, and raw responses are not public. The useful claim is the aggregate decision, not the text the system was trained to handle.

The false-refusal failure was worse than the score

Composite scores compress several behaviors into one number. That makes them useful for comparison and dangerous for a deployment.

The adapter family developed a strong refusal bias. On the core evaluation, false refusal rose from 2.9% in the constrained baseline to 12.4% with LoRA, 18.2% with DoRA, and 12.94% in the revised adapter.

The adversarial slice was harsher. LoRA and DoRA falsely refused 80.85% of the cases. The revision reached 91.49%. The same-path untrained model was 14.89%.

Those numbers do not say the models became “safer.” They say the adapters learned a cheap way to avoid being wrong: refuse the work.

That is a common evaluation trap. If you reward refusal on dangerous inputs without separately penalizing refusal on valid inputs, the model can improve the wrong behavior. A system that says no to almost everything looks cautious in a demo and useless in production.

The serving-path gotcha

The same untrained 8B model scored 0.4878 on the original serving path and 0.5907 on the evaluation path.

Same base model. No adapter. A 0.103-point move.

That does not prove the newer path is universally better. It proves the runtime was a material experimental variable in this harness. Prompt formatting, system instructions, token handling, stop behavior, and generation defaults can all alter the result before training enters the picture.

The mistake would have been to compare a trained model on one path with an untrained model on another and label the difference “fine-tuning lift.” The evaluation would have produced a clean table and a false explanation.

The fix is boring: freeze the serving path first. Run the untrained base through the exact path you will use for the adapters. Keep the prompts, decoding settings, refusal rules, and scoring code fixed. Only then change the weights.

What the revision actually fixed

The revised adapter was not random iteration. It added intent categories and stricter checks around forbidden behavior. It cleared 12 of 30 intent-classification cases, produced two forbidden outputs, and correctly refused two of three cases where refusal was required.

That was movement. It was not acceptance.

This is where teams often change the scorecard after the result arrives. The adapter improved a subset, so the subset becomes the headline. The failed aggregate and refusal behavior move into a footnote.

I kept the original bar. The adapter lost.

This negative result stopped a larger training run, prevented a weaker system from replacing a working baseline, and exposed a serving-path confound that would have contaminated the next experiment.

What I would run next

The next useful experiment is smaller than another training job.

First, lock one serving path and rerun the untrained base until the score is stable. Second, split refusal into at least three categories: required refusal, false refusal on valid work, and malformed non-answers. Third, evaluate retrieval and generation separately so a better candidate set does not hide a worse answer policy.

Only after those controls hold would I train another adapter. The new adapter would have a written gate:

• beat the constrained baseline on composite score;
• keep false refusal on the core evaluation below the baseline ceiling;
• reduce adversarial failure without collapsing valid answers; and
• produce no forbidden outputs.

That is a deployment contract. “Loss went down” is a training log.

The broader lesson

Fine-tuning is attractive because it feels like progress. The GPU is busy. Checkpoints appear. Curves move. A retrieval baseline feels less ambitious because most of the work is in constraints, evaluation, and plumbing.

The user does not care which path looked ambitious. The user gets the behavior.

The full negative-result report is public because the failed comparison is reusable. It shows the scores, the serving-path confound, and the boundary around what the experiment can establish.

I kept the baseline. The training run still paid for itself.