The reranker made the answers better and failed the test I built it to pass.

Supported-answer coverage rose from 72.6% to 79.3% with the top-6 variant. Observed false refusals fell from 2.1% to zero. Retrieval precision, the declared target, moved from 22.7% to 24.6%.

The adaptive variant did worse: 19.3% precision.

Calling the experiment a success would require changing the goal after seeing the data.

The problem was noisy retrieval

The baseline already found most of the relevant material. Retrieval recall was 88.9%. The weak point was precision at 22.7%.

That pattern means the right documents are usually present, but they are buried among irrelevant candidates. The generator has to separate support from noise. More recall alone does not solve the problem. It can add more clutter.

The trial used 61 queries over 100 synthetic, non-personal documents. I tested three configurations:

  • baseline retrieval;
  • a top-6 reranker; and
  • an adaptive reranker.

The scorecard covered retrieval precision and recall, citation accuracy, supported-answer coverage, invented sources, refusal on unanswerable queries, false refusal on answerable queries, and mean end-to-end latency.

This was a workflow evaluation, not a benchmark of the reranker in isolation.

The answer metrics improved

The top-6 variant increased recall from 88.9% to 94.8%. Supported-answer coverage rose from 72.6% to 79.3%. False refusal fell from 2.1% to zero in the sample.

The adaptive variant pushed recall to 97.9% and supported-answer coverage to 81.7%. It also recorded zero false refusals.

No configuration invented a source in this bounded set. All three correctly refused the unanswerable cases.

Those are useful results. They suggest that the reranked candidate sets gave the generator more usable evidence, even though the precision metric did not become clean.

They do not erase the primary miss.

Precision barely moved

Top-6 precision improved by 1.9 percentage points, from 22.7% to 24.6%. The adaptive variant fell by 3.4 points from the baseline to 19.3%.

The adaptive path appears to have bought recall by admitting more non-supporting material. The generator still produced better-supported answers, but the retrieval layer became noisier by the declared metric.

This distinction matters because “better RAG” can mean several different things:

  • the right document appears somewhere in the candidate set;
  • fewer wrong documents appear beside it;
  • the answer uses the available evidence;
  • citations point to the right source;
  • the system refuses when support is absent; or
  • the system avoids refusing valid work.

One number cannot stand in for all six.

The latency result is a warning, not a victory

Mean latency fell from 18,992 milliseconds at baseline to 9,485 for top-6 and 9,786 for adaptive.

I do not attribute that improvement to the reranker.

The test measured the end-to-end workflow. Runtime state, request ordering, caching, or other pipeline effects could have moved the number. The trial was designed around retrieval quality, not a component-level latency comparison.

This is the same discipline that prevents a model change from receiving credit for a serving-path change. If the experiment did not isolate the cause, the result stays an observation.

The primary goal stays primary

The honest verdict is mixed:

  • supported answers improved;
  • recall improved;
  • observed false refusals improved;
  • no invented-source or unanswerable-refusal regression appeared;
  • retrieval precision did not improve enough; and
  • the adaptive variant regressed on precision.

That is more useful than a green or red label. It tells me which part of the pipeline changed and which problem remains.

The public aggregate table contains every reported metric. The mixed-result report keeps the decision and limits in one place.

What I would change next

The next run should diagnose the false positives rather than add another reranker.

I would label why each irrelevant document survived: word overlap, entity collision, broad category match, stale chunk context, or scores too close to separate. Then I would set separate gates for candidate precision and answer support.

The acceptance rule should read something like:

  • precision must clear the chosen floor;
  • recall cannot fall below the baseline;
  • supported-answer coverage must improve;
  • false refusal cannot exceed the baseline; and
  • no invented-source regression is allowed.

That prevents one metric from buying a win with another metric’s loss.

The reranker made the system more useful. It did not fix the retrieval problem. Those statements can live together.