Seven Local Models, Zero Clean Passes

I tested seven local models and did not get a winner.

That sounds like a failed benchmark. It was a useful acceptance test.

The fastest candidate passed every coding task and missed half of the required behavior tasks. The strongest extractor still failed one behavior gate. The 70B model was neither the most correct nor the most practical in this matrix.

A leaderboard would have hidden those decisions behind one score.

The gate came before the model

The matrix used three small suites:

• six required behavior tasks;
• five deterministic coding tasks; and
• extraction over 50 synthetic documents, scored as 250 individual fields and as exact documents.

I also recorded single-stream decode speed. Strict acceptance required a model to clear every required gate. A strong extraction result could not cancel a required behavior failure. Fast decoding could not cancel broken code.

The candidates covered several sizes and runtimes:

Every row failed strict acceptance.

The fastest model was not the default

qwen3-coder:30b reached 86.639 decode tokens per second. It also passed all five coding tasks and reached 95.6% field accuracy.

If I had sorted the sheet by speed, it would have won. If I had sorted by coding, it would have tied for first. The behavior suite stopped the promotion: three passes out of six.

That is the point of required gates. They preserve the reason the model exists.

A background coding assistant can tolerate slower extraction. A structured-data worker cannot tolerate malformed fields. A model that may answer users cannot trade required behavior for a better token rate. The weights do not know the deployment contract. The test matrix has to carry it.

Model size did not rescue the decision

llama3.3:70b used the largest dense parameter count in the set. It reached 94.4% field accuracy and 72% exact documents, which were respectable results.

It still missed two behavior tasks and one coding task. It decoded at 3.841 tokens per second.

That does not make the model bad. It makes it a poor strict fit for this job under this runtime and quantization. “70B” describes a model class. It does not describe acceptance.

The same caution applies in the other direction. qwen3:14b produced the strongest extraction result at 96.8% field accuracy and 84% exact documents while passing all five coding tasks. It still missed one required behavior task. A smaller model coming close is useful for the next experiment. It is not permission to erase the miss.

This is not a global model ranking

Six behavior tasks are not a broad safety evaluation. Five coding tasks are not a broad coding benchmark. Fifty synthetic documents are not a production distribution.

The matrix is small on purpose. It is a screening layer that catches obvious deployment mismatches before a larger evaluation. It tells me where to spend the next hundred test cases.

It cannot establish that one model is generally smarter, safer, or better at coding. The runtimes and model families differ. Prompt templates can move outcomes. Single-stream speed is not batch throughput. Every value is tied to this hardware, this serving path, and this task set.

The public matrix reports the aggregate outcomes without publishing prompts, responses, or environment-specific details. The acceptance report states the limits beside the decision.

The next matrix should be harder

The screen did its job. The next pass should expand the failure modes it found.

For behavior, each failed task should become a family of prompt variants rather than one memorized example. For coding, every task should execute against a test instead of relying on surface inspection. For extraction, the documents should vary field order, missing values, irrelevant text, and adversarial formatting.

I would also separate runtime acceptance from model acceptance:

1. Can the runtime load and serve the exact artifact reliably?
2. Can the model clear the task gates?
3. Can it clear them at an acceptable cost and latency?

That order prevents a fast but unreliable runtime from hiding inside a model score. It also prevents a correct but unusably slow model from becoming a theoretical winner.

What changed after the test

I stopped asking which local model was best.

The useful question became: which model clears the required behavior on the cheapest reliable path? None of these rows answered that completely. Two rows identified promising next candidates. Several rows saved me from deploying the wrong default.

Zero clean passes is not an empty result. It is a model-selection system refusing to lie.