I have a valid Q4 result for Gemma 4 E4B and no NVFP4 result.

That is not an incomplete speed comparison. It is a completed runtime-compatibility test.

The Q4 path produced five valid repetitions with a median 58.116 decode tokens per second. The vLLM NVFP4 path never passed its health gate. Putting those paths beside each other in a “Q4 versus NVFP4” chart would create a benchmark that did not happen.

The comparison had a gate

The intended trial pinned the model revision, weight artifact, container digest, runtime version, prompt length, output length, and isolation checks.

Before an NVFP4 performance number could count, the path had to pass:

  1. preflight;
  2. model metadata;
  3. artifact hash;
  4. resident-process isolation;
  5. server launch; and
  6. endpoint health.

The final attempt passed the first five and failed the sixth.

That ordering prevented a container log, a loaded process, or a partially initialized endpoint from becoming a benchmark result. “The server started” was not enough. The endpoint had to answer the pinned request.

The Q4 leg was healthy

Across five valid repetitions, the local Q4 path recorded:

  • 58.116 median decode tokens per second;
  • 1,229.393 median prefill tokens per second;
  • 387.183 milliseconds median time-to-first-token; and
  • 817.023 milliseconds median total time.

The request used 476 prompt tokens and 25 output tokens.

Those values describe the Q4 path under the pinned request. They do not establish model quality, long-context behavior, batch throughput, or power efficiency.

Most important: they have no valid NVFP4 counterpart.

Eight attempts did not produce a second bar

The trial used eight troubleshooting attempts. Earlier runs failed at different points, including artifact acquisition, resident-process isolation, and endpoint health.

The final path narrowed the problem. The model artifact was present. The runtime launched. The endpoint did not become healthy within the 15-minute window.

The runtime reported that it could not select a scaled matrix-multiplication kernel for the linear layer. It suggested a forced FP8 Marlin test path.

That is a compatibility boundary between the pinned model artifact, the tested vLLM build, and the DGX Spark execution path. It is not evidence that NVFP4 is slow. It is evidence that this runtime combination did not reach a measurable state.

The invalid comparison was tempting

The Q4 number exists. The NVFP4 model is designed around lower-precision execution. A title promising a quantization showdown writes itself.

The missing bar creates pressure to use something else:

  • launch time instead of inference time;
  • a log line instead of a response;
  • a result from a different runtime;
  • a result from a different model revision; or
  • a partial earlier attempt with weaker isolation.

Each substitution makes the chart more complete and the experiment less true.

The correct value for the NVFP4 cell is not zero. It is “no valid measurement.”

Zero would imply the endpoint ran and produced no tokens. It did not reach the point where throughput existed.

Compatibility is part of performance

A kernel path that cannot initialize has no operational throughput. That does not make a launch failure a speed score, but it does make compatibility a first-class acceptance gate.

This matters on new hardware and fast-moving runtimes. Model repositories, quantization formats, kernels, container builds, and compute capabilities can all move on different schedules. A model being public does not mean every advertised path is executable on every supported-looking stack.

The benchmark method has to preserve those layers:

  • artifact identity;
  • runtime identity;
  • hardware identity;
  • health evidence; and
  • performance evidence only after the first four hold.

Without that chain, a result is difficult to reproduce and easy to misattribute.

The next valid experiment

I would rerun only after a pinned runtime demonstrates a healthy NVFP4 endpoint for the same artifact on the same hardware.

Then I would repeat the Q4 leg in the same window, keep the prompt and output shape fixed, run multiple repetitions, and report latency beside throughput. If the runtime requires a forced kernel path, that setting belongs in the result name and method.

The old Q4 number can remain as historical evidence. It should not be silently combined with a later NVFP4 number from a materially different environment.

The machine-readable failure summary records the pinned trial and invalid measurement state. The public report explains the boundary without exposing local artifact paths.

The comparison never started. The experiment still finished.