Negative results, kept on purpose
The things that didn't work.
Low-bit quantization results are unusually easy to overstate, and this project has overstated some of its own before catching them. What follows is the register of rejected methods and standing limitations, each with the lesson it bought. It is published because a method's failures are the part a reader cannot reconstruct from the code.
Claims this project retracted.
Grouped-KL cost surrogate: a −3.52% perplexity win.That was a local screen. It lost the vLLM A/B outright and was archived. The lesson became a rule: promote on the serving metric, never the screen.Staged production render: a −0.0232 last-token KL improvement.The narrow screen improved while direct perplexity regressed from 8.33 to 10.83. Marked "do not ship". This is the canonical screen-versus-gold inversion.A "4× lower KL" framing, and a 17-promotions polish headline.Both withdrawn by the author once the comparisons were found non-rigorous.A GPTQ damping sweep worth +137.5% if disabled.That figure came from a cheap hook screen and inverted on the gold lane. The sweep's evaluator turned out to be in-sample, with held-out basins inverting 31 times out of 31. Fixed damping now ships, at roughly 4.4× less render time.
The pattern repeats often enough to be a design constraint rather than an anecdote: a cheap screen and the served artifact disagree, and the screen is the one that is wrong. Every rule on the method page exists because something got through.
The graveyard.
Archived behind dated walls, several failing fast with a pointer to the archive if a stale configuration tries to invoke them.
| Method | Why it lost — and the durable lesson |
|---|---|
| The three-level cost cascade | The second level beat the additive baseline by −1.5%; a better single cost beat it by −38.5%. One faithful cost beats another level. |
| CLADO's full integer-quadratic solver | O(N²) per-pair measurement for an optimum the O(N) path recovers to within 1–2%. The decision-unit framing was kept; the solver was not. |
| Post-frontier MSE promotion | Re-ranked an already KL-selected point by a local per-bit heuristic. A post-allocator rewrite cannot beat a better cost inside the solve. |
| Smart-union render cache | Offered an 8-bit rung only above a percentile of a surrogate — a render-budget heuristic quietly deciding the allocator's candidate set. |
| Block-output match | Unreachable, not merely unmeasured: the production cache returned first, so it never executed on the shipping recipe — zero hits across two real export logs. Check that code runs before funding an A/B for it. |
| Scale sweep as a default | +77.5% KL on a 4B model. It re-picked block scales after GPTQ, mis-calibrating the error compensation GPTQ had just computed. |
| Analytical closed-form damping | +100–161% KL versus a five-candidate discrete sweep. The fit's per-Linear error compounds. Cheap and discrete won. |
| Column-permutation activation ordering | Failed its own stated objective, and was redundant with GPTQ's full-Hessian propagation. |
| Layer-wise rotations | Would need a residual-transition adapter — a custom kernel — at serve time. That is forbidden in an artifact whose whole promise is vanilla vLLM. |
| Expert pruning | The cost model under-counts token redistribution and misrouting. Size comes from factorization and better formats, not from deleting experts. |
| Surrogate-only knee selection | On 27B the surrogate knee picks 5.857 bpp at KL 0.056; the validated frontier picks 5.31 at 0.015. Outside the additive trust region, bit-rate order is not KL order. |
| Sparse pairwise QUBO | Covering 8 of ~500 Linears is too local to fix a global problem, and expensive enough to dominate the budget while doing it. |
| Top-down polish | Spends its entire budget on cheap high-bit flips and never reaches the bit-rate range anyone ships at. |
Standing limitations.
Some artifacts cannot be quality-checked at all
At 295B the BF16 teacher will not run on the target hardware, so there is no KL to measure against. Those artifacts ship with load and coherent-generation smokes and bit-exact packing — and with no quality claim. The same applies to Laguna-S-2.1 at 117B.
Lower mean KL can hide a heavier tail
The shipped 27B PrismaSCOUT has a worse maximum-prompt NLL than the older 5.5, and a stable adversarial tool-call regression against it, despite winning on mean KL. Mean metrics are not the whole distribution.
Most improvements are small
The honest base rate: most pipeline changes move perplexity by less than 5%, and the cost surrogate is mis-ranked against perplexity at the margin — on one 4B model, 5.5 bpp beats 6.0 bpp. Single-seed measurements are dangerously noisy; a +10% result has flipped to −5.2% across repeats.
Reproducibility is not free
KL is bit-identical within a container session but has drifted 4–8× across sessions from stale model state. Runs bake in commit, calibration hash and cache statistics; a number that cannot be reproduced is quarantined rather than trusted.
Bit-rate labels are not comparable across eras
The accounting convention changed. The public 5.31 artifact's body rate is about 4.76 under the current convention. Comparisons against uniform baselines have to use the same convention on both sides or they are meaningless.
The cost model was blind to activation quantization
AURA's cost is weight-space. NVFP4 (W4A4) and NVFP4A16 (W4A16) render weights bit-identically — maximum difference 0.0 — so they price identically at identical bit-rate, and the allocator is exactly indifferent between quantizing activations and not. Every artifact shipped before 14 August 2026 was allocated under that blind spot, and it was not a small one: on the first cost stage that measures the missing term, the activation side is 84.5% of NVFP4's total predicted Δloss. The worked example and the measurement are on the sensitivity page.
The fix has shipped once, and is still unproven
Qwen3.8-27B PrismaAQUA is the first artifact allocated with the activation term on. Turning it on moved 109 of 496 Linears at a matched 5.5 bpp budget, in the predicted direction. But the arm it is compared against was only ever solved, never rendered or served: there is no served A/B isolating the activation term, and that is what is still owed. The artifact's own measurements are evidence about the artifact, not about the method that chose it.
Research levers are not defaults
A robust Fisher outlier clip remains opt-in with no served A/B behind it, and stays research-only until one exists, however good the screen looked.
The rules that follow from all of it.
- Promote on the serving metric, not the screen. A win counts only when it holds on exact full-vocab vLLM KL-vs-BF16 and direct perplexity on the served artifact at matched bit-rate.
- KL screens; it does not ship alone. Candidates also clear direct perplexity, p99 per-prompt NLL, and tool-use benchmarks before publication.
- Held-out means held-out. Selection KL uses text the cost stage never saw. An audit once found otherwise.
- No hand-tuned bans. If the allocator picks something that breaks at runtime, the measurement is wrong, not the optimizer. Fix the cost model so it sees the real cost. The only constants allowed are ones derived from a dtype's numerical precision.
- Never sell a screen as a result.