The problem

Which model should run on my own machines, with which engine and which settings? Leaderboards and vendors’ speed figures don’t answer that: they were measured on someone else’s hardware, prompts and context lengths.

Measuring it yourself is easy to do badly. A request can “succeed” and return an empty answer. A tuner’s favourite setting can be noise. Two configurations can differ in a hidden reasoning budget, so the comparison is meaningless. I wanted a lab where a number is quoted only if the evidence behind it can be re-read.

The hardware is two machines at home: a Strix Halo laptop (AMD Ryzen AI Max+ 395) with 64 GB of memory shared between CPU and GPU, and a node with an RTX 3090 (24 GB) and 128 GB of system memory.

What I built

  • modelbench, a command-line harness that evaluates a whole model setup: the exact weights, the quantisation, the chat template, the engine and its build, the machine, and the serving and sampling settings. It runs benchmarks through their own pinned harnesses, and it reports quality and effort side by side, never merged into one score.
  • Device descriptors. Each machine publishes the setups it can serve and the exact commands to start, check and stop them. The harness verifies that the running server is the setup it expects, stops only what it started and never touches a server it did not start. A machine stays marked unqualified until every required check has evidence.
  • An engine qualification kit. Each new engine build goes through 17 captured checks before it counts as qualified; a missing capture or an unknown observation is not a pass. On 1 November 2026, two builds passed all 17 on the Strix Halo machine.
  • Campaign runners and report generators. Every request’s body, usage and timings are stored; the report is generated from the stored numbers, with each number pointing at its source file. Nothing in a report is typed by hand.

The engines themselves are other people’s open-source work, including llama.cpp and Strata. My part is everything around them: how a setup is pinned, started, checked, measured and reported.

How it works

  1. input Pin the setup Weights, engine release and built binary identified by checksum
  2. Start under ownership One workload at a time; refuses if another workload owns the GPU
  3. check Verify what is serving Ask the running server its build and limits; refuse a mismatch

    Loop from step 3: wrong setup, refuse and fix: back to step 1

  4. Measure Smoke test, fixed grid, real answers, concurrency, 20-minute soak
  5. check Grade outside the model Hidden tests run in a fresh grader after the agent has exited
  6. output Report and archive Report built from stored numbers; every file hashed and re-read
One benchmark run, from pinned files to archived evidence

Status words are mechanical, not optimistic: a setup that could not start is blocked, a phase that did not run is not run, a value nobody measured is not measured. Each of those stays in the summary instead of disappearing from it.

Results

An overnight campaign on the RTX 3090 node (October 2026). Six serving setups were entered against the daily default; five were measured, and one was blocked before it could serve and reported as blocked, not as a weak model. The campaign sent 536 requests and 533 came back valid: all 90 normalised requests, 187 of 189 in the concurrency ladder and 256 of 257 under mixed load. The fastest setup decoded about 85 tokens per second with a 1,000-token prompt and 79 with a 32,000-token one. The verdict was to change nothing: no candidate was a better default across the board. The evidence archive holds 2,367 files, and every checksum was re-read and matched.

A 125-billion-parameter model on one 24 GB card (November 2026). The model is a mixture of experts with about 6 billion parameters active per token, at 3-bit quantisation; its experts stream from system memory while the card does the rest. Shutting the graphical desktop down and lowering the engine’s graphics-memory reserve from 3 GB to 1 GB, which leaves about 2 GB more for the expert cache, raised writing speed to 101–103 tokens per second on real answers, from 93–97 with the desktop running. That change counted only because all four rounds landed outside the unchanged setup’s range; sixteen other single-setting changes did not. Long prompts held up: the first token of an 87,000-token prompt arrived after 33 seconds and of a 178,000-token one after 71, with 5 of 5 test facts recalled.

For agent work. Replaying two recorded coding-agent sessions, serving them one request at a time finished both in 87 seconds; running them side by side took 100. The price is a longer wait for the first word, so on this card one request at a time is the faster way to serve two agents.

On the laptop. A small mixture-of-experts model (8 billion parameters, 1 billion active, 4-bit) answered real prompts at about 151 tokens per second.

What I’d do differently

  • Pin the reasoning budget from the first run. One model’s template silently defaulted to its highest reasoning effort, and a 4,096-token cap could use up the budget before any answer appeared. That pair could not be compared fairly.
  • Count stored answers, not exit codes, from day one. A request that returns is not a request that answered; the counts are now reconciled against the per-request files.
  • Plan the repeat rounds up front. A single round is a screen, not a verdict; the schedule should reserve the replications before the first result tempts a conclusion.
  • Watch the host, not only the model. The worst incident was a desktop resource service looping 934 times a second, which made stopping a server take up to 605 seconds. With the desktop off during runs, the longest stop was 4 seconds.
  • Get a real quality score. Two repair tasks per setup is a smoke test, and no scored benchmark I run reaches beyond about 64,000 tokens. That is the next gap to close.