Methodology

What counts as a number

The site is allowed to be incomplete. It is not allowed to dress up a guess. These are the rules a figure has to pass before it leaves a placeholder cell.

Measured means a run

A speed is measured when it came from a run on hardware in the lab. The row names the model, the quant, the context length, and the software version, and it has a date. One shot is still a measurement if the note says it was one shot. It is not a distribution, and the note should not pretend it is.

An expected figure, when one is published, is labeled expected and says what it is based on. Version 1 has none. A placeholder is an empty slot with the word PLACEHOLDER on it. A blank is not a zero, and it is not "slow."

Two speeds, not one

Generation tokens per second is how fast output tokens arrived after the prompt was processed. Prompt tokens per second is how fast that prompt was processed. A coding session with a long file can be limited by the second number while the first looks fine. Both go in the benchmark table.

Context length is part of the result. A number at 4k does not answer the question for 32k. The row records the length that was requested for that run.

Prices are snapshots

A price on a build has three parts: the amount, the day it was pulled, and the source. The page says "price pulled on" that date. Sold prices and asking prices are different facts and do not share a cell. The source line says which one it was.

Prices are not live, and they are not updated in silence. A new pull is a new date. If the date is missing, the cell stays a placeholder even if someone remembers a number. Affiliate links, if they are added later, do not choose the part. The disclosure says how those links will be marked.

Power is measured at the wall

Idle watts and load watts come from a meter on the machine that produced the speed row, when the lab can do both on the same day. A manufacturer TDP, a reviewer's chart, and a guess from the PSU sticker do not go in the power cells. Those sources can be discussed in the gotchas. They are not the measurement.

Memory figures on the model pages

Weight memory is computed, not measured:

weights (GiB) = parameter count × bits per weight / 8 / 1024³

Bits per weight are the whole-file averages llama.cpp publishes for Llama 3.1 8B: Q4_K_M = 4.8944, Q8_0 = 8.5008. Source: the llama.cpp quantize README. The same averages are applied to other architectures, including mixture-of-experts models. That is an approximation. MoE memory uses the total parameter count, because every expert is stored. Active parameters are what the model computes with, not what it occupies.

KV cache at context C, in fp16:

  • Grouped-query attention: 2 × layers × KV heads × head dim × C × 2 bytes.
  • Gemma 3: the published config uses a sliding window of 1024, and a full-attention layer when (layer index + 1) is divisible by 6. Sliding layers use min(C, 1024). The vision encoder is not in this number.
  • gpt-oss: the published config alternates sliding attention (window 128) and full attention, and it starts with sliding.
  • Latent KV, used here for Mistral Small 4: layers × C × (kv_lora_rank + rope dim) × 2 bytes. That follows the latent sizes in its config. It is not a measured allocation. If a runtime stores more than the latent, real KV is higher.

The 8,192-token column is that formula at C = 8192. It does not include a runtime allocator or the operating system. The recommender adds a separate planning allowance — 1 GiB on the GPU, 2 GiB on the system — and labels it as planning, not a measurement.

A vendor memory claim, such as gpt-oss MXFP4 “within 16 GB”, is quoted from the model card. It is not this formula and not a lab measurement.

What the recommender scores

/recommend runs in the browser. It does not send the form anywhere.

  1. It computes weight plus KV at the context you asked for, using the formula above.
  2. Fit, in order: full VRAM if VRAM covers the total plus 1 GiB; otherwise partial offload if the GPU has at least 4 GiB and VRAM plus system RAM cover the total plus 3 GiB; otherwise CPU-only if system RAM covers the total plus 2 GiB; otherwise it does not fit. Apple Silicon unified memory is one pool: the RAM field is that pool, and discrete VRAM is ignored.
  3. Quant: speed always uses Q4_K_M. Quality uses Q8_0 when that quant still fits. Balanced uses Q8_0 only when it fits in the same class as Q4_K_M (full VRAM, unified, partial, or CPU).
  4. For each goal you selected, models whose card lists that job score higher. A model that is not a vision model cannot win the vision goal. The privacy goal matches every model in the catalog, because the constraint is that the prompt stays on your machine, which is how this page works.
  5. Speed preference ranks smaller active-parameter counts higher. Quality preference ranks larger total-parameter counts higher. That is a size ordering, not a tokens-per-second estimate.
  6. A model whose published context is shorter than you asked is ranked down. If the card documents a YaRN extension that covers the request, the penalty is smaller and the page says the extension is required.
  7. A measured tok/s is shown only for Q4_K_M rows from the Frankenstein run, and the sentence names that machine. A CPU row is not copied onto a GPU. The P40 row is not copied onto a GDDR6 card. Q8 speeds are not yet measured. The performance tier on the model pages is a CUDA-class card with 12 GB or more of GDDR6 or GDDR6X. Its tok/s is not yet measured.
  8. Cores and OS are stored in the share link and are not scored. This lab does not have a per-core scaling rule. On Frankenstein, 48 threads was slower than the default, and that result stays on the benchmark table rather than becoming a formula.
  9. The upgrade section lists the next VRAM or RAM step at which a model that does not fit today would fit at Q4_K_M. It does not include a price.

The top scoring model that fits is the pick. The next three that fit are the alternates. A model the publisher has retired is ranked down, not hidden.

What the site will refuse

  • A tokens-per-second estimate published as if the lab had run it.
  • A ranking of hardware that is not here.
  • A price with no date and no source.
  • Deleting a bad run to tidy the table. The bad run gets a note.

How a row gets onto the site

Builds are Markdown files in src/content/builds. Benchmark rows are objects in src/data/benchmarks.json. Model cards are objects in src/data/models.json. Notes are Markdown files in src/content/posts. The build fails if a placeholder build smuggles in a price or a speed, if the benchmark file is marked measured while a sample row is still in it, or if a model lists a benchmark name that is not in the table. The README has the templates.

The first measured rows are Frankenstein, 18 Nov 2026: two Xeon E5-2697 v2 CPUs, 288 GB DDR3, a Tesla P40, Ollama 0.35.0. Generation tok/s is the median of three runs. Prompt tok/s is the first uncached prefill; a cached repeat is not entered. The write-up is the Xeon notebook draft.