Mistral

Mistral Small 4

Mistral's docs say 119B total and 6.5B active. The launch post says about 6B active per token, and about 8B if embeddings are counted.

Apache-2.0 · 119B total, 6.5B active · context 262,144

The product docs say 256k context. config.json max_position_embeddings is 1,048,576. This page uses the 256k figure from the docs.

From the model card

Strengths

  • The docs describe a multimodal model (text and image) with tool use, a 256k context, and Apache-2.0 weights.
  • Mixture-of-experts: 128 routed experts in config.json, 4 experts per token.

Weaknesses

  • Weight memory follows the 119B total, not the active count.
  • The KV figure uses the latent sizes in config.json (kv_lora_rank and the rope dim). It is not a measured allocation. If a runtime stores more than that latent, real KV is higher. The weight term dominates.
  • No Frankenstein row yet.

Sources: Mistral Small 4 docs · Launch post · config.json

Memory at 8,192 tokens

Computed, not measured. Weights use llama.cpp’s whole-file averages (Q4_K_M = 4.8944 bits/weight, Q8_0 = 8.5008) on the total parameter count. KV is fp16 and does not depend on the quant. The formula, including the sliding-window and latent-KV cases, is on the methodology page.

Computed memory for Mistral Small 4 at 8192 context
Q4_K_MQ8_0
Weights67.8 GiB117.77 GiB
KV at 8,1920.18 GiB0.18 GiB
Total67.98 GiB Computed117.94 GiB Computed

36 layers. Latent KV uses kv_lora_rank 256 plus a rope dim of 64, not a full KV head times head dim.

Tokens per second

Not yet measured. This lab has no Frankenstein row for Mistral Small 4, so there is no estimated tok/s either.

Hardware tiers

Minimum is the smallest RAM that holds Q4_K_M plus 8k KV. Suggested does not name a second machine or a speed this lab has not run. Performance means a CUDA-class card with 12 GB or more of GDDR6 or GDDR6X and the weights fully on the card. The Tesla P40 is 24 GB of GDDR5, so a P40 speed is not the performance-tier number.

TierExample hardwareQuantFittok/s
MinimumCPU-only, with at least 69.98 GiB of system RAM for the Q4_K_M weights, 8k KV, and a 2 GiB planning allowance.Q4_K_MCPU-only, if the RAM is thereCPU: not yet measured
SuggestedA machine you are willing to wait on, or a GPU that can hold the quant. This lab has not measured a separate “comfortable” machine, so this row does not invent one.Q4_K_M until a Q8 run existsWhichever of the machines above actually holds the weightsnot yet measured
PerformanceA CUDA-class card with enough GDDR6 or GDDR6X to hold about 68.98 GiB (Q4_K_M weights, 8k KV, and a 1 GiB planning allowance). 12 GB is not enough for that total.Q4_K_MFull VRAM only if the card is larger than 12 GBnot yet measured

Where it sits

Best-for is a judgment from the card’s stated jobs and the model’s size. It is not a speed ranking. The matching build is CPU-only used Xeon. Score it against a machine you have on What can my rig run?