Llama 3.1 8B Instruct
The Llama 3.1 model card lists an 8B model. This page uses that size class, not a separate counted total.
The model card says 128k context. The public config mirror has max_position_embeddings 131072.
From the model card
Strengths
- The Llama 3.1 card describes an instruct model for dialogue, with tool use and a 128k context.
- The card lists eight languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
Weaknesses
- The license requires attribution and has a monthly-active-user clause. It is not Apache-2.0.
- The card says the knowledge cutoff is December 2023. The model is text only.
- No Frankenstein row yet. This page does not copy Meta's benchmark table.
Sources: Llama 3.1 model card · config.json (public mirror; official weights are gated)
Memory at 8,192 tokens
Computed, not measured. Weights use llama.cpp’s whole-file averages (Q4_K_M = 4.8944 bits/weight, Q8_0 = 8.5008) on the total parameter count. KV is fp16 and does not depend on the quant. The formula, including the sliding-window and latent-KV cases, is on the methodology page.
| Q4_K_M | Q8_0 | |
|---|---|---|
| Weights | 4.56 GiB | 7.92 GiB |
| KV at 8,192 | 1 GiB | 1 GiB |
| Total | 5.56 GiB Computed | 8.92 GiB Computed |
32 layers, 8 KV heads, head dim 128. Every layer stores the full context.
Tokens per second
Not yet measured. This lab has no Frankenstein row for Llama 3.1 8B Instruct, so there is no estimated tok/s either.
Hardware tiers
Minimum is the smallest RAM that holds Q4_K_M plus 8k KV. Suggested does not name a second machine or a speed this lab has not run. Performance means a CUDA-class card with 12 GB or more of GDDR6 or GDDR6X and the weights fully on the card. The Tesla P40 is 24 GB of GDDR5, so a P40 speed is not the performance-tier number.
| Tier | Example hardware | Quant | Fit | tok/s |
|---|---|---|---|---|
| Minimum | CPU-only, with at least 7.56 GiB of system RAM for the Q4_K_M weights, 8k KV, and a 2 GiB planning allowance. | Q4_K_M | CPU-only, if the RAM is there | CPU: not yet measured |
| Suggested | A machine you are willing to wait on, or a GPU that can hold the quant. This lab has not measured a separate “comfortable” machine, so this row does not invent one. | Q4_K_M until a Q8 run exists | Whichever of the machines above actually holds the weights | not yet measured |
| Performance | A CUDA-class card with 12 GB or more of GDDR6 or GDDR6X, weights fully on the card. | Q4_K_M | Full VRAM on a 12 GB or larger GDDR6-class card | not yet measured |
Where it sits
Best-for is a judgment from the card’s stated jobs and the model’s size. It is not a speed ranking. The matching build is Family server. Score it against a machine you have on What can my rig run?