Methodology

1. Will it fit? (memory)

needed GB = weights + KV cache + 1 GB overhead
weights GB = parameters (billions) × bits-per-weight ÷ 8
KV cache GB = (full-attention layers × context + sliding-window layers × min(context, window)) × bytes per token per layer ÷ 10⁹
bytes per token per layer = 2 (K+V) × KV heads × head dim × 2 bytes (fp16)  or  (kv_lora_rank + RoPE dim) × 2 bytes for MLA models
usable GB  = VRAM − 0.5 GB   (discrete GPUs)
           = 75% of unified memory   (Macs, Ryzen AI Max, DGX Spark)

Bits per weight (approximate, llama.cpp GGUF incl. scales): Q4_K_M 4.85 · Q5_K_M 5.69 · Q6_K 6.56 · Q8_0 8.5 · F16 16 · MXFP4 4.25. "Fits" means needed ≤ 90% of usable. "Tight" means 90–100%. Parameter counts, layer counts and KV-head counts come from each model's Hugging Face metadata/config (linked on every model page). Active-parameter counts for MoE models come from the model card. The 75% unified-memory figure follows the default GPU memory limit described by llama.cpp's maintainer in discussion #4167. It can be raised manually. Most 2026 models are hybrids. Qwen3.5/3.6/3.8 use linear attention in 3 of every 4 layers. Gemma 4 and gpt-oss mix sliding-window and global layers. Nemotron 3.5 and LFM2.5 mix Mamba/conv layers with a few attention layers. GLM and DeepSeek use compressed (MLA-style) caches. We read the layer layout from each config.json and only grow the cache for layers that actually keep per-token K/V. Fixed-size recurrent state is covered by the overhead. Where config.json isn’t public (gated or GGUF-only repos), the KV cache is shown as "not modelled" and real needs are higher. For MoE models, memory uses total parameters (all experts must be loaded) and speed uses active parameters (from the model card). If the card doesn’t publish active parameters, we show no speed. Partial CPU/RAM offload of MoE experts is popular but not modelled yet.

2. How fast? (tokens/sec)

Text generation is limited by memory bandwidth: each new token reads the active weights once.

estimated tok/s = efficiency × bandwidth (GB/s) ÷ active weights (GB)

Measured figures are the Llama 2 7B Q4_0 tg128 results from the llama.cpp community scoreboards: ggml-org/llama.cpp/discussions/15013, ggml-org/llama.cpp/discussions/10879, ggml-org/llama.cpp/discussions/15021, ggml-org/llama.cpp/discussions/4167. For a device with a measured run, its efficiency = measured ÷ (bandwidth ÷ 3.79 GB). For a device without one, we use the median across measured devices: 74.1% (from 27 devices). Anything not directly measured is labelled estimate. Prompt processing, long-context slowdown, batching and MoE routing overhead are not modelled.

2b. Graphics card + system RAM (offload): estimates

MoE:   on GPU  = (total − routed-expert weights) + KV cache + 1 GB, plus as many experts as still fit in 90% of usable VRAM
       in RAM  = remaining expert weights;  fits if ≤ 90% of (RAM − 8 GB reserved for the OS/apps)
       expert weights = MoE layers × experts × (3 × hidden × expert FFN size) × bits/8   (from config.json; 2 matrices for Nemotron-H)
Dense: every layer can move to RAM; 5% of weights (embeddings/output) stay on the GPU
Speed: 1 / tok/s = active weights on GPU ÷ (GPU efficiency × GPU bandwidth) + active weights in RAM ÷ (60% × 89.6 GB/s)

Assumptions (labelled so you can judge them): system RAM is dual-channel DDR5-5600 (89.6 GB/s theoretical), and we assume 60% of that is achieved. We reserve 8 GB of RAM for the OS. Active experts are treated as spread evenly between GPU and RAM. PCIe transfers, CPU compute limits and prompt processing are not modelled, so these speeds are optimistic upper bounds. One public report (Qwen3.8-35B-A3B distill, RTX 3060 12GB) shows ~50 tok/s, where our formula gives ~80 tok/s for the same-architecture Qwen3.6-35B-A3B. When reader results exist, trust them over our estimate. Unified-memory machines (Macs, Ryzen AI Max, DGX Spark, RTX Spark) don't need this: their RAM is already the GPU memory.

2c. Community benchmarks

Readers submit results with a proof link (form). Each one is checked automatically. It's flagged if it's above the card's memory-bandwidth limit (bandwidth ÷ active weights, ×1.25 headroom for speculative decoding), more than 5× below our estimate, for unknown hardware/models/quants, or as a duplicate. Results with no flags are published as community measured, and flagged ones wait for a human.

3. Prices

A daily script fetches retailer category pages or public product feeds. It checks robots.txt first, honours Crawl-delay, and skips any site that shows a bot challenge. Each price is recorded with the page URL and the time it was fetched. When a retailer has several listings, we show the cheapest in-stock one, or the cheapest listed if none are in stock (flagged). If nothing matches, the item shows price unavailable. Prices are never guessed or converted from USD. Apple prices are Apple AU's "From" price for the stated base configuration.

Last run: 20 Oct 2026, 7:38 am AEST