Quantization Guide¶

Recommendation: Use NVFP4 as your default — best quality-to-speed tradeoff, MLX-native, fastest cold start. Use Q4_K_M if you are on Ollama (it is the default GGUF quant). Use Q8_0 only when quality is critical and you have memory to spare.

Quantization reduces model precision to fit in memory and improve throughput. This page compares all major quantization formats available on Apple Silicon.

Quantization formats overview¶

Perplexity comparison¶

Perplexity (lower is better) measured on a held-out validation set, with BF16 as the baseline:

Speed comparison¶

Small Models (< 12B params)¶

For small models, MXFP8 is actually faster than NVFP4 — the extra precision does not cost much on small parameter counts.

Large Models (≥ 12B params)¶

For large models, NVFP4 is the clear winner — MXFP8's higher precision comes with a significant speed penalty.

File size comparison¶

When to use each format¶

NVFP4 — Recommended Default¶

Use NVFP4 for: Most users, most models, most of the time.

• MLX-native 4-bit block floating point
• Best perplexity-to-speed tradeoff (17.95)
• Fastest cold start for large models
• Excellent tool calling support
• Supported by all MLX engines (mlx_lm, Rapid-MLX, oMLX)

# Convert to NVFP4
mlx_lm.convert --model <model> --quantize nvfp4

MXFP8 — When Quality Matters Most¶

Pick MXFP8 for: Small models where you want maximum quality and have memory to spare.

• 8-bit microscaling float — closest to BF16 quality
• Actually faster than NVFP4 on small models (< 12B params)
• 2x the file size of NVFP4
• Slower than NVFP4 on large models

mlx_lm.convert --model <model> --quantize mxfp8

MXFP4 — The Middle Ground¶

MXFP4 shines when: You want 4-bit efficiency but need slightly different quantization behavior.

• Similar quality to NVFP4
• Larger model files than NVFP4
• Slower cold start than NVFP4
• Not widely tested — use NVFP4 instead unless you have a specific reason

mlx_lm.convert --model <model> --quantize mxfp4

4bit-MLX — mlx_lm Native¶

Choose 4bit-MLX when: Using mlx_lm's built-in quantization.

• mlx_lm's native 4-bit format
• Similar quality to NVFP4
• Slightly different quantization behavior
• Use when the model is already in 4bit-MLX format from Hugging Face

# Load a pre-quantized 4bit-MLX model
mlx_lm.server --model Qwen3.5-35B-A3B-abliterated-4bit

Q4_K_M — llama.cpp K-Quant¶

Q4_K_M works best with: Ollama (which uses llama.cpp under the hood).

• llama.cpp's 4-bit K-quant
• Smallest file sizes
• Slightly lower quality (perplexity 18.36 vs NVFP4's 17.95)
• Good for memory-constrained scenarios
• Supported by Ollama out of the box

# Pull a Q4_K_M quantized model in Ollama
ollama pull gemma4:e4b-q4_K_M

Q8_0 — llama.cpp 8-Bit¶

Reach for Q8_0 when: You want maximum quality in Ollama and have memory to spare.

• llama.cpp's 8-bit block quantization
• Near-lossless quality (~17.6 perplexity)
• 2x memory of Q4_K_M
• Good for critical tasks where quality matters most

ollama pull gemma4:e4b-q8_0

Quantization decision flowchart¶

Summary¶