Memory Management Guide¶

Apple silicon unified memory is shared by CPU and GPU — great for bandwidth, but everything competes for the same pool. Memory is the bottleneck. Here is how to manage it.

Apple silicon unified memory architecture¶

Unlike traditional computers with separate CPU RAM and GPU VRAM, Apple silicon uses a unified memory pool shared by the CPU, GPU, Neural Engine, and other coprocessors.

Key Properties¶

• No copying: CPU and GPU access the same memory — no PCIe transfer overhead
• Zero-copy inference: Model weights are read directly from shared memory
• Bandwidth: M1: 68 GB/s, M1 Pro: 200 GB/s, M1 Max: 400 GB/s, M2 Ultra: 800 GB/s
• Capacity: 8 GB (base M1/M2) to 192 GB (M2 Ultra)

The Trade-off¶

The unified architecture means everything competes for the same memory:

• macOS system processes
• Your browser with 50 tabs
• VS Code with extensions
• The LLM model weights
• The KV cache during generation
• Other GPU workloads

GPU memory limit¶

macOS enforces a soft limit on GPU-addressable memory via the iogpu.wired_limit_mb sysctl.

Default Limits¶

# Check current GPU memory limit
sysctl iogpu.wired_limit_mb

Raising the GPU Memory Limit¶

For 64 GB+ systems, you can raise the limit:

# Check current limit
sysctl iogpu.wired_limit_mb

# Raise to 57344 MB (56 GB) — requires sudo
sudo sysctl iogpu.wired_limit_mb=57344

# Make permanent (add to /etc/sysctl.conf)
echo "iogpu.wired_limit_mb=57344" | sudo tee -a /etc/sysctl.conf

Warning: Raising the limit too high causes system instability. macOS needs memory for the UI, kernel, and apps. If the GPU consumes too much, the system hangs or force-kills apps.

Safe Limits by RAM¶

KV cache calculation¶

The KV cache stores key-value pairs from previous tokens during generation. It grows linearly with context length.

Formula¶

KV Cache (bytes) = context_length × num_layers × kv_heads × head_dim × bytes_per_element × 2

Where: - context_length: Number of tokens in the context window - num_layers: Number of transformer layers - kv_heads: Number of key/value attention heads - head_dim: Dimension of each attention head (typically 128) - bytes_per_element: 2 for FP16, 1 for Q8_0, 0.5 for Q4_0 - × 2: Separate K and V caches

Simplified Formula¶

KV Cache (GB) = context_length × num_layers × kv_heads × 128 × bytes_per_element × 2 / 1,073,741,824

Calculator Script¶

#!/usr/bin/env python3
"""Calculate KV cache memory for a given model configuration."""

def kv_cache_gb(context_length, num_layers, kv_heads, head_dim=128, bits=8):
    """Calculate KV cache size in GB."""
    bytes_per_element = bits / 8
    bytes_total = context_length * num_layers * kv_heads * head_dim * bytes_per_element * 2
    return bytes_total / (1024 ** 3)

# Common model configurations
models = [
    # (name, layers, kv_heads, head_dim)
    ("Gemma 4 9B",       42,  8, 128),
    ("Gemma 4 26B",      62, 16, 128),
    ("Qwen 3 Coder 14B", 40,  8, 128),
    ("Qwen 3 Coder 30B", 62,  8, 128),
    ("Llama 3.3 70B",    80,  8, 128),
    ("Qwen 3 72B",       80,  8, 128),
]

print(f"{'Model':<20} {'4K Q8':>8} {'8K Q8':>8} {'16K Q8':>9} {'32K Q8':>9} {'32K F16':>9}")
print("-" * 65)

for name, layers, kv_heads, head_dim in models:
    c4k  = kv_cache_gb(4096, layers, kv_heads, head_dim, 8)
    c8k  = kv_cache_gb(8192, layers, kv_heads, head_dim, 8)
    c16k = kv_cache_gb(16384, layers, kv_heads, head_dim, 8)
    c32k = kv_cache_gb(32768, layers, kv_heads, head_dim, 8)
    c32k_f16 = kv_cache_gb(32768, layers, kv_heads, head_dim, 16)
    print(f"{name:<20} {c4k:>7.2f}G {c8k:>7.2f}G {c16k:>8.2f}G {c32k:>8.2f}G {c32k_f16:>8.2f}G")

Context Window Impact¶

KV cache reality: At 128K context, the KV cache alone can exceed the model weights in memory usage for dense-attention models. Gemma 4 26B uses a sliding window (only 5 full-attention layers), so its KV cache stays small even at 128K. This is why flash attention and KV cache quantization are critical for long-context work with dense models.

KEEP_ALIVE=0s for burst-and-unload¶

The OLLAMA_KEEP_ALIVE=0s setting is the top memory optimization.

How It Works¶

export OLLAMA_KEEP_ALIVE=0s

• Without KEEP_ALIVE=0s: Model stays in memory for 5 minutes after last use
• With KEEP_ALIVE=0s: Model is unloaded immediately after the response is complete

Memory Impact¶

When to Use¶

• Burst-and-unload workflows: Send a query, get a response, free memory
• Multi-model setups: Switch between models without manual unloading
• Memory-constrained systems: 8-16 GB RAM where every GB counts

When NOT to Use¶

• Real-time chat: Adds 1-3 seconds of load time per query
• Batch processing: Loading/unloading overhead adds up
• Interactive use: The delay between queries can be annoying

Alternative: Longer Keep-Alive¶

# Keep model for 30 seconds after last use
export OLLAMA_KEEP_ALIVE=30s

# Keep model for 2 minutes
export OLLAMA_KEEP_ALIVE=2m

OLLAMA_MAX_LOADED_MODELS tuning¶

Controls how many models can be loaded simultaneously.

export OLLAMA_MAX_LOADED_MODELS=2

Guidelines¶

How It Works¶

When a new model is requested and the limit is reached, Ollama unloads the least recently used model. This is automatic — no manual management needed.

Memory pressure monitoring¶

Using memory_pressure (built-in)¶

# Quick check
memory_pressure

# Continuous monitoring
while true; do
    clear
    memory_pressure | head -20
    sleep 2
done

Using macmon (recommended)¶

macmon provides detailed Apple silicon thermal and memory monitoring:

# Install
brew install macmon

# Run (no sudo needed)
macmon

# One-shot
macmon --once

Output shows: - CPU/GPU/Memory/ANE power usage - Thermal pressure level - Memory pressure - Fan speed

Using powermetrics (detailed)¶

# GPU power and memory
sudo powermetrics --samplers gpu_power -i 1000 -n 10

# System power
sudo powermetrics --samplers tasks -i 5000 -n 5

# Memory stats
sudo powermetrics --samplers mem_pressure -i 2000 -n 5

Interpreting Memory Pressure¶

Swap Impact on Performance¶

Stale swap from previous model runs costs 28–31% tok/s on small models — even when you have plenty of free RAM. macOS does not reclaim swap after models unload, so yesterday's model run can silently degrade today's inference.

# Quick swap check
sysctl vm.swapusage

If swap is above 1 GB and you have not loaded a large model, you have stale swap. Use macmon --once for a fuller picture.

Full analysis and fixes →

Next steps¶

• Model Selection — Choose the right model for your RAM
• Ollama Setup — Configure Ollama with optimal memory settings
• MLX Setup — MLX memory management specifics