Running large language models in production is expensive and slow because of one stubborn bottleneck: tokens are generated one at a time. Speculative decoding is the technique that finally breaks that serial barrier, delivering 2–3x faster LLM inference without changing a single output token. In 2026, it has moved from research curiosity to default setting in vLLM, TensorRT-LLM, and most managed inference platforms.
This guide explains how speculative decoding works, compares the leading variants (vanilla draft model, Medusa, EAGLE-3, and the new P-EAGLE), and shows you how to enable it in vLLM with a minimal configuration. If you ship anything backed by an LLM, this is the cheapest latency win available right now.
What Is Speculative Decoding?
Speculative decoding is an inference optimization in which a small, fast draft model proposes several tokens ahead, and a larger target model verifies all of them in a single forward pass. Accepted tokens commit; rejected tokens are corrected. The math guarantees the output distribution is identical to standard autoregressive sampling — this is not an approximation, it is provably the same model.
The intuition is simple: a 7B model can guess what a 70B model will say next surprisingly often. Each accepted guess is a free token, and each verification step amortizes the GPU memory bandwidth cost across multiple tokens instead of one.
Why It Works in 2026
Modern LLM decoding on H100, H200, and B200 GPUs is memory-bandwidth bound, not compute bound. Loading the model weights for one token is almost the same cost as loading them for eight tokens. Speculative decoding turns that physics in your favor.
- Latency: 2–3x faster time-to-first-token and tokens-per-second on most chat and code workloads.
- Throughput: Up to 2.8x improvement reported by the vLLM team on real production traffic.
- Quality: Bit-exact output distribution — no eval regressions to defend.
- Cost: Same hardware, more tokens per second, lower $/1M tokens.
Speculative Decoding Variants Compared
Not all speculative decoding is created equal. The four approaches you will encounter in 2026 differ in how they generate draft tokens.
1. Draft Model (Classic)
Pair a small model from the same family with the target — for example, Llama 3.2 1B drafting for Llama 3.3 70B. It is the easiest to set up because both models already exist, but you pay extra GPU memory for the draft and acceptance rates depend heavily on family alignment.
2. Medusa
Instead of a separate model, Medusa bolts multiple lightweight prediction heads onto the target model itself. Each head predicts the next token at a different offset. Lower memory overhead than a draft model, but acceptance rates are typically lower than EAGLE.
3. EAGLE-3
EAGLE-3 attaches a small autoregressive head to internal hidden states of the target model. Because it reuses the target’s own representations, acceptance rates are higher (often 0.6–0.8) and memory overhead is minimal. This is the strongest off-the-shelf option for most users in 2026.
4. P-EAGLE (2026)
The newest entry, P-EAGLE, generates all draft tokens in parallel rather than sequentially, removing a serial bottleneck inside the drafter itself. AWS reports up to 1.69x speedup over vanilla EAGLE-3 on B200 hardware. Expect this to become the default in vLLM and TensorRT-LLM through 2026.
How to Enable Speculative Decoding in vLLM
Modern vLLM exposes a single unified flag, --speculative-config, replacing the deprecated --speculative-model. Here is a minimal example using a Llama 3.2 1B drafter for a Llama 3.3 70B target.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
--speculative-config '{
"method": "draft_model",
"model": "meta-llama/Llama-3.2-1B-Instruct",
"num_speculative_tokens": 5,
"draft_tensor_parallel_size": 1
}'For EAGLE-3, swap "method": "eagle" and point model at a published EAGLE head (search the vLLM-Project organization on Hugging Face for matching weights). Tune num_speculative_tokens between 4 and 7 — higher values help on long-form generation but hurt latency on short replies.
Tuning Tips
- Measure acceptance rate first — below 0.5, switch drafters or methods.
- Disable on highly random sampling (high temperature, top-p near 1.0); the speedup collapses.
- Code generation and chat see the biggest gains; pure stochastic creative writing benefits least.
- Always benchmark with your real prompt distribution, not synthetic short prompts.
Real-World Performance Numbers
Published 2026 benchmarks consistently report:
- vLLM with EAGLE-3 on Llama 3.3 70B: 2.4–2.8x throughput improvement on chat workloads.
- NVIDIA TensorRT-LLM with EAGLE-3 on B200: 2–6x speedup on summarization and code tasks.
- Red Hat’s gpt-oss benchmarks: roughly 2x improvement with minimal tuning.
- P-EAGLE in vLLM on B200: an additional 1.69x on top of EAGLE-3.
If you are still running plain autoregressive decoding in production, you are leaving roughly half of your inference capacity on the floor. For background on related optimizations, see our deep dives on vLLM vs TGI vs SGLang, Prompt Caching, and LLM quantization.
When Not to Use Speculative Decoding
It is not a free lunch. Skip it when:
- You are running a tiny model (under 3B); verification overhead dominates.
- You sample at very high temperature for creative tasks — acceptance rate cratters.
- You are batch-bound, not latency-bound, and already saturating GPUs at high throughput.
- You cannot afford the additional 5–15% GPU memory for the drafter or heads.
Frequently Asked Questions
Does speculative decoding change model output?
No. With proper rejection sampling, the output token distribution is mathematically identical to standard decoding. Your evals will not move.
EAGLE-3 vs Medusa — which should I pick?
EAGLE-3 wins for most 2026 workloads because it reuses the target’s hidden states, giving higher acceptance rates and lower memory cost. Medusa is fine if you already have trained heads or need the simplest possible deployment.
How much speedup should I expect?
Plan for 2–3x on chat and code at acceptance rates of 0.6–0.8. Pure greedy or low-temperature decoding can reach 4x; high-temperature creative sampling may see only 1.3–1.5x.
Does speculative decoding work with quantized models?
Yes. AWQ, GPTQ, and FP8 targets all work with speculative decoding in modern vLLM and TensorRT-LLM. Quantize the target; the drafter can stay full precision or also be quantized.
Conclusion: Turn It On
If you are running open-weights LLMs in 2026 and have not yet enabled speculative decoding, do it this week. EAGLE-3 in vLLM is a five-line config change that typically halves your latency and roughly doubles throughput, with bit-exact outputs. P-EAGLE will push that further on B200 hardware as it lands in stable releases. For deeper engineering reading, the vLLM speculative decoding docs and the NVIDIA technical blog are the best starting points.
Try it next: spin up vLLM with the config above, log acceptance rate and tokens/sec for a day of real traffic, then graduate to EAGLE-3 once your baseline numbers are clean. Subscribe to NewsifyAll Technology for more practical AI infrastructure guides.
