For small batch sizes, LLM decoding is generally memory-bandwidth-bound and not compute-bound. One token produced means the entire weight set of the model is streamed from HBM into the compute units, billions of parameters moved for one token, then moved again, in full, for the next. And that wait is what actually dominates the cost of interactive LLM serving, so the GPU spends the majority of its cycles waiting on memory. Speculative decoding addresses this bottleneck as an inference optimization: a smaller model guesses a few tokens ahead, and the target model verifies them all in a single pass, without altering the target model's output distribution. A couple of generations of drafters have come and gone in the field since the original draft-then-verify idea in 2023.
I'll explain how each generation works and why it exists. To illustrate, I ran the released checkpoints for all three methods on the same target model (Qwen3-8B) and the same hardware (a single H100 on Modal).
Same prompt, same model, three drafters plus a vanilla baseline, each decoding at its measured speed:
All four methods decode the same prompt with the same target model. Playback matches the measured throughput of each method on a single H100 (batch 1).
How speculative decoding works
Speculative decoding was introduced in 2023 as a way to accelerate autoregressive generation without changing the target model's output distribution. A smaller draft model proposes several tokens ahead, the target model verifies the whole block in a single pass, accepts the longest valid prefix, samples a correction after the first rejection, and repeats.
This guarantee of exactness comes from the acceptance rule. Each proposed token x is accepted with probability min(1, p(x)/q(x)), where p is the target, q the draft, and on rejection the token is resampled from the normalized difference max(0, p − q). A short proof shows that the output distribution of the result is exactly that of the target model, and that is why speculative decoding is called lossless: the samples produced are exactly distributed as the target model's output distribution, there are just fewer weight loads per token.
From here the whole field boils down to one question: how many tokens are accepted by the target per verification step? This number is the accepted length and is what the speedup is driven by. And every method since 2023 is an attempt to push it higher.
EAGLE-3: start from the target's features
EAGLE-3 keeps the draft-then-verify algorithm unchanged, but replaces the draft model itself. Instead of predicting tokens from the text alone, the drafter starts from the target model's own hidden representations, which already encode the current context. It reuses features from low, middle, and high layers, fuses them into a single representation g, and predicts draft tokens from there. Because the drafter begins from the target's internal state rather than reconstructing it independently, its predictions stay much closer to the target.
This method changes how the drafter is trained by exposing it to the same intermediate states it must rely on at inference time. After the first speculative step, the target features for newly drafted tokens do not yet exist, so the drafter must continue from its own intermediate outputs. EAGLE-3 reproduces this situation during training by feeding those outputs back into the model, a technique the authors call training-time test. This removes the mismatch between training and inference and lets the drafter predict tokens directly instead of being constrained to imitate the target's hidden states.
The limitation is that drafting remains autoregressive. Each new draft token depends on the previous one, so producing a block of K tokens still requires K small decoder passes. Drafting time still grows linearly with K.
DFlash: predict the whole block at once
Instead of generating draft tokens autoregressively one by one, DFlash replaces the sequential loop with a block-diffusion drafter that predicts the entire block of K tokens in a single forward pass. This removes the linear drafting cost and makes it possible to speculate longer blocks at essentially the same cost.
Predicting every draft token simultaneously changes the acceptance pattern. Early positions still have the full prompt and verified prefix as context, but later positions cannot condition on earlier draft tokens because they are being predicted at the same time. As a result, later draft tokens are accepted less often than earlier ones, a pattern known as suffix decay.
This leaves us with a question: can you recover the missing sequential dependencies without giving up parallel drafting?
DSpark: keep the parallelism, restore the dependencies
To answer this question, DSpark preserves DFlash's parallel draft backbone, adds a lightweight sequential correction, and makes the verification length adaptive.
Recovering sequential dependencies
The backbone still predicts the entire draft block in one forward pass, but a small Markov head then moves from left to right, using each sampled token to adjust the distribution of the next position before it is sampled. Most of the computation remains parallel, while later draft positions regain the context from earlier ones that DFlash lacks, reducing suffix decay.
Adaptive verification
Rather than sending a fixed number of draft tokens to the target model, DSpark predicts a confidence score for every position and uses a hardware-aware scheduler to choose the prefix length. The scheduler balances the expected number of accepted tokens against the measured cost of verification, allowing low-confidence suffix positions to be dropped before the target verifies the selected prefix in parallel.
Accepted length on Qwen3-8B
Evaluated under the paper's protocol—500 GSM8K examples at temperature 1.0—the official DeepSpec checkpoints reach accepted lengths of 5.30 for EAGLE-3, 5.44 for DFlash, and 6.13 for DSpark. Because all three were trained for the same target model under the same conditions, the results provide a direct comparison of the architectural progression described above.
On a real serving engine
So far, we've only talked about speculative decoding algorithms. Real deployments add another layer: the serving engine. Instead of running in a plain PyTorch loop, production inference is handled by systems such as SGLang. To see how the two interact, I benchmarked three successive configurations on the same H100: vanilla PyTorch, vanilla SGLang, and DSpark running inside SGLang.
Replacing the PyTorch decoding loop with SGLang increases throughput from 51.7 to 150.9 tok/s, a 2.92× improvement. Running DSpark on top of the same serving engine raises throughput further to 651.1 tok/s, another 4.32× increase over vanilla SGLang and 12.61× over the original PyTorch baseline. These results show that speculative decoding remains highly effective even when integrated into a production-grade serving engine rather than evaluated in isolation.
Beyond better drafters
Speculative decoding has already gone through several generations in just two years, evolving from independent draft models to production systems such as DSpark. As inference stacks continue to mature, it is becoming less useful to think about decoding algorithms and serving engines separately. The next advances will likely come from designing them together.