speedLM

A serving layer for speculative decoding that puts idle GPU time to work: instead of leaving the accelerator unused between requests, speedLM trains the draft model on that time so the drafts it proposes get accepted more often.

Built with: LLM, Speculative Decoding

GitHub: RyanKim17920/speedlm

• OpenAI-compatible serving layer built on vLLM
• Trains speculative drafts during idle GPU time rather than only at setup
• Improved accepted tokens per verifier step by 13.4% on Qwen3-8B

July – August 2026