Skip to content

Gaps & Roadmap Improvements ​

  • P0: Implement speculative decoding to accelerate decode iterations.
  • P1: Add tensor-parallel and pipeline-parallel multi-GPU partitioners.
  • P2: Implement host-to-device KV cache swapping for memory oversubscription.