F1: Continuous Iteration-Level Batching

Technical Specification

Dynamically updates batch membership at every autoregressive forward step.

---

Architecture & Flow

Edge Cases Handled

• KV Cache Exhaustion: In-flight requests are gracefully preempted or swapped to host memory rather than crashing.
• Worker Timeout: Worker heartbeats expire after 5 seconds of silence, automatically re-routing requests to healthy replicas.

---

Say It Out Loud

"For this feature, I prioritized GPU hardware efficiency and predictable user latency. By decoupling token generation into atomic iterations, the scheduler achieves maximum throughput while upholding strict latency SLAs."