F3: Chunked Prefill & Decode Interleaving

Technical Specification

Slices long prompt embeddings to protect inter-token latency.

---

Architecture & Flow

Edge Cases Handled

• KV Cache Exhaustion: In-flight requests are gracefully preempted or swapped to host memory rather than crashing.
• Worker Timeout: Worker heartbeats expire after 5 seconds of silence, automatically re-routing requests to healthy replicas.

---

Say It Out Loud

"For this feature, I prioritized GPU hardware efficiency and predictable user latency. By decoupling token generation into atomic iterations, the scheduler achieves maximum throughput while upholding strict latency SLAs."