F2: Paged KV-Cache Memory Allocator

Technical Specification

Virtual memory page tables for non-contiguous GPU KV attention blocks.

---

Architecture & Flow

Edge Cases Handled

• KV Cache Exhaustion: In-flight requests are gracefully preempted or swapped to host memory rather than crashing.
• Worker Timeout: Worker heartbeats expire after 5 seconds of silence, automatically re-routing requests to healthy replicas.

---

Say It Out Loud

"For this feature, I prioritized GPU hardware efficiency and predictable user latency. By decoupling token generation into atomic iterations, the scheduler achieves maximum throughput while upholding strict latency SLAs."