Appearance
Step-by-Step Build Order
- Request Schemas & Token Data Structures: Implement
backend/src/models/defining prompt, generation params, and token statuses. - Admission Controller & FIFO Queue: Build
scheduler/admission.pyandscheduler/queue.pywith capacity limits. - Paged KV Cache Allocator: Implement
kv/paged.pyblock manager and logical-to-physical page table mapper. - Continuous Batching Iteration Loop: Construct
batching/continuous.pyinterleaving prefill slices and decode steps. - Worker Registry & Router: Implement
coordination/workers.pyandrouting/router.py. - FastAPI REST & SSE Streaming: Wire endpoints in
api/app.pywith streaming async generators. - React Real-Time Telemetry Dashboard: Build
frontend/src/visualizing active queues and latency metrics.