It takes exactly 294 seconds for Tensorfuse infrastructure to spin up a Llama 3.1 8B model on an L40S accelerator from zero to first response. When query traffic to a service drops to just a few times per day, the GPU sits idle most of the time, yet the company must pay for the allocated node around the clock. The drive to optimize budgets pushes platform engineers toward a scale-to-zero scheme—completely tearing down pods during periods of zero load and redeploying them on demand. However, scaling LLMs down to zero saves hardware rental costs at the expense of total interactivity loss: the service becomes useless for real-time applications due to protracted engine initialization.
In a standard web service, replica spin-up takes mere seconds, but here the infrastructure hits a hard performance barrier. Where does this nearly five-minute pause actually go?
Anatomy of a cold start Initial measurements show that physical data transfer accounts for only a minor fraction of the problem. At Tensorfuse, out of 294 seconds of waiting time, downloading the model and transferring weights to the GPU took a mere 94 seconds.
The remaining 200 seconds were consumed by compilation, CUDA graph capture, engine initialization, and imports. Instead of rapidly loading parameters, the machine spends the bulk of its time launching the software pipeline.
In larger-scale systems, latencies multiply. In July, the H Company team reported that a new replica of their largest models took up to 27 minutes to reach a ready state. The Kubernetes autoscaler adds pods as agent workloads grow, but accelerating image downloads, compilation, and engine warming through standard scaling settings is physically impossible.
The metric interception problem Standard hardware telemetry proves useless for managing the lifecycle of such containers. GPU utilization during continuous batching hovers near 100% across completely different task volumes, and occupied VRAM fails to reflect the real incoming message count: the vLLM engine pre-allocates memory for weights and KV cache—90% by default.
The internal queue vllm:num_requests_waiting becomes available on the /metrics endpoint only after vLLM fully boots, meaning it cannot issue a start command when no active pods exist.
In the KEDA HTTP Add-on, this task falls to an interceptor—a reverse proxy that counts requests in processing and passes metrics to KEDA. While the container spins up, the proxy holds incoming packets locally. However, the KEDA HTTP Add-on remains in beta status, introducing direct operational risks for production deployments. The wait queue has rigid limits: for instance, the threshold can be set at 200 requests, after which the server begins returning 503 error codes to clients.
Moreover, the incoming load threshold for spawning a new replica, sourced from the llm-d guidelines, sits at 16 requests.
The promise of zero costs for idle hardware turns into a severe engineering deadlock. Attempting to tear down nodes within a standard cooldownPeriod results in brief traffic lulls deleting a freshly spawned replica, forcing the next request to wait several minutes for startup all over again. Consequently, engineers are forced to manually stretch this interval to half an hour and restrict scaling increments to adding just one pod every five minutes, turning nominal scale-to-zero into standard maintenance of a continuously warmed infrastructure.
Financial consequences of the five-minute wait For a business, a five-minute downtime or response delay translates to direct customer churn and the collapse of real-time AI use cases. When a user or autonomous agent encounters a service that remains silent for nearly five minutes due to cold starts, the enterprise sacrifices the operational agility that justified deploying language models in the first place. Attempts to slash idle GPU rental costs via scale-to-zero yield a paradoxical outcome: companies either pay for continuous hardware operation to keep infrastructure warm, or they forfeit users unwilling to wait for software pipeline initialization, compilation, and CUDA graph capture.
The catch
Cutting cloud bills without shedding customers hits an immediate dead end: scale-to-zero breaks language model responsiveness completely. Platform teams face a hard engineering trade-off — idling nodes saves compute costs, but the very next prompt stalls during heavy engine warm-up and compilation steps. Stretching cooldown periods up to thirty minutes prevents premature shutdowns, yet it quietly brings back the round-the-clock invoice for keeping GPUs warm. The tidy promise of paying strictly for inference seconds ended up requiring fully powered accelerators running around the clock.