Accelerating LLM Inference Cold Starts: From the First Instance to Scale-Out
This article was translated from the Chinese original using AI and reviewed by the author.
As large language model applications grow, inference services have become a core workload in AI infrastructure. For platform teams, the challenge extends beyond simply getting a model to run; they must also ensure that compute capacity keeps pace with changing demand.
On a Model-as-a-Service (MaaS) platform, request volumes can vary significantly between different models, and even the same model’s traffic can fluctuate throughout the day. Ideally, GPU resources would follow demand, scaling out quickly to meet peak serving capacity, and releasing resources for other workloads during quieter periods. In practice, even when GPUs are allocated promptly, it can still take time for an inference service to reach its target capacity.
Obtaining GPU resources is only the first step. Framework initialisation, model weight loading, cache preparation, kernel compilation and CUDA Graph capture all still need to happen before the instance can handle requests. These steps can take a considerable amount of time, which delays the point at which newly allocated GPUs can serve requests. In order to accommodate rising traffic, platforms must either scale out in advance or maintain spare instances during periods of low demand, which incurs additional resource costs.
Startup and recovery speed therefore affect how quickly inference services respond to traffic changes and how flexibly GPU resources can be used.
The key question is how to reduce the need for repeated preparation each time the service scales out. Options include caching model files in advance, sharing downloaded files between nodes, or obtaining model weights from an existing instance’s GPU. While these approaches address different parts of the process, they share one idea: preparation should be completed ahead of time or existing data reused to reduce repeated work when a new instance starts.
This article explores this concept through two scenarios: starting the first instance and scaling out to multiple instances. First, it covers model weight prefetching, caching and concurrent loading. It then examines P2P model file distribution and direct GPU weight transfer, looking at which preparation steps can be completed in advance and which data can be reused across instances.
1. The Cold Start Path and Its Latency
Cold starts in inference serving mainly occur in two scenarios: when a model is deployed for the first time and an instance must be created from scratch, or when a model is already running and the service needs to scale out quickly. Both scenarios require new inference engines, but the resources available for reuse are different. The first instance typically obtains its weights from a model registry, whereas subsequent instances may be able to reuse state from instances that are already running. Accelerating cold starts therefore requires attention to both the startup time of an individual instance and the efficiency of starting multiple instances concurrently.
1.1 How an Inference Service Starts
Before discussing acceleration techniques, consider the steps between creating an inference instance and making it ready to serve. From the platform’s perspective, the process includes resource allocation, container startup, inference engine initialisation, and service readiness. Engine initialisation includes model loading, runtime compilation, and cache preparation. The instance must then pass readiness checks and be added to the service’s routing configuration before it can accept new requests.

Startup time can be decomposed as follows:
T_ready ≈ T_resource_allocation + T_container_startup + T_engine_initialization + T_service_readiness
Here, T_ready represents the time between a scale-out request and the point at which the instance can accept traffic. In practice, some tasks can be completed in advance or run in parallel with others, so startup latency needs to be analysed in terms of the critical path.
Looking inside the engine, the MLSys 2026 paper Breaking the Ice: Analyzing Cold Start Latency in vLLM studies vLLM v0.10.1.1 and divides engine startup into six main stages:
- Framework Bootstrapping: initialise the runtime environment, dependencies, and workers.
- Tokenizer Initialization: load the vocabulary and construct the data structures needed for tokenisation.
- Model Loading: load pretrained weights into GPU memory.
- Torch Compilation: transform the computation graph through TorchDynamo, then compile it or load existing compiled artifacts.
- KV Cache Profiling: run a dummy forward pass to measure runtime memory usage and determine the GPU memory available for the KV cache.
- CUDA Graph Capture: warm up and capture graphs for predefined execution shapes to reduce subsequent CPU scheduling and kernel launch overhead.
The paper further analyses the latency of each stage. For Llama 3.2-3B on an H100, total vLLM engine startup time was 20.32 seconds. Framework dependency loading, graph transformation, and loading compiled artifacts accounted for substantial portions of that time.

The experiment focuses on the initialisation of the engine itself. Since the model weight files were already in the Linux file cache and the compilation caches had been prepared, weight loading accounted for a relatively small share of startup time. When deploying larger models, however, weight loading may become the main bottleneck. For example, the SGLang Fast Engine Recovery analysis of Ling-2.6-1T FP8 reported approximately 495 seconds of weight loading out of roughly 527 seconds of startup time (93.9%) in its test environment.
These results show that the startup latency breakdown depends strongly on model size, cache state, and the execution environment. Weight retrieval and loading, framework initialisation, graph compilation, and CUDA Graph capture can each become bottlenecks. Before choosing an acceleration approach, it is therefore necessary to examine the measured startup times and identify which preparation steps are worth optimising and which existing state can be reused.
The following sections focus on model weights. They first examine how prefetching can reduce data preparation time for the first instance, then consider how reusing existing weights can reduce repeated transfer and loading during scale-out.
2. Starting the First Instance: Model Weight Prefetching and Multilevel Caching
When a model is first deployed, its files—weights, model configuration, tokenizer files, and associated index files—must be retrieved from a remote model registry and downloaded to local storage on the compute node. The inference engine then loads the weights into GPU memory.

Downloading and loading model files sit on the startup critical path and can substantially delay readiness. Preparing weights in advance reduces the wait for data download and loading when the first instance starts.
2.1 Model Weight Prefetching and Multilevel Caching
If the target node has no cached weights, the instance must retrieve model data from remote storage during startup. For models occupying hundreds of gigabytes, remote reads and data transfer can take on the order of five minutes.
One straightforward approach is to prepare the model weights before the instance starts. Based on deployment and scheduling requirements, weights can be prefetched onto the target nodes so that the instance can read them directly from local disks at startup. This reduces the time spent retrieving weights from remote storage.
Two approaches are possible, depending on cache placement and storage medium.
Option 1: Local SSD/RAID Caching
In this approach, model weights are prefetched from remote storage onto the compute node’s local SSDs. The instance then reads them directly from local disks at startup, reducing its dependence on remote storage and network bandwidth. Multiple SSDs can also be combined into a RAID array, allowing their aggregate bandwidth to improve throughput when reading large model weights.

The data path for local caching is relatively simple, but its capacity is limited by the disks on each node. The cache is also tied to the compute node that holds it. As the number of models grows, deciding where to prefetch weights and which cached files to evict requires consideration of model versions and node resources to avoid running out of storage.
Option 2: Distributed In-Memory Caching
Another approach uses host memory across multiple cache nodes to build a shared model weight cache. Model files can be sharded across nodes, allowing an inference instance to read the required data concurrently over RDMA at startup.

Compared with local caching, a distributed memory cache aggregates capacity across nodes, reduces dependence on a single node’s disk capacity, and makes cached data easier to share among compute nodes. However, this approach depends on network transfers and requires additional mechanisms to manage cache shards, handle network contention and recover from failures. These requirements increase system complexity.
The two approaches involve different trade-offs. Local caching is more suitable when deployment locations are relatively stable and weights can be prefetched in advance. Distributed caching is more suitable when there is a wider variety of models and a greater need to share cached data. The two can also be combined to form a multilevel cache across different storage media.
Weight prefetching does not eliminate the initial cost of retrieving weights; it moves data preparation ahead of instance startup. The actual benefit depends on cache hit rate, when prefetching occurs, and the weight-loading path used at startup.
2.2 Concurrent Loaders and Pipelined Loading
After weights are prefetched onto local disks, the inference engine reads them and transfers them to GPU memory. This stage can be improved by increasing read concurrency and overlapping reads with transfers to GPU memory.
| Concept | Meaning | Problem addressed |
|---|---|---|
| Concurrent Loading | Read different weight shards or data blocks simultaneously through multiple threads or asynchronous I/O. | Increase storage I/O concurrency and make full use of local SSD bandwidth. |
| Streaming Loading | Deliver weight data to the loader as batches become ready, without waiting for the entire model to be read. | Begin loading part of the weights before all model weights have been read. |
| Pipelined Loading | Read batch N+1 while transferring batch N to the GPU. | Overlap storage I/O and transfers to GPU memory, reducing serial waits and total model loading time. |
Concurrent reading addresses the problem of storage bandwidth not being fully used. With serial reading, each I/O request must finish before the next set of weights can be read. This makes it difficult to take full advantage of a RAID array built from multiple SSDs or a high-performance storage system. Multiple threads or asynchronous I/O can read different shards simultaneously and improve throughput when storage bandwidth permits. In the diagram, each of four weight blocks takes two seconds to read. With serial reads, D cannot begin until A, B, and C finish. If sufficient bandwidth remains available, the four requests can run concurrently.
Streaming loading changes how weights are delivered. Instead of waiting for the entire model to be read before loading begins, ready weights are passed to the inference engine incrementally. This reduces the wait between reading all the weights and loading them onto the GPU, but does not itself imply that reading and GPU transfers run in parallel. In the diagram, A, B, and C are read in the same order. With streaming loading, A can be passed to the loader as soon as its read completes at the two-second mark, before B and C have been read.
Pipelined loading exploits concurrency between storage reads and transfers to GPU memory. While one batch is being transferred from host memory to the GPU (H2D), the loader reads the next batch. Reads and transfers for different batches overlap, reducing serial waits and allowing total loading time to approach the limit set by the slower stage rather than the sum of all stages. In the diagram, each batch takes two seconds to read and one second to transfer from host memory to the GPU. Reading B overlaps with transferring A, and reading C overlaps with transferring B, reducing the three-batch total from nine seconds to an ideal seven seconds.
Concurrent reading, streaming loading, and pipelined loading can be combined to improve storage throughput and overlap reads with GPU transfers as much as possible, shortening an individual instance’s model loading time.
When a service scales out to multiple instances of the same model, each instance may still read and load the same weights through a slower data path. Improving scale-out performance therefore requires both more efficient loading within each instance and the reuse of existing weight data across instances.
3. Scaling Out: From Model File Distribution to Direct GPU Weight Transfer
When a service scales from one instance of a model to many, each node needs to obtain the same model weights. If every node downloads them independently from remote storage, the repeated transfers place increasing demand on the source’s bandwidth as the number of instances grows. This can extend the time needed to start the entire group of instances.
Unlike when starting the first instance, the cluster already contains model data that can be reused. This includes both model files cached on nodes and weights loaded by existing instances. Reusing this data can reduce the time spent downloading model files and loading weights.
| Dimension | P2P model file distribution | Direct GPU weight transfer |
|---|---|---|
| Reused data | Model files cached on nodes | Model weights held by existing instances |
| Main benefit | Fewer repeated downloads from remote storage | Fewer repeated file reads and a reduction in some weight-loading work |
| Remaining work on the target | Load weights from files into GPU memory and complete engine preparation | Receive weights and complete remaining engine initialisation |
| Example projects | Dragonfly | ModelExpress, Mooncake TransferEngine |
3.1 P2P Model File Distribution
Model files already downloaded into the cluster can be shared between nodes through P2P distribution, rather than downloaded anew from the registry by every node. The target node can retrieve chunks from multiple peers concurrently and make the chunks it has already received available to other nodes. This allows more nodes to participate in distribution and reduces demand on central storage bandwidth.

Distribution must account for cache placement and available bandwidth to avoid creating new transfer hotspots. Dragonfly is a representative implementation, using chunked downloads and peer scheduling to improve large-scale file distribution. Dragonfly documentation
P2P primarily addresses model file distribution. Once the files arrive, the inference engine still needs to read the weights and load them into GPU memory. Concurrent reads and pipelined loading can further reduce the instance’s preparation time.
3.2 Direct GPU Weight Transfer
If instances of the same model are already running in the cluster, reuse can extend to weights in GPU memory. A new instance can obtain weights directly from a GPU that already holds them, reducing repeated reads and weight processing.

Transferring weights across nodes also requires consideration of how the data moves from the source GPU to the target GPU. Where GPUDirect RDMA is supported, the network interface card can access GPU memory directly, reducing the need to copy data through CPU memory. However, network bandwidth can still become a bottleneck when multiple new instances are started at the same time.
Existing projects explore this approach. ModelExpress combines weight source discovery, GPU weight transfer, and model loading. Mooncake TransferEngine primarily supplies the underlying data transfer capability. The two are not mutually exclusive. ModelExpress coordinates weight loading and transfer, and can use Mooncake TransferEngine as its transport backend.
4. Conclusion
Inference cold starts involve several stages, with the bottleneck depending on model size and cache state. For the first instance, prefetching and caching prepare weights ahead of time, while concurrent reading and pipelined loading improve loading efficiency. During scale-out, existing nodes’ model files or GPU weights can be reused to reduce repeated downloads and loading.
These approaches primarily optimise weight retrieval and loading. Each new instance must still complete the necessary framework initialisation, compilation, warmup, and readiness checks. Before choosing an acceleration approach, it is therefore necessary to break down the actual startup time and identify which parts of the process need improvement.