A Replicate cold start is the delay between sending a prediction request and the model actually starting work, and it can range from a few seconds to several minutes. If you are shipping a product on top of Replicate, that delay is the difference between a snappy feature and a spinner your users abandon. This guide explains exactly why cold starts happen, how long they take for common models, what they quietly add to your bill, and every practical way to reduce or eliminate them. It also covers when the right answer is not tuning Replicate at all but moving latency-sensitive workloads to a platform with warm pools, which is the approach Wireflow takes by keeping popular image and video models hot behind a single API.
What actually causes a Replicate cold start
Replicate runs on a scale-to-zero architecture. When a model has no traffic, its containers are shut down so you are not paying for idle GPUs. That is great for cost, but it means the first request after an idle period has to rebuild everything from scratch: pull the container image, boot the runtime, fetch the model weights from network storage, and load them into GPU memory. For large models this last step dominates, because moving tens of gigabytes of weights into VRAM is bound by hardware transfer speeds, not software cleverness. Teams that migrate from Replicate to a canvas API usually cite this boot sequence as the single biggest source of unpredictable latency.
Three factors decide whether your specific request hits a cold start:
- Model popularity. High-traffic public models like FLUX and SDXL stay warm because shared usage keeps containers alive. Niche community models and private models scale to zero constantly.
- Idle window. If your traffic is bursty, every burst after a quiet period pays the boot cost again.
- Model size. An 8 GB Whisper Large v3 checkpoint takes 1 to 2 minutes to boot; a small utility model may be ready in seconds.
If your product chains several models together, the problem compounds, because each model in the chain can cold start independently. That is one reason running Replicate models through an MCP server or a managed pipeline layer has become a common pattern: the orchestration layer can keep the whole chain warm instead of leaving each step to its own idle timer.
How long Replicate cold starts actually take
Replicate does not publish per-model boot benchmarks, but community reports and official announcements give a consistent picture. These are typical ranges, not guarantees:
| Model / scenario | Typical cold start |
|---|---|
| Fine-tuned SDXL and Llama 2 (optimized path, post-2023) | Under 1 second |
| Popular public models (FLUX, SDXL) with warm containers | None, runs immediately |
| Community image models, cold | 10 to 60 seconds |
| Whisper Large v3 (~8 GB weights) | 1 to 2 minutes |
| Large custom models and 70B-class LLMs, cold | 2 to 5 minutes |
| Infrastructure incidents (reported on GitHub and Hacker News) | Up to 30 minutes |
The 2 to 3 minute range comes up repeatedly in user reports for custom models, and the worst cases stretch far beyond that during platform incidents. If you are evaluating providers on latency, put these numbers next to the per-request pricing on the Replicate pricing breakdown so you are comparing total time-to-output, not just GPU-second rates.
A related question that trips people up: the "starting" state in the Replicate dashboard covers both container boot and your model's own setup() code. If your prediction sits in "starting" for minutes, it is almost always weight loading, not a queue problem.
The hidden cost: cold starts show up on your bill
Cold starts are not just a latency problem. On hardware-billed platforms you pay for GPU time from the moment your container starts doing work, and model setup time can land on your side of the meter. The math is easy to ignore and painful at volume: on an A100 at roughly $0.00115 per second, a 30 to 60 second cold boot costs $0.035 to $0.069 before your actual prediction begins. If a fifth of your requests hit cold containers, that overhead can exceed what you pay for the inference itself on cheap models. Per-output platforms price this differently, which is why the fal.ai pricing model of flat per-image billing looks attractive for bursty traffic: the provider absorbs boot costs instead of passing them through.
This is also where budget control matters. Bursty traffic plus per-second billing plus unpredictable boot times makes monthly costs hard to forecast, and an AI generation API with spend limits removes at least the worst-case surprise even if it does not fix latency.
Five ways to reduce Replicate cold starts
You have five practical levers, ordered from cheapest to most involved.
- Use popular public models where possible. FLUX, SDXL, and other high-traffic models are effectively always warm. If a mainstream model gets you 95% of the quality of a niche fork, the latency win usually justifies the switch. Browse what is consistently hot in a shared model catalog before committing to an obscure variant.
- Send keep-warm pings. A cheap no-op prediction every few minutes prevents scale-down. It works, but it is brittle: you are gaming an idle timer that Replicate can change, and you pay for every ping.
- Use Deployments with
min_instances: 1. This is the official fix. A dedicated instance stays booted, so cold starts disappear for your model. The catch is cost: an always-on H100 runs in the neighborhood of $36 per day, which only makes sense with sustained traffic. Do the break-even math against your request volume first. - Shrink what has to load. Quantized weights, smaller variants, and trimming your
setup()code all cut boot time directly. Teams comparing options in the best Replicate pricing tools roundup consistently find that model choice moves latency more than any infrastructure toggle. - Batch and queue by design. If your use case tolerates async processing, queue requests and process them in bursts so one cold start is amortized across many predictions, a pattern covered in how to run batch image generation via API.
When the right fix is leaving scale-to-zero behind
If your product needs consistent sub-5-second responses, no amount of tuning changes the fundamental trade Replicate has made: scale-to-zero saves idle cost by pushing boot cost onto the first request. Providers built on warm pools make the opposite trade. fal.ai keeps common image models hot and typically serves the first request in under 3 seconds. Together AI bills per output with always-warm serving. Managed canvas platforms go further by keeping entire multi-model pipelines warm, which matters once you chain AI models together and a single user action touches three or four models in sequence.
The decision framework is simple:
- Steady, high traffic on one custom model: stay on Replicate, pay for a deployment with a minimum instance.
- Bursty traffic on popular models: move latency-sensitive paths to a warm-pool provider and keep Replicate for offline jobs.
- Multi-model pipelines: use an orchestration layer with hosted, always-warm models, as described in how to chain multiple AI models in one API call, so you never multiply cold starts across steps.
Whichever route you take, measure before and after. Log time-to-first-byte per prediction for a week, split by warm and cold, and you will know exactly what a fix is worth, the same discipline recommended for any AI inference API evaluation.
Try it yourself: Wireflow has a pre-built MCP pipeline you can open and run, configured to run hosted models with no cold start management on your side.
FAQ
How long does a Replicate cold start take?
Anywhere from under a second for optimized fine-tunes to several minutes for large custom models. Typical community models take 10 to 60 seconds cold, Whisper Large v3 takes 1 to 2 minutes, and community reports document multi-minute waits for 70B-class models.
Does Replicate charge for cold start time?
Container boot on Replicate's side is generally not billed, but your model's own setup time can count toward billed compute on hardware-billed models. The practical effect is the same either way: users wait, and bursty workloads carry overhead that per-output providers absorb.
How do I stop cold starts on Replicate completely?
Create a Deployment for your model and set min_instances to 1 or more. A dedicated instance stays warm at all times. Expect to pay for the idle GPU, roughly $36 per day for an H100-class instance.
Why is my prediction stuck in "starting" for minutes?
The "starting" state covers container boot plus model weight loading. For large models, transferring weights from network storage into GPU memory is the bottleneck. It is normal for 8 GB-plus models, and it recurs every time the model scales to zero.
Do popular models like FLUX avoid cold starts?
Mostly, yes. High-traffic public models stay warm because shared usage keeps containers alive. You only reliably hit cold starts on niche community models, private models, and custom fine-tunes with sporadic traffic.
Are keep-warm pings a good idea?
They work as a stopgap: a cheap prediction every few minutes prevents scale-down. But you pay for every ping, the idle timeout is not contractual, and a deployment or a warm-pool provider is more reliable for production traffic.
What is the fastest alternative for latency-sensitive workloads?
Warm-pool providers. fal.ai typically serves first requests in under 3 seconds for hosted image models, and per-output platforms like Together AI keep serving infrastructure always on. For multi-model pipelines, a managed workflow platform keeps the entire chain warm.
Does cold start behavior differ between image and language models?
The mechanics are identical; the difference is weight size. A 2 GB image model boots much faster than a 140 GB language model, so LLM-heavy stacks feel cold starts hardest and benefit most from dedicated instances.
Conclusion
Cold starts are the tax Replicate charges for scale-to-zero economics. For steady traffic on one model, a deployment with a minimum instance is a clean fix. For bursty or multi-model products, the better move is usually shifting latency-sensitive paths to always-warm infrastructure. Platforms like Wireflow sidestep the problem entirely by keeping hosted models hot behind one API, so the first request is as fast as the hundredth. Whatever you choose, measure warm and cold latency separately first; the data usually makes the decision for you.
