Cold starts and long requests
Inference starts each Base model's GPU when requests come and stops it when they end, so a request can wait for a start. The API also waits for Inference only so long, and these limits decide how to send a long request.
Warm and cold models
Inference keeps no container of a Base model running while no requests come, because a GPU that waits all day costs money all day. An admin can keep a model Warm at set hours instead (Keep a model Warm). After its last request, a container of llama-8b stays up for 30 minutes, and a container of llama-405b or trinity-truebase for 5 minutes. Then it stops. The next request waits while a new container starts and loads the model. After an update to Inference, the first few starts take longer.
A start of llama-405b or trinity-truebase loads eight GPUs. In testing, the two starts took 25 and 17.6 minutes, longer than the 14-minute wait. So the first request to /v1/completions for a cold one times out.
A Job's requests, and the App's Runs and Branches, wait up to 45 minutes for Inference's answer to start. No client holds a connection open for them, so they wait through a start of a cold model.
A Warm model is one with a container that serves, so that its next Completion starts at once. GET /v1/models/{model}/status answers warm: true for such a model. While a model starts, the route also says the start's stage in boot.stage_label and how long the stage has run in boot.stage_age_s. Inference reports each container's start, so the route never starts a GPU.
Check whether a model is Warm shows the route.
The 14-minute wait
The API waits up to 14 minutes for Inference to answer a request. In a stream, it waits up to 14 minutes between two chunks. If Inference does not answer in that time, the API answers 504 upstream_deadline with the message Could not reach Inference: Request timed out. and the header x-should-retry: false. In a stream that has started, the same error comes as the last event before [DONE].
The header tells the openai SDK not to send the request again, and the API does not try the request again either. Inference did not answer for 14 minutes, so another try at once would wait as long. It would also start the work on a GPU again. Which errors to try again says when to send the request again.
A request that times out before Inference sends a token costs nothing, because the API releases its Hold. A stream that times out between two chunks pays for the tokens that it sent.
Requests with a stream
A stream starts when Inference starts its answer. From then on, the stream has a : keepalive comment line every 5 seconds, also while Inference is silent between two chunks. SSE clients and the openai SDK skip comment lines, and the comments keep the connection busy, so a read timeout does not cut a slow stream. A stream that fails before Inference starts its answer gets an error status, as Errors lists.
Requests without a stream
A request without a stream gets its whole answer at the end. If Inference takes more than 150 seconds, the API starts the answer anyway. It sends status 200, then a space every 5 seconds while Inference works, then the Completion's JSON. JSON allows spaces before a value, so the openai SDK and json.loads read the body as usual. The spaces keep the connection busy, so a read timeout never cuts it for being idle.
After the 200 has gone out, the status cannot change. A failure after that point, such as the 14-minute timeout, comes as the body instead: the error object, with its request_id. So a long request can answer 200 with an error, and only the body tells the two apart.
A request without a stream must finish within the 14 minutes. The API counts on Inference to write 20 tokens a second. At that rate, 13 minutes is 15,600 tokens, a minute inside the wait. So a request without a stream that sets max_tokens above 15,600 cannot finish in time. The API answers it with 400 deadline_infeasible at once, before it holds any tokens, and with x-should-retry: false.
When a request without a stream leaves max_tokens out, it gets the room left in the model's context. The API lowers that to 15,600 when the room is larger. The header X-Acs-Max-Tokens-Clamped then says reason=deadline. The context of trinity-truebase is 8,192 tokens, so on that model the room is always smaller, and only a max_tokens that you set meets this limit. The context of llama-8b and llama-405b is 131,072 tokens, so a short prompt there gets 15,600.
A stream has no such limit, because its client reads each token as it comes. For a long output, a stream is the way to send the request. A client that waits for a long request without a stream needs a read timeout above 14 minutes. It also needs automatic retries off, because each try starts the work on a GPU again, and each cut try pays its whole Hold.
Next steps
Set the client for long requests gives the timeout and retry settings for the openai SDK. Stream a Completion reads the tokens as Inference writes them. Time limits lists every limit in time.