Retry requests safely

A broken connection or a 5xx answer can leave you unsure whether a request ran. This guide shows how to send it again so that the API does the work and charges your Budget once. Errors describes each answer.

Send a request again safelyCopy Anchor Link

POST /v1/completions, POST /v1/completion-jobs, POST /v1/harvest, and every Loom write take the Idempotency-Key header. A cancel of a Job or a harvest needs none: a second cancel answers the Job as it is.

1. Make a new key for each new request, such as a UUID. A key has 1 to 255 characters.
2. Send it in the Idempotency-Key header.
3. If the request fails or the connection breaks, send the same request with the same key again.

KEY=$(uuidgen)
curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $KEY" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'

uuidgen makes the key in the shell. Run the curl command again with the same KEY, and it prints the same Completion. The Python program prints True: its second request got the first request's Completion, and Inference ran once. In the openai SDK, extra_headers sends the header, and the SDK sends it again with each of its own tries.

With a key, the API answers each repeat of a request in this way:

• After the first request finished, the same request gets the first answer. A change to your Budget or to the served models since then does not change it. A Completion comes with the first answer's X-Acs-Max-Tokens-Clamped header. A Job, a Loom, or a Branch comes as it is now, and a Job comes with 200 in place of 202.
• While the first request still runs, the same request gets 409 with the code running and Retry-After: 1.
• A stream runs once. The same stream request again gets 409 no_answer with x-should-retry: false, and the API does not run it or charge it again. Send a new key to run it again.
• A Completion without a stream runs on when your client goes away, and pays only for its usage. The same request again gets its answer.
• A request that failed before the API charged it, for example with 429 budget_exceeded or an error from Inference, frees its key. The same request with the key runs.
• Another request with the same key, or the same key on another route, gets 409 idempotency_conflict with x-should-retry: false.

The API keeps each key for 24 hours. All your API keys share one set of keys, and another User's keys are apart from yours. A request without the header runs each time that it arrives.

Which errors to try againCopy Anchor Link

Send the same request again, with the same Idempotency-Key, after these answers:

• 429 rate_limited or queue_full: wait for the time in the Retry-After header. A full Job cap is different: it answers 429 job_capacity with x-should-retry: false, so submit again when one of your Jobs ends.
• 429 harvest_concurrency_exceeded: wait for the time in the Retry-After header, or until one of your harvests ends.
• 429 harvest_capacity_exceeded: wait for the time in the Retry-After header.
• 409 running: wait 1 second.
• 503 db_pool_exhausted or circuit_open: wait for the time in the Retry-After header. It is 1 second when the API's database was busy, and up to 60 seconds while the API holds a model back. Nothing ran, and you pay nothing.
• Any error with retryable: true, such as 502 upstream_unreachable or vllm_oom: wait retry_after_seconds, the same as the Retry-After header.
• 504 upstream_deadline: wait a few minutes. Inference did not answer for 14 minutes, so another try at once waits as long, and starts the work on a GPU again.

After these answers, check or change the request before you send it again:

• 500 internal_error, with x-should-retry: false: the API failed, and the work may have run. If the request had an Idempotency-Key, send it again with the same key. It gets the first answer if the work finished, and the API does not charge it twice. Without a key, look for the result first, for example in GET /v1/looms. Report the error with its X-Request-Id.
• 429 budget_exceeded: ask for fewer tokens, with a smaller max_tokens or fewer prompts or choices, or ask for more tokens.
• 429 spend_cap: new GPU work is paused (Spend limit). Try again after the Retry-After header's hour.
• 409 no_answer: the stream ran already. Send a new key to run it again.
• 409 idempotency_conflict: the key was used with another request. Send a new key with each new request.
• 429 harvest_quota_exceeded: the API key started its most harvests this month. Submit the Job in the next UTC month.
• 429 spend_cap: the API paused new harvests to limit GPU spending. Wait for the time in the Retry-After header.
• Any other 4xx status. Fix what message and param name.

Status codes lists every answer.

Set the client for long requestsCopy Anchor Link

A Completion can wait while Inference starts a cold model, and a long one runs for minutes. The API waits up to 14 minutes for Inference (Cold starts and long requests).

1. Create the client with timeout=900. The SDK then waits 900 seconds, longer than the API's 14 minutes.
2. Create it with max_retries=0. Each try that the SDK sends after a broken connection starts the work on a GPU again. Without a key, a cut Completion without a stream also pays its whole Hold.
3. Send each request with an Idempotency-Key, and send it again yourself after a broken connection.
4. Check the answer for error. After 150 seconds, a Completion without a stream gets its 200 status, and a later failure comes as the body.

import os
import uuid

from openai import OpenAI

client = OpenAI(
    base_url=os.environ["ACS_API_BASE"],
    api_key=os.environ["ACS_API_KEY"],
    timeout=900,
    max_retries=0,
)
headers = {"Idempotency-Key": str(uuid.uuid4())}
completion = client.completions.create(
    model="llama-8b", prompt="The sky is", max_tokens=2, extra_headers=headers
)
error = getattr(completion, "error", None)
if error:
    print(error["message"])
else:
    print(completion.choices[0].text)

The SDK raises no exception for an error in the body of a 200 answer. It returns a Completion whose error field holds the error object. With curl, set --max-time above 840 seconds, or leave it out.

Retries in the openai SDKCopy Anchor Link

The openai SDK sends a request again by itself, two more times by default, after:

• a connection error or a timeout
• 408, 409, and 429 answers, and every 5xx answer

Before each try, it waits as long as the retry-after-ms or Retry-After header says, up to 2 minutes. Without either header, it waits about half a second, then twice as long each time, up to 8 seconds.

It does not try again after an answer with x-should-retry: false, or with a Retry-After of more than 2 minutes. The API sends x-should-retry: false with these answers, so the SDK raises them at once:

• 409 for a Node that is not in a state for the action, 409 harvest_not_cancellable, 409 idempotency_conflict, and 409 no_answer
• 429 budget_exceeded, 429 job_capacity, 429 harvest_quota_exceeded, and 429 spend_cap
• its own 500
• 413 request_too_large, 504 upstream_deadline, and each error of Inference whose retryable is false

To raise every error at its first answer, create the client with max_retries=0. Errors in the openai SDK lists the error classes.