Handle the end of your Budget

Keep a script running near the end of your tokens. Detect when the API lowers max_tokens, tell the 429 errors apart, and find what holds your tokens. Near the end of your tokens explains the rules that this recipe acts on.

You need Python and the openai SDK, set up as in the Quickstart.

Detect a lowered max_tokensCopy Anchor Link

Before Inference runs a request, the API holds the most tokens that the request can use. When your tokens cannot hold all of them, POST /v1/completions lowers max_tokens to the most that fits. It runs the request and says so in the header X-Acs-Max-Tokens-Clamped.

1. Send the request with client.completions.with_raw_response.create. It takes the arguments of client.completions.create, and gives you the response's headers too.
2. Read the header X-Acs-Max-Tokens-Clamped. The API sends it only when it lowered max_tokens.
3. Read the Completion with response.parse().

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
response = client.completions.with_raw_response.create(
    model="llama-8b", prompt="The sky is", max_tokens=200
)
completion = response.parse()
print("X-Acs-Max-Tokens-Clamped:", response.headers.get("x-acs-max-tokens-clamped"))

With 25 tokens left, the program prints the header below. The prompt "The sky is" holds 4 tokens: 3 for its text and 1 for the BOS token. That leaves room for 21 generated tokens.

X-Acs-Max-Tokens-Clamped: requested=200,applied=21,reason=budget,source=client

The header has these fields:

• requested is the max_tokens that you sent, or the default when you left it out.
• applied is the max_tokens that the request ran with.
• reason is budget when your tokens set the limit. It is deadline when a request without a stream left out max_tokens and got 15,600, and deadline+budget when both applied.
• source is client when you set max_tokens, and default when you left it out. A header with reason=deadline has no source.

A Completion that reaches the lowered max_tokens ends with finish_reason "length", as any Completion that reaches its max_tokens does.

Only POST /v1/completions lowers max_tokens. The requests of a Job and the Nodes of a Branch keep theirs, and the Budget refuses them with budget_exceeded instead (Holds of background work).

Handle the 429 errorsCopy Anchor Link

Four errors have the status 429. Branch on error.code:

The openai SDK raises openai.RateLimitError for each. A 429 budget_exceeded or spend_cap has the header x-should-retry: false, so the SDK raises it at once. For a 429 rate_limited or queue_full, the SDK first sends the request two more times, and waits as Retry-After asks. Retries in the openai SDK gives the SDK's rules.

1. Create the client with the SDK's default retries, so that the SDK tries a rate_limited or queue_full again for you.
2. Catch openai.RateLimitError, and branch on error.code.
3. Print the error's message, which says what refused the request.

import os

import openai
from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])

try:
    client.completions.create(model="llama-8b", prompt="The sky is", max_tokens=200)
except openai.RateLimitError as error:
    if error.code == "budget_exceeded":
        print("Out of tokens:", error.body["message"])
    elif error.code in ("rate_limited", "queue_full"):
        print("Still limited after the SDK's tries:", error.body["message"])
    elif error.code == "spend_cap":
        print("GPU work is paused:", error.body["message"])
    else:
        raise

With 10 tokens left in the API key's own Budget, not even one generated token fits after the prompt's 11. The program prints:

Out of tokens: This request can use up to 211 tokens, more than this API key's token budget has left.

The end of the message names what refused the request:

• When the key's own Budget refused it, as here, the message ends more than this API key's token budget has left.
• When your Budget refused it and you have no monthly allowance, the message says more than your token budget has left.
• When your monthly allowance and your Budget refused it, the message says more than your monthly allowance has left this month and more than your token budget has left. Then it gives the day the allowance refills.
• When your own tokens refused it, the message ends with the address of the App's form to ask for more: Ask for more tokens at https://v2.infra.acsresearch.org/usage/request.

Its number is the Hold of the request, here the prompt's 11 tokens and the 200 of max_tokens. Requests that run at the same time can take the room after the API lowered max_tokens. The lowered request then gets this error, and its number is the lowered Hold.

error.body is the error object, and error.code is its code. error.message is the SDK's own summary, which starts with Error code: 429.

Find what holds your tokensCopy Anchor Link

Each running request holds its tokens until it ends, so the requests that run at the same time share what you have left. The Usage page of the App shows the tokens left after their Holds, and the requests that hold them.

1. In the App, open Usage.
2. Under Monthly allowance and Token budget, read the tokens left. The App counts them after the Holds of your running Completions, so a new request fits only in them.
3. Under Completions, find the rows with the status Running. Each of these Completions keeps its Hold until it ends.
4. To see one API key, choose it in All keys.

Read your usage describes each part of the page.

Next stepsCopy Anchor Link

Budgets and Holds explains the monthly allowance, the Budget, and the Hold. Ask for more tokens says how to get a Grant or a larger monthly allowance. Errors shows the 429 error object.