Budgets and Holds

Before Inference runs a Completion, the API holds the most tokens that the Completion can use, from your monthly allowance or your Budget. So requests that run at once never spend more tokens than you have.

Your monthly allowanceCopy Anchor Link

Your monthly allowance is the number of tokens that you may spend in each calendar month, in UTC, across your API keys and the App. An approval gives you a monthly allowance of 5,000,000 tokens by default, so you can start work without waiting for a Grant. An admin can set another allowance for you. Tokens lists the default.

A request takes its Hold from this month's allowance when the whole Hold fits there, and from your Budget when it does not. The API does not split one Hold between the two. Your allowance is spent first, so your Budget lasts longer.

On the first day of each month, the count starts again from 0. Tokens that you did not spend do not carry over. A Hold counts toward the month in which it started. Tokens that it gives back after the month changed do not change the new month's count.

Your BudgetCopy Anchor Link

Your Budget is the number of tokens that you may still spend when this month's allowance cannot hold a request. It has no period and no reset. It grows only when an admin adds tokens with a Grant or sets it to a higher number, or when an Invite brings tokens. It shrinks by the Usage of each Completion that it pays for.

An approval starts your Budget at 0 tokens, unless the admin adds a Grant with it, and the monthly allowance pays for your first requests. If you joined with an Invite, your Budget starts with the Invite's tokens.

ACS can also give an API key its own Budget, with a Grant for that key. A key's own Budget limits what one script can spend, and Ask for more tokens says how to ask for one. Each Hold of that key comes out of your allowance or your Budget, and out of the key's Budget as well. The key stops when either one cannot hold a request. A key without its own Budget spends only from your allowance and your Budget.

The HoldCopy Anchor Link

Before the API sends a request to Inference, it takes a Hold: the most tokens that the request can use. For each prompt, the Hold is the prompt's tokens and n × max_tokens, with max_tokens counted as at least 1. With echo and max_tokens: 0, Inference still generates and counts 1 token for each choice.

The API counts a text prompt with the model's own tokenizer, the files that Inference loads. It counts the tokens that Inference reads. The count includes the BOS token when add_special_tokens is on, as it is by default. A prompt of token ids counts one token for each id.

Some models have no tokenizer in the API. For them, a text counts one token for each byte of its UTF-8 text, and 1 for the BOS token. A byte-level tokenizer never makes more tokens than bytes, so this count can only be too high. The tokenizer reads at most 8 bytes of a request's texts for each token of the model's context. The texts after that count by their bytes in the same way.

For example, "The sky is" with max_tokens: 2 on llama-8b holds 6 tokens: 4 for the prompt with its BOS token, and 2 for max_tokens. A request that stops before max_tokens gives back the tokens that it did not write.

While a request runs, its Hold counts as spent. Two requests at once never hold more tokens than you have, and each one sees what the others hold. The App's Usage page shows the tokens left after their Holds, and which requests still run, as Read your usage says.

How a Hold endsCopy Anchor Link

Each Hold ends once, by one rule for POST /v1/completions, Jobs, Looms, and the Playground. It settles to the tokens that the request used, or the API releases it and the request costs nothing.

A Hold settles in these cases:

• When Inference answers, you pay the Usage that it reports, and the rest of the Hold goes back.
• When a stream stops early, because you closed it or it broke, you pay for the tokens that it sent. A Loom Node runs as a stream, so a Node that you stop pays for the text that it grew.
• When a request without a stream stops after it reached Inference, you pay the whole Hold. A Job that you cancel while it runs is such a request.

A request without a stream pays its whole Hold because Inference reports its Usage only at the end. When the request stops in the middle, the API has no count of the tokens, and Inference may have done the work. A stream tells the API how many tokens it sent so far, so a cut stream pays only for those.

A released Hold costs nothing, and the Usage page does not list its Completion. The API releases a Hold in these cases:

• Inference refuses the request, or fails before it sends any token, a timeout included.
• The request stopped before it reached Inference.
• The API itself failed.
• The API stopped a request of a Job, a Node, a Playground Run, or a harvest, so that it runs again later.

If a Completion uses more tokens than its Hold, the API takes the rest from what you have left, down to 0.

Near the end of your tokensCopy Anchor Link

When the tokens that you have left cannot hold a request to POST /v1/completions, the API does not refuse the request at once. It lowers max_tokens to the most that fits, and runs the request, so that a script near the end of its tokens keeps making progress. The answer then has this header:

X-Acs-Max-Tokens-Clamped: requested=200,applied=21,reason=budget,source=client

Here, 25 tokens were left, and the prompt "The sky is" holds 4 of them on llama-8b. So the API lowered max_tokens from 200 to 21, and a choice can end at 21 tokens with finish_reason: "length". source is client when you set max_tokens, and default when you left it out. reason is budget here. It is deadline+budget when the API first lowered a max_tokens that you left out to the time limit of a request without a stream. Requests without a stream explains that limit.

Only when not even one token for each choice fits does the API refuse the request. It answers 429 budget_exceeded, and the request does not reach Inference. The message names what ran short: your Budget, the key's own Budget, or your monthly allowance, with the day on which it refills. Not enough tokens shows the error.

Jobs, Loom Branches, and Playground Runs never lower a max_tokens that you set. A request of theirs that your tokens cannot hold fails with 429 budget_exceeded, and its max_tokens stays as you set it.

Holds of background workCopy Anchor Link

A Job, a Branch, or a Run holds no tokens when you make it. Each Job, each Node, and each Run takes its own Hold when it starts to run, as a request to POST /v1/completions does.

When you make a Job, a Branch, or a Run, the API checks only that your tokens can hold its smallest request. If they cannot, it refuses the work with 429 budget_exceeded and makes nothing. The request can still find too few tokens when it runs. A Job then fails, and its result is the 429 error object. A Node fails with the same error.

A harvest generates no tokens, so it holds the tokens of its prompts only. It counts one token for each byte of each prompt's UTF-8 text, and 1 more for each start token. When you submit it, the API checks that your tokens can hold all of that, and answers 429 budget_exceeded if they cannot. The harvest takes its Hold when the API starts it, and keeps it while its GPU run lasts.

A harvest's Hold ends by its own rule, because Inference counts its tokens only when it completes:

• A completed harvest pays the tokens that Inference read, up to its Hold. Its tokens field shows them.
• A harvest that you cancel while it is running and its GPU run started pays its whole Hold. Any other cancel costs nothing.
• A failed harvest costs nothing.

The App's Usage page lists a harvest's Hold under the source Harvests.

A Node's prompt is the text of the path from the root to its parent. So a Node deep in a Loom holds more tokens than a Node near the root. A Branch of 4 Nodes takes 4 Holds, one for each Node.

A Completion that waits in its API key's line for a place to run holds no tokens until it starts. Traffic gives the size of that line.

Activations and Steering vectors cost no tokens of their own. A request with them holds and spends tokens as any other Completion does.

Next stepsCopy Anchor Link

Handle the end of your Budget reads the header and the 429 errors in code. Ask for more tokens says whom to write to. Tokens lists every number about tokens.