Limits
The API and the App limit what one request, one API key, and one User can do. Each table gives the limit, its number, and what the API or the App answers past it. An error with a code is the error object, as The error object shows.
Tokens
| Limit | Number | Past the limit |
|---|---|---|
| Monthly allowance of a User | 5,000,000 tokens each UTC month, by default. An admin can set another for one User. | 429 budget_exceeded. Near the end, POST /v1/completions first lowers max_tokens to what fits. |
| Generated tokens of one request | 32,000 in all: the number of prompts × n × max_tokens | 400, param max_tokens |
max_tokens of a request without stream | 15,600. A request without max_tokens gets 15,600 at most. | 400 deadline_infeasible |
| Prompt and generated tokens of one request | The model's max_model_len: 131,072 for llama-8b and llama-405b, 8,192 for trinity-truebase | 400 context_length_exceeded |
Your monthly allowance is spent first, then your Budget. Budgets and Holds says how a request holds and spends tokens, and when the API lowers max_tokens.
The other caps of one Completion, such as logprobs and the size of a capture, are in Size caps. GET /v1/models lists them for each model in limits (Models).
Traffic
| Limit | Number | Past the limit |
|---|---|---|
| Requests that start work, for each API key | 600 a minute: POST /v1/completions, POST /v1/completion-jobs, POST /v1/harvest, and Branches | 429 rate_limited, with Retry-After in seconds |
Other requests to /v1, for each API key | 600 a minute | 429 rate_limited, with Retry-After in seconds |
| Completions that run at once, for each API key | 16 | A Completion waits in the key's line. |
| Completions that wait in the line, for each API key | 64 | 429 queue_full, with Retry-After: 5 |
| Requests in flight to one model, from all keys and the App | 100 | 503 with Retry-After: 5 for POST /v1/completions. A Job, a Run, and a Branch wait for a place. |
| Playground Runs, Completions, Jobs, and Branches that you start in the App | 600 a minute for each User | The App says when to try again. |
Requests to /v1 with a missing, wrong, or revoked API key, from one address | 60 a minute | 429 rate_limited, with Retry-After in seconds |
Each rate limit is a bucket. A full bucket takes a burst of 600 requests, then refills one request every 0.1 seconds. Each API key has its own buckets, and the App's limit is apart from them. The message of a 429 names the limit and says how many seconds to wait.
The line counts only POST /v1/completions. A Completion takes its place when it starts and keeps it until it ends, also a stream. A waiting Completion holds no tokens and runs in the order it came. It leaves the line when you close the connection. Jobs, harvests, and Loom Nodes run in the background and take no place.
The openai SDK sends a request that got 429 rate_limited again by itself. Retries in the openai SDK says how.
API keys
| Limit | Number | Past the limit |
|---|---|---|
| API keys of one User | 10. A paused key counts, and a revoked key does not. | 409 when you make a key. At the limit, the App turns off New key and shows You have 10 API keys, the most that a User may have. Delete one to make another. |
Jobs
| Limit | Number | Past the limit |
|---|---|---|
| Requests in one Job | 1 | A body is one request of POST /v1/completions. |
Jobs queued or running with one API key | 8 | 429 job_capacity, with x-should-retry: false |
Jobs queued or running of one User | 32 | 429 job_capacity, with x-should-retry: false |
Jobs queued or running of all Users | 128 | 429 job_capacity, with x-should-retry: false |
| Ended Jobs of one User whose results the API keeps | 100 | 429 job_capacity, with x-should-retry: false |
| Ended Jobs of all Users whose results the API keeps | 1,000 | 429 job_capacity, with x-should-retry: false |
| Jobs that run at once | 4, for all Users | A Job waits queued. |
A Job past a cap is not created, and the message names the cap. A cap of Jobs queued or running frees a place when a Job ends. A cap of kept results frees one when the result of a Job expires, 24 hours after the Job ended (Kept data).
Harvests
| Limit | Number | Past the limit |
|---|---|---|
| Prompts in one harvest | 1 to 4,096, each of 1 or more characters | 400, param prompts |
| Tokens of one harvest's prompts, counted as characters / 4 | 2,000,000 | 400, param prompts |
| Tokens of one prompt in the files | The model's max_model_len - 1: 8,191 on each served model | The harvest keeps the first 8,191 tokens of the prompt. |
Layers in a list in layers | 1 to 256, each a Layer of the model | 400, param layers |
Prompts in one shard file (shard_size) | 1 to 4,096, and 32 when you leave it out | 400, param shard_size |
batch_size | 1 to 64, or null | 400, param batch_size |
Directions in project_onto | 1 to 64, each of the model's capabilities.hidden_size | 400, param project_onto |
| Harvests that one API key starts in a UTC month | 200. An admin can set another number for your keys. | 429 harvest_quota_exceeded, with x-should-retry: false |
Harvests on llama-8b pending or running of one User | 3 | 429 harvest_concurrency_exceeded, with Retry-After: 30 |
Harvests on llama-405b or trinity-truebase pending or running of one User | 1 | 429 harvest_concurrency_exceeded, with Retry-After: 30 |
Harvests on llama-405b or trinity-truebase pending or running of all Users | 2 | 429 harvest_capacity_exceeded, with Retry-After: 30 |
Reads of a harvest that wait (wait) at once | 4 for each API key, and 8 in all | The API answers at once with 200, X-Acs-Longpoll: declined, and Retry-After: 3. |
A harvest past a cap is not created, and the message names the cap. A cap of harvests pending or running frees a place when one of them ends. The count of the month leaves out a harvest whose GPU run never started, such as one that you canceled while it was pending. It starts again on the first day of each UTC month.
Without layers, a harvest reads the quartiles of the model's Layers: capabilities.n_layers / 4, / 2, and × 3 / 4. That is Layers 8, 16, and 24 on llama-8b. The cap of 2,000,000 counts characters only. The tokens that a harvest holds are counted another way, as Holds of background work says. Harvest Activations shows each refusal.
Looms
| Limit | Number | Past the limit |
|---|---|---|
| Child Nodes of one Branch | 1 to 8 (n) | 400, param n |
| Tokens of one Node | 1,000 (max_tokens) | 400, param max_tokens |
| Nodes of one Loom | 2,000 | 409 for a Branch, a split, or a sibling. 400 for an imported file. |
| Looms in the list | 100: the pinned ones first, then the others by their last change | An older Loom stays, and a read of its id finds it. |
| Nodes that run at once | 8, for all Users | A Node waits queued. |
| Characters of a Loom's title | 120 | The API keeps the first 120 characters. |
| Characters of a title from the root | 40, from the root's first line | The API keeps the first 40 characters. |
The 409 for a full Loom says how many more Nodes fit. A Loom's Nodes include each Node that you have not deleted, at any depth.
Spend limit
The API pauses new GPU work when the GPUs cost $3,000 in 24 hours, and an admin can pause it too. Only an admin lifts the pause.
| While the pause is on | What happens |
|---|---|
POST /v1/completions, POST /v1/completion-jobs, POST /v1/harvest, a Branch, and the App's Runs | 429 spend_cap, with Retry-After: 3600 and x-should-retry: false |
| A Completion, a Job, a harvest, or a Node that already runs | It finishes. |
A Job or a Node that is queued, or a harvest that is pending | It waits, and runs when the pause ends. |
| A Job, a harvest, or a Node that the API puts back in the queue during the pause, as at a restart | It waits too (When the API restarts). |
Request size
| Limit | Number | Past the limit |
|---|---|---|
Body of one request, on every route but POST /v1/harvest | 8 MiB (8,388,608 bytes) | 413. A body sent in chunks, without Content-Length, gets 413 when it passes the cap. The API checks the key first. |
Body of POST /v1/harvest | 50 MiB (52,428,800 bytes) | 413. A body sent in chunks, without Content-Length, gets 413 when it passes the cap. The API checks the key first. |
Idempotency-Key | 1 to 255 characters | 400 |
| Fields of one Completion | The caps in Size caps | 400, param names the field |
Time limits
| Limit | Number | Past the limit |
|---|---|---|
| Wait for Inference | 14 minutes for an answer, and 14 minutes between two chunks of a stream | The error object of 504 upstream_deadline, Could not reach Inference: Request timed out., with retryable: false. The API has sent status 200 by then. A request without a stream gets the error object as its body. A stream gets it as its last event (The 14-minute wait). |
| Wait for Inference's answer to start, for a Job or the App's Runs and Branches | 45 minutes, through a start of a cold model | The request fails with Could not reach Inference: Request timed out. |
| Wait for the answer of a Completion without a stream | 150 seconds | The answer starts: 200, then a space every 5 seconds until the Completion's JSON. |
| Wait for Inference's answer to a stream to start | 2 seconds | The stream starts: 200, then a : keepalive comment line every 5 seconds until Inference answers. A later error is the last event. |
| Tries to sign in, ask for a reset email, or set a new password, from one address | 5 a minute | The App says how many seconds to wait. |
| Password link of the approval email | 7 days, used once. Only the newest link works. | The link no longer works. |
| Password reset link | 1 hour, used once | The link no longer works. |
| Session in the App | 30 days from its last renewal. The App renews it at most once a day while you use it. | The App takes you to sign-in. |
| GPU run of one harvest | 60 minutes on llama-8b, and 150 minutes on llama-405b and trinity-truebase | The harvest fails, its error says that it timed out, and it costs nothing. |
wait of a read of a harvest | 0 to 60 seconds | 400 |
Cold starts and long requests explains the 14 minutes, the 45 minutes, the 150 seconds, and the 2 seconds.
Kept data
| Data | Kept | After that |
|---|---|---|
| Result of an ended Job | 24 hours after the Job ended | The result route answers 410 job_result_expired. The Job stays. |
| Activations of a Playground Run | 7 days after the Run ended | The Run shows its text and tokens without them. |
| A deleted Loom, Node, or Playground Run | 14 days after the delete | The API erases it. You cannot read it from the moment you delete it. |
An Idempotency-Key and its answer | 24 hours | A request with the same key runs as a new request. |
| Files of a completed harvest | With no end | The API deletes none of them. |
| A link to a file of a harvest | 7 days from the answer that made it | The link no longer works. Each answer of GET /v1/harvest/{job_id} for a done Job makes new links. |