# Read Activations (/docs/activations)
Read the Activations of a Completion, which are the residual stream of the Base model at the Layers you choose. [The residual stream](/docs/concepts/residual-stream#what-a-read-returns) explains what the numbers are.
Ask for them with the field `output_residual_stream` on `POST /v1/completions`. The field has the name that vllm-lens uses. The `openai` SDK does not know it, so the Python programs send it in `extra_body`. A request with Activations holds and spends tokens as any other Completion does, and the Activations cost no tokens ([Budgets and Holds](/docs/concepts/budgets)).
The Python programs need Python 3.14 (for `compression.zstd`), NumPy, and the `openai` SDK.
## Read the residual stream [#read-the-residual-stream]
List the Layers in `output_residual_stream`. The API sorts them and reads a repeated Layer once. Layer 0 is the first Layer. `llama-8b` has 32 Layers, from 0 to 31, as `capabilities.n_layers` in [`GET /v1/models`](/docs/models) says. To read every Layer, send `"output_residual_stream": true`. Each Layer counts against the [size cap](/docs/completions#size-caps) of a capture, so a read of every Layer takes a short prompt.
This request reads Layers 0 and 2:
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "output_residual_stream": [0, 2]}'
```
```python tab="Python"
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={"output_residual_stream": [0, 2]},
)
residual_stream = decode(completion.activations["residual_stream"])
print(residual_stream.shape)
print(residual_stream[:, 0, :4].tolist())
```
The curl command prints the Completion, with an `activations` field after `usage`:
```json title="Completion"
{
"id": "cmpl-998dce4da6c3c3b2",
"object": "text_completion",
"created": 1791780538,
"model": "meta-llama/Llama-3.1-8B",
"choices": [
{
"index": 0,
"text": " burning\n",
"logprobs": null,
"finish_reason": "length",
"prompt_logprobs": null
}
],
"usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 },
"activations": {
"residual_stream": {
"data": "KLUv/aAAQAEArf4HDgDU8z9FEBCrlaQDMmACBRI4cdIDCRRI...",
"dtype": "int16",
"original_dtype": "torch.bfloat16",
"shape": [2, 5, 4096],
"compression": "zstd"
}
}
}
```
The Python program sends its own request. It decodes the residual stream, and prints its shape and the first 4 numbers of the first token at each Layer:
```text title="Python output"
(2, 5, 4096)
[[0.004058837890625, -0.0002899169921875, -1.33514404296875e-05, -0.002349853515625], [0.0087890625, 0.01513671875, -0.07421875, -0.0537109375]]
```
The shape is 2 Layers, then 5 tokens, then the hidden size of 4096. The 5 tokens are the 4 prompt tokens, with the BOS token first, and the first of the 2 generated tokens. A read never has a row for the last generated token, as [What a read returns](/docs/concepts/residual-stream#what-a-read-returns) explains.
## Decode the tensor [#decode-the-tensor]
`activations.residual_stream` is an array in vllm-lens's encoding:
| Field | What it holds |
| --- | --- |
| `data` | The array's bytes in C order, compressed with zstd, in base64. |
| `dtype` | The NumPy type of the bytes. |
| `original_dtype` | The model's type, for example `torch.float32` or `torch.bfloat16`. |
| `shape` | `[layers, tokens, hidden size]`. |
| `compression` | Always `zstd`. |
The Layers come in ascending order, each one once. The tokens are the prompt tokens and the generated tokens, except the last generated token. So the array has `usage.prompt_tokens + usage.completion_tokens - 1` tokens.
The API checks that shape before it answers. When Inference's answer has no `activations`, or another shape, the API answers 502 `upstream_server_error` with the message `Inference's answer does not fit the request`, and the request costs nothing. In a stream, the error comes as the last event, and you pay for the tokens that came before it.
Every Python program in the guides decodes the array with this `decode` helper. Copy it as it is:
```python
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
```
`llama-8b` computes in bfloat16, and NumPy has no bfloat16 type. So its numbers come as `"dtype": "int16"` with `"original_dtype": "torch.bfloat16"`, and each int16 value holds the bits of one bfloat16 number. A bfloat16 number is the upper 16 bits of a float32 number, so `decode` shifts the bits into a float32 array. With PyTorch, use `torch.from_numpy(array).view(torch.bfloat16)` on the int16 array instead. Do not use `array.astype(np.float32)` on the int16 array, because it converts the integers and not the bits.
A Completion without `output_residual_stream` has no `activations` field.
## Check the tensor before you use it [#check-the-tensor-before-you-use-it]
Your code assumes which Layers and tokens each row holds. Check the shape and the numbers before you use them, so that a wrong assumption fails at once. For example, a prompt can have one more token than you counted, such as the BOS token that the model adds.
1. Decode the array to float32 with `decode`.
2. Check that the first axis has one row for each Layer that you asked for.
3. Check that the second axis has `usage.prompt_tokens + usage.completion_tokens - 1` tokens.
4. Check that every number is finite.
This program does the four steps for the request above:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={"output_residual_stream": [0, 2]},
)
residual_stream = decode(completion.activations["residual_stream"])
layers, tokens, hidden_size = residual_stream.shape
usage = completion.usage
assert layers == 2, residual_stream.shape
assert tokens == usage.prompt_tokens + usage.completion_tokens - 1, residual_stream.shape
assert np.isfinite(residual_stream).all(), "a number is not finite"
print(layers, tokens, hidden_size)
```
```text title="Output"
2 5 4096
```
The last axis is the model's hidden size. Each row of a [Steering vector](/docs/steering) needs that many numbers.
## Read Activations for text you already have [#read-activations-for-text-you-already-have]
To read the Activations of a text, send it as the prompt with `max_tokens: 1`. The one generated token gets no row, so the array has one row for each prompt token.
The text of a Completion can split into other tokens when you send it back as text. If you have its token ids, send them as the prompt instead. Inference adds special tokens, such as BOS, only to a text prompt. So start the list of ids with the BOS id, 128000 for `meta-llama/Llama-3.1-8B`, as the model does for a text.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": [128000, 791, 13180, 374], "max_tokens": 1, "output_residual_stream": [1]}'
```
This array has 4 tokens, one for each id.
## Read Activations for many prompts [#read-activations-for-many-prompts]
A request with `output_residual_stream` takes one prompt and `n: 1`. To read the Activations of many prompts, choose one of two ways:
- If you want the Activations of each prompt's tokens as files, for example to train a probe, [harvest them](/docs/harvest). A harvest reads thousands of prompts on GPUs of its own and writes safetensors files. It generates no text.
- If you want each prompt's Completion too, or a Steering vector, submit a Job for each prompt. The result of each Job is then a Completion with its own `activations`. [Ask for Activations in a Job](/docs/jobs#ask-for-activations-in-a-job) shows such a Job.
## Stream with Activations [#stream-with-activations]
With `stream: true`, the Activations come in one last chunk before `[DONE]`. That chunk has the same `id`, `object`, `created`, and `model` as the others, `"choices": []`, and `activations`.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "output_residual_stream": [1]}'
```
```text title="Events"
data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" the","logprobs":null,"finish_reason":null}]}
data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" limit","logprobs":null,"finish_reason":"length"}]}
data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[],"activations":{"residual_stream":{"data":"KLUv/WAAn/X+Aw4ACvYfQhAQ6jWkA8YkXqo52Z6TdczcoPxB...","dtype":"int16","original_dtype":"torch.bfloat16","shape":[1,5,4096],"compression":"zstd"}}}
data: [DONE]
```
## Requests that the API refuses [#requests-that-the-api-refuses]
The API answers 400 `invalid_request_error` for these requests before it holds any tokens, and they do not reach Inference. `param` is `output_residual_stream`.
- An empty list, or a negative Layer.
- A Layer that the model does not have. Layer 32 of `meta-llama/Llama-3.1-8B` gets `meta-llama/Llama-3.1-8B has Layers 0 to 31: it has no Layer 32.`
- `n` other than 1, or several prompts.
- A capture over its [size cap](/docs/completions#size-caps): Layers × tokens × the model's hidden size. The tokens are the prompt's, as the Hold counts them, and `max_tokens`. On `meta-llama/Llama-3.1-8B`, Layers × tokens is at most 8,192, for example all 32 Layers for 256 tokens.
- A capture of `llama-8b` or `llama-405b` without `max_tokens`, or of more than one Layer of `trinity-truebase` without it. Left out, `max_tokens` is the room left in the context after the prompt. That is up to 15,600 without a stream and 32,000 with one, more tokens than the capture's cap allows. Set `max_tokens` with every capture.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "n": 2, "output_residual_stream": [0]}'
```
```json title="Error"
{
"error": {
"message": "Set n to 1: activations and steering vectors are for one choice only.",
"type": "invalid_request_error",
"param": "output_residual_stream",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
The API checks the Layers against the facts that [`GET /v1/models`](/docs/models) lists. For a served model without those facts, Inference checks the Layers, and the API passes its 400 on.
## Next steps [#next-steps]
[Steer a Completion](/docs/steering) adds a vector to the residual stream and reads the change. [Steer with a contrast vector](/docs/cookbook/contrast-steering) builds a Steering vector from the Activations of two sets of prompts.
# Use API keys (/docs/api-keys)
Every `/v1` route of the API needs an API key. `GET /health` and the spec at `GET /openapi.json` do not. Make a key in the App, then send it in the `Authorization` header of each request. You make keys after you [sign in](/docs/get-access#sign-in).
## Make an API key [#make-an-api-key]
1. In the App, open **API Keys**. Until you have a key, the App opens **API Keys** first after sign-in.
2. Select **New key**. Until you have a key, the button is under **No API keys yet**.
3. Enter a **Name** of up to 32 characters, such as `laptop`. No two of your keys can share a name, whatever the case of its letters. A revoked key's name is free again.
4. Select **Create key**.
5. In **Copy your key**, select **Copy**. The App shows the key only this once, because the API keeps only a hash of it. Every key starts with `sk-acs-`.
6. Select **Done**. If you did not copy the key, the App asks first. It asks too before you close or reload the tab.
7. Put the key and the API's address in shell variables. The examples in these docs read both:
```bash
export ACS_API_KEY=sk-acs-...
export ACS_API_BASE=https://api.v2.infra.acsresearch.org/v1
```
If the browser does not let the App copy the key, the App shows it selected, for you to copy by hand. **Set the client for long requests** in **Copy your key** opens [Set the client for long requests](/docs/retries#set-the-client-for-long-requests) in a new tab.
## Send the key [#send-the-key]
Send the key as a Bearer key in the `Authorization` header. With curl, add the header to each command:
```bash
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Models"
{
"object": "list",
"data": [
{
"id": "llama-8b",
"object": "model",
"owned_by": "acs",
"served_model_name": "meta-llama/Llama-3.1-8B",
"gpu_shape": "1xL40S",
"status": "live",
"default": true,
"capabilities": {
"max_model_len": 131072,
"max_logprobs": 100,
"max_output_work_tokens": 32000,
"max_inflight_per_key": 16,
"max_queued_per_key": 64,
"rate_limit_per_minute": 600,
"logprobs": true,
"prompt_logprobs": true,
"prompt_logprobs_full_vocab": true,
"full_vocab_max_prompt_tokens": 256,
"activations": true,
"max_steering_vectors": 8,
"max_activation_prompt_tokens": 256,
"n_layers": 32,
"hidden_size": 4096,
"chat_template": null
}
}
]
}
```
With the `openai` Python SDK, give the key as `api_key`. If you leave out `api_key`, the SDK reads the key from the environment variable `OPENAI_API_KEY`:
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(model="llama-8b", prompt="The sky is", max_tokens=2)
print(completion.usage.total_tokens)
```
```text title="Output"
6
```
### Fix a 401 error [#fix-a-401-error]
The API checks the key before it reads the request body. A request without a key gets 401 with the code `missing_authorization`:
```bash
curl "$ACS_API_BASE/models"
```
```json title="Error"
{
"error": {
"message": "You didn't provide an API key. You need to provide your API key in an Authorization header using Bearer auth (i.e. Authorization: Bearer YOUR_KEY).",
"type": "invalid_request_error",
"param": null,
"code": "missing_authorization"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
The `code` and the `message` say what is wrong with the key:
| `code` | `message` | What to do |
| --- | --- | --- |
| `missing_authorization` | `You didn't provide an API key. ...` | Send the `Authorization` header. |
| `invalid_api_key` | `Incorrect API key provided.` | Check the key. It starts with `sk-acs-`, or with `acs-bm-` for a key of the old platform. |
| `key_revoked` | `This API key has been revoked.` | Make a new key. |
| `key_disabled` | `This API key is paused. Resume it on API Keys in the App to use it again.` | Resume the key, as [Manage your keys](#manage-your-keys) shows. |
| `account_suspended` | `Your access to ACS Infra is suspended. Write to infra@acsresearch.org if you think this is a mistake.` | Read [If your access is suspended](/docs/get-access#if-your-access-is-suspended). |
When many requests from one address send a key that does not work, the API answers 429 `rate_limited` instead, with `Retry-After`.
## Call the API from a browser [#call-the-api-from-a-browser]
A page in a browser can call the API, for example with the `openai` JS SDK and its option `dangerouslyAllowBrowser: true`. The API answers every origin with CORS on `/v1`, `/health`, and `/openapi.json`. The page can read each answer, errors too, with the headers `X-Request-Id`, `Retry-After`, `retry-after-ms`, `X-Should-Retry`, and `X-Acs-Max-Tokens-Clamped`.
The API reads the API key and no cookie, so the page needs your key. Anyone
who opens the page can read the key and spend your tokens. Make a key for that
page alone, use the page only yourself, and revoke the key when you are done.
## Manage your keys [#manage-your-keys]
The table on **API Keys** lists your active and paused keys. When you have none, the page shows **No API keys yet** and **New key** in its middle, with no table. It shows the same after you revoke your last key. The table shows each key's **Name** and first characters, its **Status**, and its tokens **This month**. A name that does not fit on one line is cut short. The table also shows the tokens left in the key's own **Budget**, and when the key was **Last used** and **Created**. A time today shows in UTC, such as `05:02 UTC`, and an earlier one shows its date. On a phone, the table leaves out **Budget**, **Last used**, and **Created**.
The page reads the numbers again every minute while its tab is open. When a read fails, the page keeps the numbers that it read last, and shows the note `Could not refresh.` below the table. When you have no tokens left, the notice under the title says so, with **Request more** ([Ask for more tokens](/docs/app#ask-for-more-tokens)).
Each row ends with a menu of the key's actions.
1. To rename a key, select **Rename** in its menu, enter the new **Name**, and select **Rename**. The same name rule holds as for a new key.
2. To pause a key, select **Pause** in its menu. Requests with the key get 401 with the paused message. Jobs and Loom Branches that the key started wait, and run when you select **Resume** ([Paused keys and suspended access](/docs/concepts/background-work#paused-keys-and-suspended-access)).
3. To read what a key used, select its tokens **This month**, or **View usage** in its menu. **Usage** opens for that key alone ([Read your usage](/docs/app#read-your-usage)).
4. To stop a key for good, revoke it, as [Revoke a key](#revoke-a-key) shows.
Make one key for each script or machine. You can then pause or revoke one key without stopping the others, and the **Usage** page shows the tokens of each key ([Read your usage](/docs/app#read-your-usage)).
An admin can give a key its own Budget with a Grant for that key. That key then stops when either its own Budget or yours is empty ([Your Budget](/docs/concepts/budgets#your-budget)). A key without its own Budget shows a dash in **Budget**.
You can have at most 10 keys that are not revoked ([API keys in Limits](/docs/limits#api-keys)). At the limit, **New key** is off, and a note below the table says why: `You have 10 API keys, the most that a User may have. Delete one to make another.` Revoke a key to make room for a new one.
## Revoke a key [#revoke-a-key]
1. On **API Keys**, open the menu at the end of the key's row, and select **Revoke**.
2. Select **Revoke key** to confirm, or **Cancel**.
Requests with the key get 401 with the message `This API key has been revoked.` at once. Requests of Jobs and Loom Branches that the key started and that have not run yet fail, and spend no tokens. You cannot undo a revoke. The key leaves **API Keys**, which lists only the keys that you can use. **By key** on **Usage** still shows the tokens that the key used, under its name, with the status Revoked ([Read your usage](/docs/app#read-your-usage)).
# Use the App (/docs/app)
The App is where you make API keys, run prompts in the browser, grow Looms, and read your usage. It uses the same API and the same Budget as your scripts. [Sign in](/docs/get-access#sign-in) to open it.
## The App's pages [#the-apps-pages]
The App's navigation lists these pages:
| Page | What you do there |
| --- | --- |
| **Playground** | Run a prompt with every setting of the API, compare models side by side, and read each token's probability ([Use the Playground](/docs/app/playground)). |
| **Loom** | Grow a tree of continuations from a prompt, one Branch at a time ([Use the Loom page](/docs/app/loom)). |
| **Usage** | Read the tokens that you used and have left, and ask for more ([Read your usage](#read-your-usage), [Ask for more tokens](#ask-for-more-tokens)). |
| **Jobs** | Find the way to run a Job. The page has no list of Jobs: **Run a Job from a script** opens [Run a Job from a shell script](/docs/cookbook/job-script). |
| **API Keys** | Make, rename, pause, and revoke API keys ([Use API keys](/docs/api-keys)). |
| **Profile**, in the menu with your name | Read your name and email, reset your password, and link a sign-in provider ([Get access](/docs/get-access)). |
| **Admin**, for admins only | Decide Applications and token requests, make Invites, and manage Users and their Budgets ([Manage Users](#manage-users), [Decide token requests](#decide-token-requests)). Read the platform's numbers ([Read the platform's analytics](#read-the-platforms-analytics)). Keep models Warm at set hours ([Keep a model Warm](#keep-a-model-warm)). |
**Docs** links to these docs. **Support** has **Email**, **Discord**, **Feedback**, and **Help**. **Discord** opens the invite to ACS's Discord server ([Discord](/docs/support#discord)). **Feedback** opens **Send feedback** ([Send feedback](#send-feedback)). **Help** opens [Support](/docs/support) in these docs. **Sign out** is in the menu with your name.
## Read your usage [#read-your-usage]
1. Open **Usage**. It counts this month, up to today, for all your keys and models.
2. To count other days, select the days at the top of the page. Choose **Today**, **Last 7 days**, **Last 30 days**, **This month**, or **Last month**. Or select a first and a last day in the calendar, then **Apply**.
3. To count one key or one model, choose it in **All keys** or **All models**.
The page counts in UTC days, at most 366 at a time. The address keeps the days, the key, and the model, so a reload or a shared link opens the same view. The page reads the numbers again every minute while its tab is open. **Updated** beside the title says when it last read them, in UTC. Select **Refresh** beside it to read them at once. When a read fails, the page keeps the numbers that it read last and says `Could not refresh`.
The page has these parts, from the top:
| Part | What it shows |
| --- | --- |
| **Tokens used** | The input and output tokens of the Completions that the page counts. |
| **Completions** | How many Completions it counts, and how many of them still run. |
| **Monthly allowance** | The tokens left this month, of how many, and the day the allowance resets. Its meter turns orange from 90% used, and red once the allowance is spent. |
| **Token budget** | The tokens left in your Budget, and **Request more tokens** ([Ask for more tokens](#ask-for-more-tokens)). |
| **Tokens per day** | The input and output tokens of each day, as a chart. Point at a day, or select the chart and use the arrow keys, to read its numbers. **View as table** lists each day that had a Completion. |
| **By model** and **By source** | The tokens of each model, and of each source: the API, the Playground, the Loom, Jobs, or Harvests. **By source** also counts each source's Completions and its input and output tokens. |
| **By key** | Each key that ran Completions on those days, with its status and its input and output tokens. It also shows the key's tokens last month and in all, its own Budget, and when it last ran one. Select a key's name to count that key alone. |
| **Completions** | The newest 100 of those Completions. Each row has when it started (UTC), its model, source, key, input and output tokens, how they were counted, how long it took, and status. |
| **Budget history** | The newest 100 changes to your Budget or a key's, such as a Grant, newest first. |
A Completion's status is **Running**, **Completed**, **Failed**, or **Stopped**. **Failed** means that Inference failed it after its first token. **Stopped** means that its client went away, or that you stopped its Job, Branch, or Run, before Inference finished it. Each costs the tokens that it used ([How a Hold ends](/docs/concepts/budgets#how-a-hold-ends)).
**Counted** says where a Completion's tokens came from. **Reported** tokens are the ones that Inference reported. **Partial** tokens are the ones that a stream sent before it stopped. **Held** is the whole Hold, for a request without a stream that stopped after it reached Inference. **Duration** runs from the start of the Completion to its end, and includes the wait for the model to start.
The monthly allowance and the token budget are for all your keys and models. With one key or one model chosen, the page calls them "Your account's monthly allowance" and "Your account's token budget". When the chosen key has its own Budget, **This key's token budget** shows the tokens left in it first. The tokens left already leave out what your running Completions hold.
A released Completion costs nothing, and the page does not list it, for example one that Inference refused before it sent a token. [Budgets and Holds](/docs/concepts/budgets) explains what a Completion holds and pays.
## Ask for more tokens [#ask-for-more-tokens]
1. Open [Request more tokens](https://v2.infra.acsresearch.org/usage/request). **Request more tokens** under **Token budget** on **Usage** opens it, and so does **Request more** in the notice under a page's title.
2. Fill in **Tokens**, **Models**, **Project**, and **Planned usage**. Write **Tokens** as a number, such as `50K` or `2.5M`, and choose each model that the work needs.
3. Answer the optional questions if you can: **How you got the number**, **Results**, **Links**, and **When you need them**. An admin decides from your answers.
4. Select **Send request**. The App says "Request sent."
If a required question has no answer, or the App cannot read **Tokens**, the App says what to fix under each such question. It keeps what you wrote.
An admin approves the request with a number of tokens, which may differ from the number that you asked for, or rejects it. An approval is a Grant to your Budget. **Your requests** lists each request with the day that you sent it, its tokens, and its status: **Waiting**, **Approved**, or **Rejected**. A rejected request also shows the admin's note. An approval for another amount shows both numbers, such as `50K of 100M`. On a phone, the note is under the status. You have one request waiting at a time. Once an admin decides it, you can ask again.
From 90% of your tokens used, a notice under the title of each page says how many are left, with **Request more**. It counts the tokens that you used this month against those tokens and what your monthly allowance and Budget have left, as **Usage** shows them. With no tokens left, it says `You have no tokens left.` Its close button hides it until you close the browser, or until it says that no tokens are left.
## Read the platform's analytics [#read-the-platforms-analytics]
An admin's **Admin** page has an **Analytics** tab. It counts the Users who arrive and who run Completions, and their monthly allowances. It also shows how the API served their requests, their tokens, and what the GPUs cost.
1. Open **Admin**, then **Analytics**. It counts this month, up to today.
2. To count other days, select the days at the top of the tab, as on **Usage**. Each tab of **Analytics** counts these days.
3. Choose a tab: **Users**, **Budget**, **Platform**, **Usage**, or **Spend**. The address keeps the tab and the days.
**Analytics** has these tabs:
| Tab | What it shows |
| --- | --- |
| **Users** | The Applications that wait for a decision, and the new Users. How many new Users ran a Completion within 24 hours of getting access. The Users active in the last 7 days, and the Feedback of each day. |
| **Budget** | The Users at or close to their monthly allowance, and the requests that the Budget could not hold. The 10 Users closest to their allowance, with the day it runs out at their pace. |
| **Platform** | The requests of each day by how they ended, and the server errors and the timeouts. The requests and tokens of each source, the Users of each hour in five time zones, and the requests of each model. |
| **Usage** | The tokens of each User, model, and source, the newest Completions, and the newest Budget changes. To count one User or one model, choose it in **All Users** or **All models**. |
| **Spend** | What the GPUs cost, and the Spend stop ([Watch the spend](#watch-the-spend)). |
**Definitions**, at the bottom of **Users**, **Budget**, and **Platform**, says how each number is counted. **Daily numbers** lists the numbers of each day. The tab leaves the team's accounts out of the numbers about Users. **Platform** also shows the GPU spend of the days and of each model, as **Spend** counts it.
## Watch the spend [#watch-the-spend]
**Spend**, a tab of **Analytics** on **Admin**, shows what the GPUs cost. The API estimates it from Inference's count of its containers each minute. It is not Modal's bill.
| Part | What it shows |
| --- | --- |
| The top | The GPU spend of the last 24 hours, with the amounts that warn the team and that pause new GPU work. The spend that the pause counts, since its last **Clear**. When Inference last counted. |
| **Spend stop** | Whether new GPU work is paused, and its last change. **Lock** pauses it, with a reason if you want one, and **Clear** lifts it ([Spend limit](/docs/limits#spend-limit)). |
| **Last 24 hours by model** | The spend and GPU hours of each model in each Modal environment. |
| **GPU spend per day** | Each day's spend of the models that cost the most. Choose the days at the top of **Analytics**. |
| **Who used each model, last 48 hours** | Each User's Completions and tokens of each model, and their part of its spend, by their share of its tokens. |
| **Changes of the stop** | The newest 10 changes, each a **Lock**, a **Clear**, or a pause at the stop amount, with who made it. |
## Keep a model Warm [#keep-a-model-warm]
A start of a cold Llama 3.1 405B or Trinity Large takes 17 to 25 minutes, longer than [the API's wait](/docs/concepts/long-requests#the-14-minute-wait). An admin keeps a model Warm at set hours on the **Models** tab of **Admin**. Each Warm hour costs the model's GPUs. That is about $36.32 for Llama 3.1 405B or Trinity Large, and $1.95 for Llama 3.1 8B.
1. Open **Admin**, then **Models**. Each model shows its **Floor**, the containers that Inference keeps running without requests, and what its windows cost in the **Next 7 days**.
2. To keep a model Warm each week, open its actions, then **Add a window**. Choose its days, **From**, **To**, and its **Time zone**, which is UTC at first. A window that ends at or before its start ends the next day. The dialog shows what the window costs a week.
3. To keep a model Warm at once, open its actions, then **Warm now**, and enter its **Hours**, from 1 to 24. The dialog shows what it costs, and it ends by itself.
4. To stop a window, open the menu at the end of its row under **Warm windows**, select **Remove**, then confirm. The confirmation shows what it costs.
Start a window 30 to 45 minutes before Users need the model, so that the model is ready in time. While the API pauses new GPU work, no window keeps a model Warm ([Spend limit](/docs/limits#spend-limit)).
## Manage Users [#manage-users]
An admin's **Admin** page has a **Users** tab. It lists each User with their organization, role, standing, tags, and engagement. It also shows when they last ran a Completion, their active weeks, and their tokens of the last 30 days. Then come the models they used, most tokens first, their Budget, monthly allowance, API keys, and the day they joined.
**Engagement** says when a User last ran a Completion:
| Engagement | The User's newest Completion |
| ----------------- | ---------------------------- |
| **Active** | In the last 7 days. |
| **Cooling** | 8 to 14 days ago. |
| **Churned** | More than 14 days ago. |
| **Not activated** | None. |
An active week is a UTC week, Monday to Sunday, in which the User ran a Completion. A Completion that Inference refused before its first token counts for none of these.
### Open a User's page [#open-a-users-page]
Select a User's email on **Users** to open their page. It has their account, their details, their tags, their Budget and usage, their API keys, their Application, and their notes. In **Details**, type their **Organization** and their **Discord handle**, and select **Save details**. The organization that you type replaces their Application's answer on the Admin page. Clear it, and the Application's answer shows again. The **Budget and usage** part counts as **Usage** does, for the days and the model that you choose. The menu beside the email has the same actions as the User's row. **Copy User id** copies the id that the API's logs and PostHog name the User by.
### Act on many Users at once [#act-on-many-users-at-once]
1. On **Users**, select the Users with the box in front of each row. The box in the header selects every row that the filters show, on every page.
2. Choose an action in **Selected Users**: **Suspend**, **Restore**, **Add tag**, **Remove tag**, **Set allowance**, or **Delete**.
3. Read the dialog, then confirm. To delete, type the words that the dialog shows, such as `delete 3 Users`.
An action changes only the selected Users that the filters show. It leaves out the Users that it does not suit, and the dialog says how many. **Suspend** and **Delete** leave admins out: change an admin from their own row. You cannot select your own row or a deleted User. When the API refuses some Users, the dialog lists each one with the reason, and the others are changed.
### Group Users with tags [#group-users-with-tags]
An admin can put a tag on Users, such as `hiring-2026-08`, to find a group of them again. A tag has lowercase letters, digits, and dashes, and at most 32 characters. The App writes what you type in that form, so `Hiring 2026 08` becomes `hiring-2026-08`.
- To list the Users of one tag, choose it in **Tag** on **Users**.
- To give each new User of an Invite a tag, write it in **Tag** when you make the Invites. A User gets the tag when they use the link.
## Decide token requests [#decide-token-requests]
An admin's **Admin** page has a **Token requests** tab. It lists each request with the User, the tokens asked, the project, the status, the tokens granted, and who decided it and when. A request sent before the form asked for a project shows **None** as its project. The status is **Waiting**, **Approved**, or **Rejected**. Each new request also sends the team an email with the answers.
1. To read the answers, select **Read the request** in its row's menu. Each answer is under its question, and an empty one says **Not answered**. A request sent before the form had these questions shows its reason under **Planned usage**.
2. To approve it, select **Approve**, set **Tokens to grant**, such as `50K` or `2500000`, and select **Approve**. The field starts at the tokens asked. The amount is a Grant to the User's Budget.
3. To reject it, select **Reject**, write a **Note to them** or leave it empty, and select **Reject**. The User reads the note with their request.
## Send feedback [#send-feedback]
1. Open **Support** in the App's navigation, then select **Feedback**.
2. Under **What is it about?**, choose **Bug**, **Feature request**, or **General**.
3. Write **Your note**, of up to 5,000 characters. The App also sends the address of the page that you are on.
4. To add screenshots, select **Choose screenshots**, drop them on the field, or paste them. Each is a PNG, JPEG, GIF, or WebP image of 2 MB or less, and a note has at most 5.
5. Read the recent errors and request ids that the App sends with your note. To send the note without them, and without your User or your browser's name, select **Send anonymously, without my account or the details below**.
6. Select **Send**. The App says "Thanks, your note reached the team."
The team tracks each note in Linear, its issue tracker. Your note's text, the page, your screenshots, and your account's email go there, so that the team can answer you. An anonymous note goes without your email. The recent errors and the request ids stay on the servers of ACS Infra. Do not paste a prompt or a Completion that you want to keep private.
An admin reads each note on the **Feedback** tab of **Admin**. The **Linear** column links to the note's issue.
To report a bug by email instead, read [Report a bug](/docs/support#report-a-bug).
# Use the Loom page (/docs/app/loom)
A Loom is a tree of continuations. Each Node holds text, and each Branch under a Node grows new Nodes that continue the text of their path. The App's [Loom page](https://v2.infra.acsresearch.org/loom) grows Looms in the browser. To grow one from a script, read [Make a Loom](/docs/loom#make-a-loom).
## Open or start a Loom [#open-or-start-a-loom]
**Loom** opens the Loom that you changed last. **Your Looms** lists your Looms, up to 100: the pinned ones first, then the others by their last change. Select **Your Looms** to fold the list away, and again to open it. The next Loom page shows the list as you left it. To start a new Loom:
1. Select **New Loom**.
2. Write the **Prompt**. In an empty prompt box, Tab puts in the gray example.
3. Select **Start the Loom**. The prompt becomes the first Node, the root.
A new Loom is **Untitled Loom** until its first Branch names it after the first line of its root, up to 40 characters. To rename a Loom, select its title. To manage it in other ways, use the buttons by its title. The menu on the Loom's row in **Your Looms** has **Pin**, **Rename**, and **Delete** too, so you can change a Loom without opening it:
- The title and **Rename** open **Rename this Loom**. Enter a **Title** and select **Save**. An empty title leaves the Loom untitled, so its next Branch names it.
- **Pin** keeps the Loom at the top of **Your Looms**. **Unpin** takes it off.
- **Delete Loom** deletes the Loom, with its prompt and every Node, after you confirm. You cannot undo it.
## Branch under a Node [#branch-under-a-node]
1. Select a Node in the tree. **Text** shows the text of its path: the earlier Nodes in gray, then the Node, with its heatmap.
2. In the settings, choose the model, **Branches** (how many Nodes the Branch grows, from 1 to 8), and how each Node is sampled.
3. Select **Branch**. Each new Node shows its text as it arrives, also after a reload.
4. To stop every Node of the Loom that grows, select **Stop**. Each keeps the text that it grew.
A Loom opens with the settings of its last Branch, with **Seed** blank. A blank **Seed** gives each Branch a new random seed: its first Node gets the seed, the next the seed + 1, and so on. When the last Branch asked for no log probabilities, **Top logprobs shown** starts at 5, for the heatmap.
As in the [Playground](/docs/app/playground#wait-for-a-model-to-start), a Branch of a model that is not Warm waits while the model starts. One line under the title says how far the start has got and about how long is left.
**Branch** works under a Node that completed or that you stopped. A Node that failed shows why, under the text that it had. Select a token in **Text** to see the likeliest tokens at that place. The heatmap marks a line break with ⏎ and a tab with ⇥, and a copy of the text leaves the marks out.
Each Node holds its own tokens while it grows ([Holds of background work](/docs/concepts/budgets#holds-of-background-work)).
### Read the tree as a graph [#read-the-tree-as-a-graph]
**Tree** and **Graph** switch the view of the Nodes, and the next Loom page opens in the same view. **Graph** shows each Node as a dot with its first words, and each Branch as a line from its parent. Scroll to pan, and use **Zoom in**, **Zoom out**, and **Fit**. Hover over a Node for its whole first line, and select it to read it.
## Change the tree [#change-the-tree]
Each change acts on the Node that you read. The note next to the buttons says why a change has to wait, for example while a Branch grows under the Node.
- **Edit** opens **Edit this Node**. Change the **Text** and select **Save**. The Node loses its heatmap, and the Nodes under it keep their own text.
- To split a Node in two, select **Edit**, put the cursor inside the text where the new child starts, and select **Split at the cursor**. A split at the start or the end would leave a Node empty, so the dialog asks you to move the cursor.
- **Sibling** opens **Write a sibling**. Write its **Text** and select **Add sibling**.
- On the root, **New root** opens **Write a new root**. **Replace the tree** replaces the whole tree, and every Node of the Loom goes.
- **Delete Node** deletes the Node and the Nodes under it, after you confirm. You then read its parent.
## Use the keyboard [#use-the-keyboard]
These keys act on the Node that you read, when the focus is not in a text box or a dialog:
| Key | What it does |
| --- | --- |
| `g` | Grow a Branch under the Node |
| `e` | Edit the Node |
| `s` | Split the Node, with **Split at the cursor** in the edit dialog |
| `i` | Write a sibling of the Node |
| `Delete` | Delete the Node and the Nodes under it |
| `↑` `↓` `←` `→`, `Home`, `End` | Move in the tree |
| `Esc` | Close a dialog |
| `?` | Show these keys |
**Keys**, at the end of the Node's buttons, also shows these keys. A key does nothing while its button is off. The root has no `i` and no `Delete`.
## Export and import [#export-and-import]
1. To copy the text of the path to the Node that you read, select **Copy text**. To download it as `loom-path.txt`, select **Path as text**.
2. To download the whole Loom as a Loom file, `loom-.json`, select **Export**.
3. To make a Loom from a Loom file, select **New Loom**. Under **Import a Loom**, choose the **Loom file**, and select **Import**.
An import makes a new Loom with new Node ids, and opens it. Its title is the file's title with ` (imported)` after it, so you can tell it from the Loom that it came from. The App imports files that **Export** wrote, and Loom files from the old platform. For a file that is not a Loom file, the App says what is wrong with it.
When the browser does not let the App copy, the App shows the text selected. Copy it by hand.
# Use the Playground (/docs/app/playground)
The App's [Playground](https://v2.infra.acsresearch.org/playground) runs Completions in the browser, with the settings of `POST /v1/completions`: the model, sampling, log probabilities, Activations, and Steering vectors. Each Completion that you start there is a Run, which the API runs on its server and keeps.
## Run a prompt [#run-a-prompt]
1. Open **Playground**.
2. Write the prompt. In an empty prompt box, Tab puts in the gray example.
3. Choose the **Model** and the settings, such as **Max tokens** and **Temperature**. Each setting is a field of the request, as [Completions](/docs/completions#request-fields) lists them. **Choices** is `n`, **Top logprobs shown** is `logprobs`, **Read Layers** reads Activations, and **Steer** adds a Steering vector. Hover over a model in the model list to see its id, and use that id as `model` in the API.
4. Select **Run**. The text appears under the prompt as it grows, and the prompt stays as you wrote it. To stop the Run, select **Stop**.
To go on from a Run's text, copy it with **Copy** and add it to the prompt.
A number out of range is marked as you type, with its range. For example, **Top p** is more than 0 and at most 1. **Reset to default** puts every setting back.
### Wait for a model to start [#wait-for-a-model-to-start]
A new Playground starts on a Warm model when one is Warm. A Run of a model that is not Warm waits while the model starts. One line above the output says how far the start has got and about how long is left. The Run then goes on by itself, also when you close the tab. [Cold starts and long requests](/docs/concepts/long-requests) explains the starts.
If the API refuses to start a Run, for example because your Budget cannot hold it, the output says why, and nothing runs. When a setting is at fault, such as a Layer that the model does not have, the App marks that setting until you change it.
### Share your settings [#share-your-settings]
The page's address holds the settings of each model, **Single** or **Compare**, and the **Heatmap**. Copy the address to share them, and a reload keeps them. The address never holds the prompt, a Run, or a Steering vector.
The browser keeps the prompt as a draft. **Playground** in the navigation opens the address that this browser used last, so your settings come back after you visit another page. When you sign out, the App forgets the draft and that address.
## Compare models and settings [#compare-models-and-settings]
1. Select **Compare**. The page lists the **Models**. When you had one model, Compare starts with a copy of it.
2. Select **Add model** to add a model, up to 6. Each model has its own settings, and all of them share the prompt. Each model has buttons to change its settings, duplicate it, or remove it.
3. Select **Run all**. The Playground starts one Run for each model. To stop them, select **Stop all**.
Each Run shows its model, and the settings in which it differs from the others. Runs with the same prompt and with log probabilities also say how many of their first tokens match the first Run's. A model's Run can fail to start, for example when your Budget cannot hold it. That model's place then says why, beside the Runs that started.
## Find your Runs again [#find-your-runs-again]
The API runs each Run on its server and keeps it until you delete it. A Run goes on when you close the tab or open another page, and the Playground follows it again when you come back. [When the API restarts](/docs/concepts/background-work#when-the-api-restarts) says what happens to a Run that is running then.
Each Run has a number: 1, 2, 3, and on, in the order that you started your Runs. A Run keeps its number. When you come back to the page, the newest Run shows above the **Runs** table.
The **Runs** table lists your newest 100 Runs, newest first, with their number, status, model, settings, text, tokens, and time. When the 100 cut the Runs of one **Run all**, the table lists the rest of them too.
1. To read a Run again, select it in **Runs**. The Playground shows it with its tokens and Activations.
2. To compare Runs, choose two or more with their boxes. The button **Compare runs** then counts them, as in **Compare 2 runs**. Select it to see them side by side.
3. To delete a Run, open the menu at the end of its row, select **Delete**, then **Delete run**. Its text, tokens, and Activations go, and you cannot undo it.
Each row of **Runs** ends with a menu of the Run's actions:
| Action | What it does |
| --- | --- |
| **Show** | Shows the Run, as a select on its row does |
| **Use these settings** | Puts the Run's model, settings, and prompt back into the form |
| **Copy settings** | Copies the model, `max_tokens`, and each setting that differs from the API's default as JSON, with the field names of `POST /v1/completions` |
| **Copy prompt** | Copies the prompt |
| **Copy output** | Copies the text of the Run, each choice under its number when it has several. A Run without text does not have it. |
| **Copy as API request** | Copies a curl command that sends the Run's request to `POST /v1/completions`, without the fields at the API's defaults |
| **Delete** | Deletes the Run after you confirm |
The curl command reads your API key from `$ACS_API_KEY` and holds no key ([Send the key](/docs/api-keys#send-the-key)). Its request streams, as the Run did, so curl prints the events as they arrive. When the browser does not let the App copy, the App shows the text selected. Copy it by hand.
The Runs of one **Run all** stay together in **Runs**, under a row that names them, such as **Comparison of runs 4 and 5**. To bring a comparison back, select **Restore** on that row. The Playground switches to **Compare** and makes a model of each Run's settings, with its seed. It puts the prompt back in the box, and shows the Runs side by side. Nothing runs until you select **Run all**. **Restore** waits until no Run of the page is running.
The API erases a deleted Run 14 days later. It keeps a Run's Activations for 7 days after the Run ends. After that, the Run shows its text and tokens without them. [Kept data](/docs/limits#kept-data) lists how long the API keeps each thing.
A Run that you stop keeps the text that it wrote, and you pay the tokens that it used ([How a Hold ends](/docs/concepts/budgets#how-a-hold-ends)).
## Read tokens and the heatmap [#read-tokens-and-the-heatmap]
When a Run asks for log probabilities (**Top logprobs shown** above 0), each token of its text is a button.
1. Select a token, or move to it with the arrow keys and press Enter. The Playground shows the likeliest tokens at that place, with their probabilities.
2. To try another token there, select it. The Playground cuts the text at that place, puts the token in, and runs on from it in a new Run.
The **Heatmap** colors each token by its probability, in five bands of one color, from "90% or more" to "Under 1%". The darker the token, the more likely it was. The heatmap marks a line break with ⏎ and a tab with ⇥. When you copy the text, the marks stay out of it. Turn the **Heatmap** on or off in the settings.
**Copy** copies **The text** of a Run, or **The prompt and the text**. When the browser does not let the App copy, as on a page over plain HTTP, the App shows the text selected. Copy it by hand.
**Use these settings** puts a Run's settings and its prompt back into the form. Next to it, the Run says how the settings on screen differ from its own. A Run from before a range narrowed keeps its value, such as a **Temperature** of 5, and the form marks it.
## Export Runs [#export-runs]
1. To export the Runs in view, select **Export** at the head of the output. To export the Runs that you chose in **Runs**, select **Export chosen**. To export every Run that you kept, also those older than the newest 100, select **Export every run**. Its file has each Run's text and log probabilities, oldest first, without the Activations.
2. Choose a file:
| File | What it holds |
| --- | --- |
| **JSON** | One object for each Run, with its request, its Completion, its usage, and its status |
| **JSONL** | One line for each choice, for `pandas.read_json(path, lines=True)` |
| **Text** | The text of each choice, with its prompt, to read |
The request and the Completion keep the API's own shapes, so code that reads the answers of `POST /v1/completions` reads them too. The Activations keep the encoding that [Decode the tensor](/docs/activations#decode-the-tensor) reads.
# Cheatsheet (/docs/cheatsheet)
Each section gives the smallest request for one task, and links to the page that explains it. Each curl command reads the API's address from `ACS_API_BASE` and your key from `ACS_API_KEY`. The commands that also need `JOB_ID`, `LOOM_ID`, or `NODE_ID` say where the value comes from.
## Send a first Completion [#send-a-first-completion]
The continuation comes back in `choices[0].text`. The [Quickstart](/docs/quickstart#send-a-prompt) shows the response.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'
```
## Set the sampling fields [#set-the-sampling-fields]
A field that you leave out gets OpenAI's default, such as `temperature` 1. The same request with the same `seed` samples the same tokens, and vLLM's batching can still change one now and then. `stop` ends the text where one of its strings starts. [Set the sampling fields](/docs/sampling#set-the-sampling-fields) shows each field.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0.7, "top_p": 0.9, "seed": 7, "stop": ["\n"]}'
```
## Sample many continuations [#sample-many-continuations]
A list of prompts and `n` return one choice for each prompt and each sample. Choice `i` of prompt `p` has the index `p * n + i`. [Send several prompts](/docs/sampling#send-several-prompts) shows more.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}'
```
## Read log probabilities [#read-log-probabilities]
`logprobs: 5` returns the log probability of each generated token, and of the 5 likeliest tokens at its position. [Get log probabilities for generated tokens](/docs/token-probabilities#get-log-probabilities-for-generated-tokens) explains the lists.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 5}'
```
## Read the prompt's log probabilities [#read-the-prompts-log-probabilities]
`prompt_logprobs: 5` returns the log probability of each prompt token, and of the 5 likeliest tokens at its position. `-1` returns the whole vocabulary, for at most 256 prompt tokens in all × `n`. The `openai` SDK sends this field only in `extra_body`. [Get log probabilities for the prompt](/docs/token-probabilities#get-log-probabilities-for-the-prompt) shows both.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 1, "prompt_logprobs": 5}'
```
## Echo the prompt [#echo-the-prompt]
`echo: true` puts the prompt in front of the generated text, with no separator. [Return the prompt with the Completion](/docs/token-probabilities#return-the-prompt-with-the-completion) shows it.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "echo": true}'
```
## Stream a Completion [#stream-a-completion]
`stream: true` sends Server-Sent Events that end with `data: [DONE]`. `include_usage` adds a last chunk with the token counts. [Stream a Completion](/docs/sampling#stream-a-completion) shows the events.
```bash
curl -N "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "stream_options": {"include_usage": true}}'
```
## Read Activations [#read-activations]
`output_residual_stream` returns the residual stream at the Layers in the list, in `activations.residual_stream`. [Decode the tensor](/docs/activations#decode-the-tensor) shows how to read it.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "output_residual_stream": [0, 2]}'
```
## Steer a Completion [#steer-a-completion]
`apply_steering_vectors` adds each vector to the residual stream at the Layers in its `layer_indices`. A vector has one number for each dimension of the model's hidden size, 4096 for `meta-llama/Llama-3.1-8B`. This example is a Python program, because it builds the vector with NumPy. [Steer a Completion](/docs/steering) explains the fields.
```python
import os
import numpy as np
from openai import OpenAI
vector = np.zeros(4096)
vector[0] = 0.5
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={"apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2.0}]},
)
print(repr(completion.choices[0].text))
```
## Run a Job [#run-a-job]
A Job runs one Completion request on the server. The first command submits a Job with an `Idempotency-Key`. The other two need `JOB_ID`, the `id` in the first command's answer. The second reads the Job's `status`. The third fetches the result once the Job has ended, with the status that the Completion ended with. [Run a Job](/docs/jobs) explains each step.
```bash
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: sky-001" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'
```
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY"
```
## Harvest Activations [#harvest-activations]
A harvest reads the residual stream of many prompts at the Layers that you choose, and writes it to files. The first command submits a harvest. The other two need `JOB_ID`, the `job_id` in the first command's answer. The second waits up to 60 seconds for the harvest's `status` to change. Once the harvest is `done`, its `result` has links to the files. The third cancels a harvest that is `pending` or `running`. [Harvest Activations](/docs/harvest) explains each step.
```bash
curl "$ACS_API_BASE/harvest" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}'
```
```bash
curl "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```bash
curl -X DELETE "$ACS_API_BASE/harvest/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
## Grow a Loom [#grow-a-loom]
A Loom is a tree of Nodes that grows from a prompt. The first command makes a Loom. The other two need `LOOM_ID`, the Loom's `id` in its answer. The second also needs `NODE_ID`, the `id` of the root Node in `nodes[0]`. It asks for a Branch of 2 Nodes under the root, and the third reads the Loom with the new Nodes. [Grow a Loom with the API](/docs/loom) explains each step.
```bash
curl "$ACS_API_BASE/looms" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "The sky is"}'
```
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID/nodes/$NODE_ID/branch" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "max_tokens": 2, "n": 2}'
```
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
## List the models [#list-the-models]
`GET /v1/models` returns the `id` to send as `model`, and the context length of each model in `capabilities.max_model_len`. [List the models](/docs/models#list-the-models) describes each field.
```bash
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"
```
## Tokens [#tokens]
Before Inference runs a request, the API holds the most tokens that the request can use, as [The Hold](/docs/concepts/budgets#the-hold) explains. When your tokens cannot hold all of them, `POST /v1/completions` lowers `max_tokens` to what fits and sends `X-Acs-Max-Tokens-Clamped`. It answers 429 `budget_exceeded` only when not even one token for each choice fits. Without `stream`, a `max_tokens` above 15,600 gets 400 `deadline_infeasible`, and a `max_tokens` that you leave out is at most 15,600.
## Headers [#headers]
| Header | Set by | What it does |
| --- | --- | --- |
| `Authorization: Bearer $ACS_API_KEY` | You | Sends your API key. Every `/v1` route needs it. |
| `Content-Type: application/json` | You | Says that the body is JSON. The API reads the body of a `/v1` request as JSON whatever its type, so `curl -d` without this header works too. |
| `Idempotency-Key` | You | Runs a request that writes only once. The same request again with the same key gets the first answer ([Send a request again safely](/docs/retries#send-a-request-again-safely)). |
| `Accept-Encoding: gzip` | You | Asks for a JSON answer of 500 bytes or more compressed, with `Content-Encoding: gzip`. The `openai` SDKs and `curl --compressed` send it and decompress the answer. A stream is not compressed. |
| `Content-Type: text/event-stream` | The API | Marks a stream. |
| `Location` | The API | The path of a Job, on the answer of `POST /v1/completion-jobs`. |
| `X-Acs-Max-Tokens-Clamped` | The API | Says that `POST /v1/completions` lowered `max_tokens`, from which value to which, and why ([Detect a lowered `max_tokens`](/docs/cookbook/budget-cap#detect-a-lowered-max_tokens)). |
| `Allow` | The API | Lists the methods of a route, with a 405 answer. |
| `Retry-After`, `retry-after-ms` | The API | Give the wait before the next try. The API sets `Retry-After` on a 429 of its traffic limits, a 429 `harvest_concurrency_exceeded` or `harvest_capacity_exceeded`, a 429 `spend_cap` (an hour), and a 409 `running`. A 202 of a Job's result has it too, a second. It sets it on a 503, and on a 502 that `retryable` says to try again. When Inference is busy, the API passes on both headers as Inference sent them, with a 429 `rate_limited`. |
| `x-should-retry: false` | The API | Tells the `openai` SDK not to send the request again at once ([Which errors to try again](/docs/retries#which-errors-to-try-again)). It comes with 400 `deadline_infeasible`, 409 `no_answer`, 409 `idempotency_conflict`, 409 `harvest_not_cancellable`, and a 409 about the status of a Node. It comes with 429 `budget_exceeded`, `harvest_quota_exceeded`, `job_capacity`, and `spend_cap`. It comes with the API's own 500, with 413 `request_too_large`, with 504 `upstream_deadline`, and with each error of Inference whose `retryable` is `false`. |
| `X-Acs-Longpoll: declined` | The API | Says that a read of a harvest with `wait` answered at once, because too many reads wait. It comes with `Retry-After: 3` ([Check on the Job](/docs/harvest#check-on-the-harvest)). |
| `X-Request-Id` | You, or the API | The id of the request, on every answer. An error has the same id in `request_id` at the top of its body. Send it when you report a problem. If you send an `X-Request-Id` of 1 to 64 letters, digits, `-`, or `_`, the API uses yours. |
## Errors [#errors]
Every error has the error object of the old platform's API. Two errors can have the same status, so tell them apart by `code`. [Errors](/docs/errors) lists every status.
| Status | `code` | More |
| --- | --- | --- |
| 400 | `invalid_request` | [Unknown field](/docs/errors#unknown-field) |
| 400 | `bad_json` | [Status codes](/docs/errors#status-codes) |
| 400 | `model_not_found` | [Unknown model](/docs/errors#unknown-model) |
| 400 | `context_length_exceeded` | [Prompt too long](/docs/errors#prompt-too-long) |
| 400 | `deadline_infeasible` | [Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream) |
| 400 | `harvest_unsupported` | [Harvests on each model](/docs/models#harvests-on-each-model) |
| 401 | `missing_authorization`, `invalid_api_key`, `key_revoked`, `key_disabled`, or `account_suspended` | [Send the key](/docs/api-keys#send-the-key) |
| 404 | `job_not_found` or `harvest_job_not_found` | [Status codes](/docs/errors#status-codes) |
| 409 | `null` | [Status codes](/docs/errors#status-codes) |
| 409 | `running` or `no_answer` | [Send a request again safely](/docs/retries#send-a-request-again-safely) |
| 429 | `budget_exceeded` | [Not enough tokens](/docs/errors#not-enough-tokens) |
| 429 | `rate_limited` | [Traffic](/docs/limits#traffic) or [Jobs](/docs/limits#jobs) |
| 429 | `queue_full` | [Traffic](/docs/limits#traffic) |
| 429 | `harvest_quota_exceeded`, `harvest_concurrency_exceeded`, or `harvest_capacity_exceeded` | [Harvests](/docs/limits#harvests) |
| 429 | `spend_cap` | [Spend limit](/docs/limits#spend-limit) |
| 500 | `internal_error` | [Status codes](/docs/errors#status-codes) |
| 502, 503, or 504 | `upstream_unreachable`, `upstream_server_error`, `vllm_oom`, `vllm_engine_dead`, `circuit_open`, or `upstream_deadline` | [Inference fails](/docs/errors#inference-fails) |
## Limits [#limits]
[Limits](/docs/limits) gives every number for tokens, traffic, API keys, Jobs, harvests, Looms, request size, time limits, and kept data.
## Next steps [#next-steps]
[Completions](/docs/completions) describes every field of the request and of the response. The [Cookbook](/docs/cookbook) has programs for whole tasks.
# Completions (/docs/completions)
`POST /v1/completions` continues one prompt, or several prompts, with a Base model. It follows OpenAI's completions API, so the `openai` SDK calls it with `client.completions.create(...)`. [Sample continuations](/docs/sampling) shows the tasks, and the [API reference](/docs/reference/completions/completions.createCompletion) has the schema of every field.
## Request fields [#request-fields]
`prompt` is required. Apart from `model` and `max_tokens`, a field that you leave out, or sets to `null`, gets OpenAI's default, or the value that turns a vLLM field off. The model's own `generation_config.json` never sets a default.
A field marked vLLM comes from vLLM, the engine that runs the Base models in Inference, and is not in OpenAI's API. A field marked vllm-lens comes from vllm-lens, the vLLM plugin that reads and changes the residual stream. The `openai` SDK sends both kinds only in `extra_body` ([Send vLLM's fields](/docs/sampling#send-vllms-fields-from-the-openai-sdk)).
### What to continue [#what-to-continue]
| Field | Default | What it does |
| --- | --- | --- |
| `model` | the `default` model | The `id` of a model from [`GET /v1/models`](/docs/models), such as `llama-8b`. Left out, the model that the list marks `default`. |
| `prompt` | required | A text, a list of token ids, or a list of several texts or several token-id lists. |
| `max_tokens` | the context left | The most tokens to write for each choice. The rules are under this table. |
| `n` | `1` | The number of choices to write for each prompt. |
| `stop` | none | A text, or a list of at most 4 texts. The model stops before it writes one of them. |
| `echo` | `false` | Put the prompt in front of the generated text. |
| `add_special_tokens` | `true` | vLLM. Add the model's special tokens, such as BOS, to a text prompt. Inference adds none to token ids. |
The rules of `max_tokens`:
- Left out or `null`, it is the model's `max_model_len` less the prompt's tokens as the [Hold counts them](/docs/concepts/budgets#the-hold). It is at least 1 and at most 32,000, and at most 15,600 without `stream`.
- A request with more than one prompt, or with `n` above 1, must set it.
- A request writes at most 32,000 tokens in all: the prompts × `n` × `max_tokens` ([Size caps](#size-caps)).
- Without `stream`, a value above 15,600 gets 400 `deadline_infeasible`. Inference cannot write more within the API's 14-minute wait ([Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream)).
- The prompt and `max_tokens` together must fit the model's `max_model_len` ([Prompt too long](/docs/errors#prompt-too-long)).
- Near the end of your tokens, the API lowers it to what your Budget can hold. The header `X-Acs-Max-Tokens-Clamped` then says so ([Near the end of your tokens](/docs/concepts/budgets#near-the-end-of-your-tokens)).
- In a Job's request and in a Loom Branch, it is 16 when you leave it out.
### How to sample [#how-to-sample]
| Field | Default | What it does |
| --- | --- | --- |
| `temperature` | `1` | Divide the scores of the tokens by this number before sampling, from `0` to `2`. `0` takes the likeliest token each time. |
| `top_p` | `1` | Sample only from the likeliest tokens whose probabilities add up to `top_p`. More than 0 and at most 1. |
| `top_k` | `0` | vLLM. Sample only from the `top_k` likeliest tokens. `0` or `-1` turns the filter off. |
| `min_p` | `0` | vLLM. Leave out the tokens less likely than `min_p` times the likeliest token, from `0` to `1`. `0` turns the filter off. |
| `presence_penalty` | `0` | Make a token that is already in the generated text less likely, or more likely for a negative number, from `-2` to `2`. |
| `frequency_penalty` | `0` | As `presence_penalty`, but more for each time that the token is already in the generated text, from `-2` to `2`. |
| `repetition_penalty` | `1` | vLLM. Make the tokens of the prompt and the generated text less likely by this factor, more than `0`. `1` turns the penalty off. |
| `seed` | none | Sample with this seed, 0 or more. |
At `temperature` 0, or with the same `seed`, the same request on the same model gets the same tokens and log probabilities again, with one exception. vLLM batches your request with other requests. The batch can change a log probability by a little, and now and then a token where two tokens are almost equally likely.
### What to return [#what-to-return]
| Field | Default | What it does |
| --- | --- | --- |
| `logprobs` | none | Return the log probability of each generated token, and of the `logprobs` likeliest tokens at its position. 0 to 100. |
| `prompt_logprobs` | none | vLLM. Return the log probability of each prompt token, and of the likeliest tokens at its position. 0 to 100, or `-1` for the whole vocabulary of one prompt. Not with `stream`. |
| `stream` | `false` | Send the Completion as Server-Sent Events. |
| `stream_options` | none | Only with `stream`. `{"include_usage": true}` adds a last chunk with the usage of the request. With `"continuous_usage_stats": true` (vLLM), each chunk has the usage of its choice so far, with or without `include_usage`. |
| `output_residual_stream` | none | vllm-lens. Return the Activations at these Layers, or `true` for every Layer. `false` returns none. Only for one prompt and an `n` of 1. |
| `apply_steering_vectors` | none | vllm-lens. Add these Steering vectors to the residual stream. Only for one prompt and an `n` of 1. |
[Read token probabilities](/docs/token-probabilities), [Read Activations](/docs/activations), and [Steer a Completion](/docs/steering) show these fields in use.
### Other fields [#other-fields]
| Field | Default | What it does |
| --- | --- | --- |
| `priority` | `0` | vLLM. 0 or more. A request with a lower number runs first. Requests with the same number wait in the order that they arrived. `null` is 0. A text of digits, such as `"5"`, and `true` or `false` count as their number. |
| `user` | none | OpenAI's id for the person who asked. Inference gets it and does nothing with it. |
## Strict validation [#strict-validation]
The API checks the type of every field before the request reaches Inference, and it refuses what it does not know. A wrong value never falls back to a default ([Why the API is strict](/docs/concepts/base-models#why-the-api-is-strict)).
- A field that the API does not know gets 400 with the code `invalid_request` ([Unknown field](/docs/errors#unknown-field)).
- A value of the wrong type gets 400, with `param` set to the field. A number in a string, such as `"2"`, is not a number, except for `priority`. `1` and `"true"` are not booleans. A whole number such as `1` is valid for a decimal field such as `temperature`.
- The API checks the ranges of `logprobs`, `prompt_logprobs`, `seed`, `priority`, and the sampling fields in [How to sample](#how-to-sample). It also checks the shape of the Activations and Steering vector fields, and the [size caps](#size-caps).
- The API refuses `n` above 1 at a `temperature` below 0.00001, where Inference picks the likeliest token each time.
- The API refuses `stream_options` without `stream: true`, and `prompt_logprobs` with `stream: true`.
- Inference checks the other values, for example that each token id of a prompt is in the model's vocabulary. The API passes on Inference's 400, with Inference's message and `param`.
This request sends `max_tokens` as a string, so the API refuses it:
```bash refused
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": "2"}'
```
```json title="Error"
{
"error": {
"message": "Invalid type for 'max_tokens': expected a number.",
"type": "invalid_request_error",
"param": "max_tokens",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
## Size caps [#size-caps]
Each cap keeps one request from using too much memory in the API or in Inference. A request over a cap gets 400 `invalid_request_error`, with `param` set to the field, and does not reach Inference. The caps count a prompt's tokens as the [Hold counts them](/docs/concepts/budgets#the-hold), with the model's tokenizer.
| Cap | Field | Why |
| --- | --- | --- |
| The prompts × `n` is at most 128 | `n` | OpenAI's limit on `n`. |
| The prompts × `n` × `max_tokens` is at most 32,000, and a `max_tokens` of 0 counts as 1 | `max_tokens` | The Hold of a request is then at most its prompt and 32,000 tokens. |
| At most 4 `stop` texts | `stop` | OpenAI's limit. |
| `prompt_logprobs: -1` takes at most 256 prompt tokens in all × `n`, or 16 with `output_residual_stream` | `prompt_logprobs` | Each prompt token then has a log probability for each token of the vocabulary (128,256 for `meta-llama/Llama-3.1-8B`). Inference holds them all in memory. |
| At most 33,554,432 (2^25) top log probabilities of the prompt. Each prompt token × `n` has k + 1 for `prompt_logprobs` k, and k + 1 more for `logprobs` k with `echo`. | `prompt_logprobs` or `logprobs` | Inference holds each one in memory, and the answer carries each one. |
| Layers × tokens × the model's hidden size is at most 33,554,432 (2^25) for one capture. The tokens are the prompt's and `max_tokens`. | `output_residual_stream` | The API reads the Activations of a capture as one text. |
| At most 8 Steering vectors, and 2,097,152 (2^21) numbers in all of them. A vector with `position_spans` counts once for each position of its spans. | `apply_steering_vectors` | The API sends the numbers to Inference as float32: at most 8 MiB. |
`llama-8b` has a hidden size of 4,096, so one capture is at most 8,192 Layers × tokens. That is all 32 Layers for 256 tokens, or 1 Layer for 8,192 tokens.
The whole request body is at most 8 MiB on every route, and a larger body gets 413 ([Status codes](/docs/errors#status-codes)). The body can cap the Steering vectors first: in JSON, 8 MiB holds about 699,000 numbers of 12 characters each. [Encoded Steering vectors](/docs/steering#encode-a-steering-vector) take about a tenth of those bytes, so eight vectors over all 32 Layers of `meta-llama/Llama-3.1-8B` fit.
This request asks for 2 choices of 16,001 tokens, which is 32,002 tokens in all:
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 16001, "n": 2}'
```
```json title="Error"
{
"error": {
"message": "At most 32000 generated tokens in all: n = 2 times max_tokens = 16001 for each of 1 prompts is 32002.",
"type": "invalid_request_error",
"param": "max_tokens",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
## Response fields [#response-fields]
The [Quickstart](/docs/quickstart#send-a-prompt) shows a whole Completion.
| Field | What it is |
| --- | --- |
| `id` | The id of the Completion, from Inference. |
| `object` | Always `"text_completion"`. |
| `created` | When Inference made the Completion, in Unix seconds. |
| `model` | The model that wrote it. |
| `choices` | One choice for each prompt and each of its `n` samples, in the order of their `index`. |
| `usage` | The `prompt_tokens`, `completion_tokens`, and `total_tokens` of the whole request. `prompt_tokens` counts each prompt once, whatever `n` is. |
| `activations` | The Activations, only when the request has `output_residual_stream`. |
Each choice has these fields:
| Field | What it is |
| --- | --- |
| `index` | `p * n + i` for choice `i` of prompt `p`. |
| `text` | The generated text. With `echo`, the prompt comes first. |
| `logprobs` | `null`, or the lists that `logprobs` asks for. |
| `finish_reason` | `"length"` when the choice reached `max_tokens` or the model's context length. `"stop"` when the model wrote its end-of-text token or a `stop` text. Inference can add other reasons. |
| `stop_reason` | The `stop` text that ended the choice, or `null`. With `finish_reason` `"stop"`, `null` means that the model wrote its end-of-text token. |
| `prompt_logprobs` | `null`, or the list that `prompt_logprobs` asks for, with `null` for the first prompt token. |
In a stream, each `data:` event is a chunk with the same `id`, `object`, `created`, and `model`. Its `choices` has the next text of one choice, with `finish_reason` and `stop_reason` `null` until the last chunk of that choice. With `include_usage`, a last chunk has `"choices": []` and the `usage` of the whole request. The Activations come in a last chunk of their own, and `data: [DONE]` ends the stream. Every 5 seconds the stream also has a `: keepalive` comment line, which SSE clients skip ([Requests with a stream](/docs/concepts/long-requests#requests-with-a-stream)).
## Not in the API [#not-in-the-api]
The API does not take these fields and routes of OpenAI's API:
- `best_of`. vLLM accepts it and ignores it, so the API refuses it as an unknown field. `n` returns each choice instead.
- `suffix`. vLLM supports it only for models that fill in the middle of a text.
- `logit_bias`.
- Chat. The API applies no chat template ([No chat template](/docs/concepts/base-models#no-chat-template)). `POST /v1/chat/completions` gets 400 `invalid_request_error` with the message `This API serves Base models, which have no chat format: send POST /v1/completions with a prompt instead of POST /v1/chat/completions with messages.`
# Work in the background (/docs/concepts/background-work)
Jobs, harvests, Loom Nodes, and Playground Runs run inside the API, not in your script or your browser, so they finish with no client connected. A few rules decide when each one runs, and what happens to it when the API restarts.
## What runs in the background [#what-runs-in-the-background]
When you submit a Job or a harvest, grow a Branch, or start a Run, the API saves the work and answers at once. The API's Runner then runs the work to its end. Your script can stop, your connection can drop, and you can close the tab. The work goes on. You read its result later.
Four kinds of work run this way:
- A Job is one Completion request. The API runs it and keeps its result.
- A harvest reads the residual stream of many prompts. The API starts it on Inference's GPUs, follows it until it ends, and gives you links to its files.
- A Loom Node is one Completion of the text on the path to its parent. It runs as a stream, so its text grows while Inference writes it.
- A Playground Run is one Completion that you start in the App. The API keeps it until you delete it.
A request to `POST /v1/completions` is not background work. Your client waits for its answer.
Each Job, each harvest Job, each Node, and each Run takes its own Hold when it starts to run. [Holds of background work](/docs/concepts/budgets#holds-of-background-work) explains how.
## The order of the work [#the-order-of-the-work]
The API runs up to 4 Jobs at once. A Job stays `queued` until it starts.
When more Jobs wait than can run, the API starts the oldest Job of a User who has no Job running. It starts a Job of a User who already has one running only when no other User's Job waits. So one User's long Jobs do not keep another User waiting long. A Job of yours can wait behind newer Jobs of other Users while another of your Jobs runs.
Loom Nodes run apart from Jobs, up to 8 at once, the oldest first. A long Job never delays a Branch. Harvests run apart from both, the oldest first. A harvest waits for no Job, and runs on GPUs of its own. Playground Runs also run apart from both, so a Run starts while your Jobs run. Inference then batches the requests that reach it at the same time.
Work whose API key is paused waits, and the API runs the work behind it. [Jobs](/docs/limits#jobs), [Harvests](/docs/limits#harvests), and [Looms](/docs/limits#looms) list the numbers.
## When the API restarts [#when-the-api-restarts]
The API restarts when a new version of it is deployed. It can also stop without warning, for example when its process crashes. Jobs, harvests, and Nodes go on after both. Each request of a Job, and each harvest, spends tokens once.
Before a planned restart, the API gives running work a short time to end. Work that does not end in time is stopped and goes back to `queued`. The API releases its Hold, so the cut request costs nothing. When the API runs again, the work runs again:
- A Job whose request already has its answer keeps that answer and does not run again. A request that the restart cut off runs again.
- A harvest keeps its GPU run, which goes on while the API restarts. The API follows the same GPU run again, and starts no second one.
- A Node starts again from the beginning, with no text.
- A Run starts again from the beginning.
While new GPU work is paused ([Spend limit](/docs/limits#spend-limit)), the work that went back to `queued` waits, and runs when the pause ends.
When the API stops without warning, its running work shows `running` for some minutes. Then the API finds that the process is gone. It releases the Holds of that work, and queues each Job, harvest, and Node again, as after a planned restart. A Playground Run fails instead and keeps the text that it had, so you see what it wrote before the API stopped. It costs nothing.
## Paused keys and suspended access [#paused-keys-and-suspended-access]
Background work runs with the API key that made it. When you pause that key, its work waits. Work that has not started stays `queued`, and holds no tokens. A running Job finishes the request that is in Inference, and then goes back to `queued`. A running harvest runs to its end. When you resume the key, the work runs from where it stopped, and each request still pays once.
A paused key gets 401 for every call, so it cannot read its own Jobs and Looms while it is paused. A revoke stops the work for good. Each of its Completions that has not run fails with 401 `invalid_api_key`, and spends no tokens. Each of its harvests that has not started fails with `error.code` `unauthenticated`. Playground Runs use no API key, so a paused or revoked key does not touch them.
When an admin suspends your access, none of your work runs. Each Completion that waits fails with 401 and the message `Your access to ACS Infra is suspended. Write to infra@acsresearch.org if you think this is a mistake.`, and spends no tokens. Work that failed while your access was suspended does not run again when an admin restores it. [If your access is suspended](/docs/get-access#if-your-access-is-suspended) says what else changes.
## Next steps [#next-steps]
[Run a Job](/docs/jobs) submits a request and fetches its result. [Grow a Loom with the API](/docs/loom) branches under a Node and follows the Nodes. [Find your Runs again](/docs/app/playground#find-your-runs-again) opens the Runs that the API kept. [Manage your keys](/docs/api-keys#manage-your-keys) pauses and resumes a key.
# Base models (/docs/concepts/base-models)
Every model that the API serves is a Base model, which treats what you send as the start of a text and continues it. So the shape of your prompt decides the shape of the answer.
## What a Base model does [#what-a-base-model-does]
A Base model is a pretrained language model without instruction or chat tuning. It predicts the next token of a text, then the one after it, and so it continues the text that you send. It does not treat your text as a question to answer. A prompt such as `"What is the capital of France?"` can go on with more questions as well as with an answer. The prompt `"The capital of France is"` asks for the continuation that you want.
The model reads tokens, not characters. For a text prompt, the tokenizer of `llama-8b` adds the BOS token `<|begin_of_text|>` in front, so `"The sky is"` is 4 tokens. Inference adds no special tokens to a prompt of token ids, so the ids that you send are the whole prompt.
The API follows OpenAI's completions API, so the `openai` SDK works with it. It also returns the log probability of each token, the model's predictions over the prompt, and the Activations at the Layers that you choose.
## No chat template [#no-chat-template]
A chat model expects its conversation inside a template, with special tokens that mark each turn and who speaks. A Base model was not trained on such a template, so the API applies none. The model continues the text that you send, with only the BOS token in front.
For the same reason, the API serves no chat route. `POST /v1/chat/completions` always answers 400 `invalid_request_error`, with this message:
```text title="Message"
This API serves Base models, which have no chat format: send POST /v1/completions with a prompt instead of POST /v1/chat/completions with messages.
```
In the `openai` SDK, that means `client.completions.create(...)` and not `client.chat.completions.create(...)`. A conversation can still go in as plain text, as a transcript that the model continues.
## Why the API is strict [#why-the-api-is-strict]
The API checks the name and the type of every field before a request reaches Inference, and it refuses what it does not know. A field that it does not know gets 400 with the code `invalid_request`. A value of the wrong type gets 400, with `param` set to the field. The API does not replace a bad value with a default.
Research depends on knowing which settings made a result. If the API ignored a misspelled field such as `temprature`, the request would run at the default temperature, and nothing in the answer would show it. A refusal turns that silent mistake into an error that you see at once.
The same rule covers fields that other servers accept and ignore. vLLM accepts `best_of` and does nothing with it, so the API refuses `best_of` as an unknown field. Types are strict too: `"2"` is a string and not a number, and `1` is not `true`. [Strict validation](/docs/completions#strict-validation) lists every check.
## Default sampling [#default-sampling]
A field that you leave out gets OpenAI's default, or the value that turns a vLLM field off: `temperature` 1, `top_p` 1, `top_k` 0, no penalties. At these defaults, Inference samples each token from the model's own distribution, unchanged.
A model's repository can set its own sampling defaults in `generation_config.json`. The API sends Inference every field, defaults included, so that file never decides a value. A request then means the same on every model. When you send one request to two models, both sample the same way, and a difference in the text comes from the models.
`max_tokens` has rules of its own, which [Request fields](/docs/completions#request-fields) lists.
## Why repeats differ [#why-repeats-differ]
A Base model samples each token, so the same request at the default `temperature` writes other text on each run. The outputs in these docs are one sample each, recorded from `meta-llama/Llama-3.1-8B`, and your outputs differ.
With `temperature` 0, Inference takes the likeliest token each time, and with the same `seed`, the same request on the same model samples the same tokens again. vLLM's batching can still change a result a little. Inference runs your request in a batch with other requests, and the numbers that the GPU computes depend a little on that batch. A log probability can then move by a small amount, and now and then a token changes where two tokens were almost equally likely. Read a small difference between two runs as no difference, and send a request more than once when the difference matters.
## Next steps [#next-steps]
[Sample continuations](/docs/sampling) sets the sampling fields and sends several prompts. [Completions](/docs/completions) lists every field and its default. [Read token probabilities](/docs/token-probabilities) measures what the model predicts at each position.
# Budgets and Holds (/docs/concepts/budgets)
Before Inference runs a Completion, the API holds the most tokens that the Completion can use, from your monthly allowance or your Budget. So requests that run at once never spend more tokens than you have.
## Your monthly allowance [#your-monthly-allowance]
Your monthly allowance is the number of tokens that you may spend in each calendar month, in UTC, across your API keys and the App. An approval gives you a monthly allowance of 5,000,000 tokens by default, so you can start work without waiting for a Grant. An admin can set another allowance for you. [Tokens](/docs/limits#tokens) lists the default.
A request takes its Hold from this month's allowance when the whole Hold fits there, and from your Budget when it does not. The API does not split one Hold between the two. Your allowance is spent first, so your Budget lasts longer.
On the first day of each month, the count starts again from 0. Tokens that you did not spend do not carry over. A Hold counts toward the month in which it started. Tokens that it gives back after the month changed do not change the new month's count.
## Your Budget [#your-budget]
Your Budget is the number of tokens that you may still spend when this month's allowance cannot hold a request. It has no period and no reset. It grows only when an admin adds tokens with a Grant or sets it to a higher number, or when an Invite brings tokens. It shrinks by the Usage of each Completion that it pays for.
An approval starts your Budget at 0 tokens, unless the admin adds a Grant with it, and the monthly allowance pays for your first requests. If you joined with an Invite, your Budget starts with the Invite's tokens.
ACS can also give an API key its own Budget, with a Grant for that key. A key's own Budget limits what one script can spend, and [Ask for more tokens](/docs/support#ask-for-more-tokens) says how to ask for one. Each Hold of that key comes out of your allowance or your Budget, and out of the key's Budget as well. The key stops when either one cannot hold a request. A key without its own Budget spends only from your allowance and your Budget.
## The Hold [#the-hold]
Before the API sends a request to Inference, it takes a Hold: the most tokens that the request can use. For each prompt, the Hold is the prompt's tokens and `n × max_tokens`, with `max_tokens` counted as at least 1. With `echo` and `max_tokens: 0`, Inference still generates and counts 1 token for each choice.
The API counts a text prompt with the model's own tokenizer, the files that Inference loads. It counts the tokens that Inference reads. The count includes the BOS token when `add_special_tokens` is on, as it is by default. A prompt of token ids counts one token for each id.
Some models have no tokenizer in the API. For them, a text counts one token for each byte of its UTF-8 text, and 1 for the BOS token. A byte-level tokenizer never makes more tokens than bytes, so this count can only be too high. The tokenizer reads at most 8 bytes of a request's texts for each token of the model's context. The texts after that count by their bytes in the same way.
For example, `"The sky is"` with `max_tokens: 2` on `llama-8b` holds 6 tokens: 4 for the prompt with its BOS token, and 2 for `max_tokens`. A request that stops before `max_tokens` gives back the tokens that it did not write.
While a request runs, its Hold counts as spent. Two requests at once never hold more tokens than you have, and each one sees what the others hold. The App's **Usage** page shows the tokens left after their Holds, and which requests still run, as [Read your usage](/docs/app#read-your-usage) says.
## How a Hold ends [#how-a-hold-ends]
Each Hold ends once, by one rule for `POST /v1/completions`, Jobs, Looms, and the Playground. It settles to the tokens that the request used, or the API releases it and the request costs nothing.
A Hold settles in these cases:
- When Inference answers, you pay the Usage that it reports, and the rest of the Hold goes back.
- When a stream stops early, because you closed it or it broke, you pay for the tokens that it sent. A Loom Node runs as a stream, so a Node that you stop pays for the text that it grew.
- When a request without a stream stops after it reached Inference, you pay the whole Hold. A Job that you cancel while it runs is such a request.
A request without a stream pays its whole Hold because Inference reports its Usage only at the end. When the request stops in the middle, the API has no count of the tokens, and Inference may have done the work. A stream tells the API how many tokens it sent so far, so a cut stream pays only for those.
A released Hold costs nothing, and the **Usage** page does not list its Completion. The API releases a Hold in these cases:
- Inference refuses the request, or fails before it sends any token, a timeout included.
- The request stopped before it reached Inference.
- The API itself failed.
- The API stopped a request of a Job, a Node, a Playground Run, or a harvest, so that it runs again later.
If a Completion uses more tokens than its Hold, the API takes the rest from what you have left, down to 0.
## Near the end of your tokens [#near-the-end-of-your-tokens]
When the tokens that you have left cannot hold a request to `POST /v1/completions`, the API does not refuse the request at once. It lowers `max_tokens` to the most that fits, and runs the request, so that a script near the end of its tokens keeps making progress. The answer then has this header:
```text title="Header"
X-Acs-Max-Tokens-Clamped: requested=200,applied=21,reason=budget,source=client
```
Here, 25 tokens were left, and the prompt `"The sky is"` holds 4 of them on `llama-8b`. So the API lowered `max_tokens` from 200 to 21, and a choice can end at 21 tokens with `finish_reason: "length"`. `source` is `client` when you set `max_tokens`, and `default` when you left it out. `reason` is `budget` here. It is `deadline+budget` when the API first lowered a `max_tokens` that you left out to the time limit of a request without a stream. [Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream) explains that limit.
Only when not even one token for each choice fits does the API refuse the request. It answers 429 `budget_exceeded`, and the request does not reach Inference. The message names what ran short: your Budget, the key's own Budget, or your monthly allowance, with the day on which it refills. [Not enough tokens](/docs/errors#not-enough-tokens) shows the error.
Jobs, Loom Branches, and Playground Runs never lower `max_tokens`. A request of theirs that your tokens cannot hold fails with 429 `budget_exceeded`, and its `max_tokens` stays as you set it.
## Holds of background work [#holds-of-background-work]
A Job, a Branch, or a Run holds no tokens when you make it. Each Job, each Node, and each Run takes its own Hold when it starts to run, as a request to `POST /v1/completions` does.
When you make a Job, a Branch, or a Run, the API checks only that your tokens can hold its smallest request. If they cannot, it refuses the work with 429 `budget_exceeded` and makes nothing. The request can still find too few tokens when it runs. A Job then fails, and its result is the 429 error object. A Node fails with the same error.
A harvest generates no tokens, so it holds the tokens of its prompts only. It counts one token for each byte of each prompt's UTF-8 text, and 1 more for each start token. When you submit it, the API checks that your tokens can hold all of that, and answers 429 `budget_exceeded` if they cannot. The harvest takes its Hold when the API starts it, and keeps it while its GPU run lasts.
A harvest's Hold ends by its own rule, because Inference counts its tokens only when it completes:
- A completed harvest pays the tokens that Inference read, up to its Hold. Its `tokens` field shows them.
- A harvest that you cancel while it is `running` and its GPU run started pays its whole Hold. Any other cancel costs nothing.
- A failed harvest costs nothing.
The App's **Usage** page lists a harvest's Hold under the source **Harvests**.
A Node's prompt is the text of the path from the root to its parent. So a Node deep in a Loom holds more tokens than a Node near the root. A Branch of 4 Nodes takes 4 Holds, one for each Node.
A Completion that waits in its API key's line for a place to run holds no tokens until it starts. [Traffic](/docs/limits#traffic) gives the size of that line.
Activations and Steering vectors cost no tokens of their own. A request with them holds and spends tokens as any other Completion does.
## Next steps [#next-steps]
[Handle the end of your Budget](/docs/cookbook/budget-cap) reads the header and the 429 errors in code. [Ask for more tokens](/docs/support#ask-for-more-tokens) says whom to write to. [Tokens](/docs/limits#tokens) lists every number about tokens.
# Cold starts and long requests (/docs/concepts/long-requests)
Inference starts each Base model's GPU when requests come and stops it when they end, so a request can wait for a start. The API also waits for Inference only so long, and these limits decide how to send a long request.
## Warm and cold models [#warm-and-cold-models]
Inference keeps no container of a Base model running while no requests come, because a GPU that waits all day costs money all day. An admin can keep a model Warm at set hours instead ([Keep a model Warm](/docs/app#keep-a-model-warm)). After its last request, a container of `llama-8b` stays up for 30 minutes, and a container of `llama-405b` or `trinity-truebase` for 5 minutes. Then it stops. The next request waits while a new container starts and loads the model. After an update to Inference, the first few starts take longer.
A start of `llama-405b` or `trinity-truebase` loads eight GPUs. In testing, the two starts took 25 and 17.6 minutes, longer than [the 14-minute wait](#the-14-minute-wait). So the first request to `/v1/completions` for a cold one times out.
A Job's requests, and the App's Runs and Branches, wait up to 45 minutes for Inference's answer to start. No client holds a connection open for them, so they wait through a start of a cold model.
A Warm model is one with a container that serves, so that its next Completion starts at once. `GET /v1/models/{model}/status` answers `warm: true` for such a model. While a model starts, the route also says the start's stage in `boot.stage_label` and how long the stage has run in `boot.stage_age_s`. Inference reports each container's start, so the route never starts a GPU.
[Check whether a model is Warm](/docs/models#check-whether-a-model-is-warm) shows the route.
## The 14-minute wait [#the-14-minute-wait]
The API waits up to 14 minutes for Inference to answer a request. In a stream, it waits up to 14 minutes between two chunks. If Inference does not answer in that time, the API answers 504 `upstream_deadline` with the message `Could not reach Inference: Request timed out.` and the header `x-should-retry: false`. In a stream that has started, the same error comes as the last event before `[DONE]`.
The header tells the `openai` SDK not to send the request again, and the API does not try the request again either. Inference did not answer for 14 minutes, so another try at once would wait as long. It would also start the work on a GPU again. [Which errors to try again](/docs/retries#which-errors-to-try-again) says when to send the request again.
A request that times out before Inference sends a token costs nothing, because the API releases its Hold. A stream that times out between two chunks pays for the tokens that it sent.
## Requests with a stream [#requests-with-a-stream]
A stream starts when Inference starts its answer. From then on, the stream has a `: keepalive` comment line every 5 seconds, also while Inference is silent between two chunks. SSE clients and the `openai` SDK skip comment lines, and the comments keep the connection busy, so a read timeout does not cut a slow stream. A stream that fails before Inference starts its answer gets an error status, as [Errors](/docs/errors#status-codes) lists.
## Requests without a stream [#requests-without-a-stream]
A request without a stream gets its whole answer at the end. If Inference takes more than 150 seconds, the API starts the answer anyway. It sends status 200, then a space every 5 seconds while Inference works, then the Completion's JSON. JSON allows spaces before a value, so the `openai` SDK and `json.loads` read the body as usual. The spaces keep the connection busy, so a read timeout never cuts it for being idle.
After the 200 has gone out, the status cannot change. A failure after that point, such as the 14-minute timeout, comes as the body instead: the error object, with its `request_id`. So a long request can answer 200 with an error, and only the body tells the two apart.
A request without a stream must finish within the 14 minutes. The API counts on Inference to write 20 tokens a second. At that rate, 13 minutes is 15,600 tokens, a minute inside the wait. So a request without a stream that sets `max_tokens` above 15,600 cannot finish in time. The API answers it with 400 `deadline_infeasible` at once, before it holds any tokens, and with `x-should-retry: false`.
When a request without a stream leaves `max_tokens` out, it gets the room left in the model's context. The API lowers that to 15,600 when the room is larger. The header `X-Acs-Max-Tokens-Clamped` then says `reason=deadline`. The context of `trinity-truebase` is 8,192 tokens, so on that model the room is always smaller, and only a `max_tokens` that you set meets this limit. The context of `llama-8b` and `llama-405b` is 131,072 tokens, so a short prompt there gets 15,600.
A stream has no such limit, because its client reads each token as it comes. For a long output, a stream is the way to send the request. A client that waits for a long request without a stream needs a read timeout above 14 minutes. It also needs automatic retries off, because each try starts the work on a GPU again, and each cut try pays its whole Hold.
## Next steps [#next-steps]
[Set the client for long requests](/docs/retries#set-the-client-for-long-requests) gives the timeout and retry settings for the `openai` SDK. [Stream a Completion](/docs/sampling#stream-a-completion) reads the tokens as Inference writes them. [Time limits](/docs/limits#time-limits) lists every limit in time.
# The residual stream (/docs/concepts/residual-stream)
The residual stream is the vector that each Layer of a Base model reads and adds to, one for each token. Knowing what a read of it returns and what a Steering vector changes lets you choose Layers, scales, and checks that you can trust.
## What a read returns [#what-a-read-returns]
A read with `output_residual_stream` returns the output of each Layer that you list. Layer 0 is the first decoder block, and Layer `i` is the output of block `i`, before the model's final norm. The Layers come in ascending order, each one once, in whatever order you list them.
Each Layer has one row for each token that went through the model. That is every prompt token, with the BOS token first for a text prompt, and every generated token except the last. The model samples the last token from the final forward pass, and that token never goes back through the model, so it has no row. A read therefore has `usage.prompt_tokens + usage.completion_tokens - 1` rows. So a text that you already have, sent with `max_tokens: 1`, reads as one row for each of its tokens.
The numbers come in the model's own precision. `llama-8b` computes in bfloat16, and NumPy has no bfloat16 type. So the tensor comes as `int16`, with `original_dtype` set to `torch.bfloat16`, and each int16 value holds the bits of one bfloat16 number. A bfloat16 number is the upper 16 bits of a float32 number, so decoding shifts the bits into a float32. A conversion of the integers with `astype` gives wrong numbers, because it converts the values and not the bits. [Decode the tensor](/docs/activations#decode-the-tensor) has the helper that does it.
## How a Steering vector changes it [#how-a-steering-vector-changes-it]
A Steering vector adds numbers to the residual stream. Inference adds `scale` times your vector at each Layer in `layer_indices`, at every token: the prompt tokens and the generated tokens. The model then runs on from the changed stream, so the change carries through every later Layer and into the tokens that it samples.
A read of a steered Layer includes the steer. It shows the Layer's own output plus `scale` times your vector. A read of an earlier Layer does not change, because the steer comes after it. Vectors that steer the same Layer add up.
These rules make a steer checkable. When one Layer `L` is steered with `norm_match: false`, a comparison with a scale-0 baseline shows the steer. That comparison works at each token that both requests share, such as the prompt with `max_tokens: 1` and `temperature: 0`. There, Layer `L - 1` does not move, and Layer `L` moves by `scale` times your vector, up to the rounding of bfloat16. On generated tokens, the steered text goes another way, and the rows differ for that reason alone. [Check a steer on your own vector](/docs/steering#check-a-steer-on-your-own-vector) walks through the check.
Inference adds a vector at every token, or at the positions that you name ([Steer chosen positions](/docs/steering#steer-chosen-positions)). It adds it only where a forward pass runs: never at the last generated token.
## Why compare with scale 0 [#why-compare-with-scale-0]
A request with Steering vectors takes another path through Inference. Inference reads Activations and steers with vllm-lens, a vLLM plugin, which runs its steering code at each forward pass. The request also computes every prompt token again instead of reading vLLM's prefix cache. A plain request can differ from a steered one for these reasons alone, even when the vector adds almost nothing.
A request with `scale: 0` takes the same path as the steered request, and adds nothing. So a difference between the scale-0 answer and the steered answer comes from your vector, or now and then from vLLM's batching. With `temperature: 0` on both requests, sampling does not change the text between them either.
Equal texts do not show that a steer ran. A small steer can leave every token as it was. A read of the steered Layer shows whether the vector arrived.
## Choose a scale [#choose-a-scale]
No one `scale` fits every Layer and every model, because the norm of the residual stream differs between Layers and between models. A vector of one length is a large change where the stream is short, and a small change where it is long. A scale that works at one Layer can be too small or too large at another.
A scale that is too large shows in the text. The text starts to repeat phrases, or it breaks into fragments. Each steered Layer's addition also stays in the stream through the later Layers. So when you steer several Layers, the changes add up, and each Layer needs a smaller scale.
That is why a search for a scale starts small. Each answer is compared with the scale-0 baseline, and the scale goes up in small steps. The search stops when the text changes as you want, before it degrades. [Steer with a contrast vector](/docs/cookbook/contrast-steering) steers with one vector at a few scales.
## Match the norm [#match-the-norm]
With `norm_match: true`, Inference first scales your vector to the norm of the residual stream at each token, and then multiplies it by `scale`. A `scale` of 1 then adds a vector as long as the residual stream at that token, and 0.1 adds a tenth of that length. So one `scale` is the same fraction of the stream at every Layer and every token, on any model. Inference does not scale the sum back to the stream's original norm.
With `norm_match: false`, the default, Inference adds `scale` times your vector as you sent it. Its effect then depends on the norm of the stream at that Layer.
## Next steps [#next-steps]
[Read Activations](/docs/activations) reads and decodes the residual stream. [Steer a Completion](/docs/steering) writes a Steering vector and compares it with a zero-scale baseline. [Steer with a contrast vector](/docs/cookbook/contrast-steering) builds a vector from two sets of prompts.
# Errors (/docs/errors)
Every error of the `/v1` routes comes as the error object of the old platform's API, with its status codes and its `code` values. Code written for that API reads these errors unchanged. The `openai` SDK raises its usual error classes. [Retry requests safely](/docs/retries) says which errors to try again.
## The error object [#the-error-object]
Every error answer has the error object, and the id of the request next to it:
```json title="Error"
{
"error": {
"message": "The model `gpt-2` does not exist. The API serves llama-8b, llama-405b, trinity-truebase.",
"type": "invalid_request_error",
"param": "model",
"code": "model_not_found"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
| Field | What it is |
| --- | --- |
| `error.message` | What went wrong, for a person to read. |
| `error.type` | `invalid_request_error` for most errors, and `rate_limit_error` for a rate limit. `upstream_error` for a failure of the model server, `service_unavailable` for a busy database, and `internal_error` when the API fails. |
| `error.param` | The top-level request field at fault, or `null`. |
| `error.code` | A fixed string that a program can check, or `null`. Two errors can have the same status, and `code` tells them apart. |
| `error.retryable` | Whether the same request can succeed when you send it again. It comes on a failure of the model server, and on an error that comes after the answer started. |
| `error.retry_after_seconds` | With `retryable: true`: how many seconds to wait first, as the `Retry-After` header says. |
| `error.upstream_status` | The status that the model server answered with, when it answered. |
| `request_id` | The id of the request, the same as in its `X-Request-Id` header. |
Every answer, an error too, has the header `X-Request-Id`, for example `X-Request-Id: req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a`. The id is new for each request. An error that comes after the answer started, as the last event of a stream, has the id in `request_id` and in `error.request_id`. The API logs each request with its id, so send the id when you [report a bug](/docs/support#report-a-bug).
## Status codes [#status-codes]
The API answers most errors itself. The rest come from Inference, and the API names them by what Inference said.
### Answers from the API [#answers-from-the-api]
| Status | `code` | When |
| --- | --- | --- |
| 400 | `invalid_request` | The request body or an id in the path is not valid, or is over a [size cap](/docs/completions#size-caps). A top-level field that the API does not know gets it too. `param` is the top-level field, and `message` says where the value is and what it takes. |
| 400 | `bad_json` | The request body is not JSON. |
| 400 | `model_not_found` | The API does not serve the model. `param` is `model`. |
| 400 | `chat_completions_unsupported` | The request went to `POST /v1/chat/completions`. Send `POST /v1/completions` with a prompt. |
| 400 | `harvest_unsupported` | `POST /v1/harvest` names a model that the API serves but does not harvest. `param` is `model` ([Harvests on each model](/docs/models#harvests-on-each-model)). |
| 400 | `deadline_infeasible` | `POST /v1/completions` without `stream` asks for a `max_tokens` above 15,600. `param` is `max_tokens`, and the answer has `x-should-retry: false`. |
| 401 | `missing_authorization` | The request has no API key ([API keys](/docs/api-keys)). |
| 401 | `invalid_api_key` | The API key is wrong. |
| 401 | `key_revoked` | The API key is revoked, also when it was revoked while its request ran. |
| 401 | `key_disabled` | The API key is paused, also when it was paused while its request ran. |
| 401 | `account_suspended` | Your access is suspended. |
| 404 | `job_not_found` | None of your Jobs has this id. |
| 404 | `harvest_job_not_found` | None of your harvests has this `job_id`. |
| 404 | `null` | No route has this path, or none of your Looms or Nodes has this id. |
| 405 | `null` | The route does not take this method. The `Allow` header lists the methods that it takes. |
| 409 | `null` | The Node is not in a state for this action. The answer has `x-should-retry: false`. |
| 409 | `harvest_not_cancellable` | The harvest is `done` or `failed`, so it cannot be canceled. The answer has `x-should-retry: false`. |
| 409 | `idempotency_conflict` | The `Idempotency-Key` came before with another request, or on another route. The answer has `x-should-retry: false`. |
| 409 | `job_cancelled` | The result of a Job that you canceled. |
| 409 | `running` | A request with the same `Idempotency-Key` still runs. The answer has `Retry-After: 1`. |
| 409 | `no_answer` | A request with the same `Idempotency-Key` ran, and its answer, such as a stream, cannot be sent again. The answer has `x-should-retry: false`. |
| 410 | `job_result_expired` | The result of the Job expired ([Kept data](/docs/limits#kept-data)). The Job stays. |
| 413 | `request_too_large` | The request body is larger than 8 MiB, or 50 MiB on `POST /v1/harvest` ([Request size](/docs/limits#request-size)). The answer has `x-should-retry: false`. |
| 429 | `budget_exceeded` | Your Budget, or the key's own Budget, cannot hold the request. The answer has `x-should-retry: false`. |
| 429 | `rate_limited` | The API key sent more requests than its rate limit allows. The `type` is `rate_limit_error`, and the answer has `Retry-After`. |
| 429 | `queue_full` | The API key has its most Completions running and waiting ([Traffic](/docs/limits#traffic)). The answer has `Retry-After: 5`. |
| 429 | `job_capacity` | A [Job cap](/docs/limits#jobs) is full, and the message names it. The answer has `x-should-retry: false`. |
| 429 | `harvest_quota_exceeded` | The API key started its most harvests this UTC month ([Harvests](/docs/limits#harvests)). The answer has `x-should-retry: false`. |
| 429 | `harvest_concurrency_exceeded` | You have your most harvests `pending` or `running` on Llama 3.1 8B, or on Llama 3.1 405B and Trinity Large ([Harvests](/docs/limits#harvests)). The answer has `Retry-After: 30`. |
| 429 | `harvest_capacity_exceeded` | All Users together have the most harvests `pending` or `running` on Llama 3.1 405B and Trinity Large ([Harvests](/docs/limits#harvests)). The answer has `Retry-After: 30`. |
| 429 | `spend_cap` | New GPU work is paused to limit spending ([Spend limit](/docs/limits#spend-limit)). A Completion, a Job, a Branch, or a harvest that you start gets it. The answer has `Retry-After: 3600` and `x-should-retry: false`. |
| 500 | `internal_error` | The API failed, and does not know how much of the work it did. The answer has `x-should-retry: false`. |
| 503 | `db_pool_exhausted` | The API's database was busy. Nothing ran and the API held no tokens. The answer has `Retry-After: 1`. |
| 503 | `circuit_open` | The API holds the model back. Inference failed its last 5 requests in a row ([Inference fails](#inference-fails)), or the API has its most requests in flight to it ([Traffic](/docs/limits#traffic)). Nothing ran, and you pay nothing. `retryable` is `true`, and the answer has `Retry-After`. |
### Answers about Inference [#answers-about-inference]
Each of these has the type `upstream_error`, except the two that Inference passes on as they are.
| Status | `code` | When |
| --- | --- | --- |
| 400 | `invalid_request` | Inference refused a value in the request. The API passes on Inference's message and `param`. The type is `invalid_request_error`. |
| 400 | `context_length_exceeded` | The prompt and `max_tokens` are longer than the model allows. `param` is `prompt`. The type is `invalid_request_error`. |
| 400 | `vllm_context_length` | Inference failed with a 5xx status, and its message says that the prompt is too long. |
| 400 | `vllm_invalid_request` | Inference failed with a 5xx status, and its message says that the request is bad. |
| 429 | `rate_limited` | Inference has too many requests. The API passes on its `Retry-After` and `retry-after-ms` headers. The type is `rate_limit_error`. |
| 502 | `upstream_unreachable` | The API could not reach Inference. `retryable` is `true`, and the answer has `Retry-After: 30`. |
| 502 | `vllm_oom` | Inference ran out of GPU memory. `retryable` is `true`, and the answer has `Retry-After: 30`. |
| 502 | `vllm_engine_dead` | Inference's engine stopped. `retryable` is `true`, and the answer has `Retry-After: 30`. |
| 502 | `upstream_server_error` | Inference failed with another status, which `upstream_status` gives. After a 502, 503, or 504, `retryable` is `true` and the answer has `Retry-After: 30`. After another status, such as 500, `retryable` is `false` and the answer has `x-should-retry: false`. |
| 502 | `upstream_auth` | Inference refused the API's own key. Your key is not the cause. The answer has `x-should-retry: false`. |
| 504 | `upstream_deadline` | Inference did not answer within 14 minutes. The answer has `x-should-retry: false`. |
## Not enough tokens [#not-enough-tokens]
Before `POST /v1/completions` sends a request to Inference, it takes a Hold of the most tokens that the request can use. Near the end of your tokens, it lowers `max_tokens` to what fits ([Near the end of your tokens](/docs/concepts/budgets#near-the-end-of-your-tokens)). It answers 429 `budget_exceeded` only when your Budget cannot hold the prompt and one token for each choice.
A Job, a Loom Branch, and the Playground do not lower `max_tokens`. A Completion of theirs that your Budget cannot hold fails with `budget_exceeded` ([Holds of background work](/docs/concepts/budgets#holds-of-background-work)). A harvest that your tokens cannot hold gets 429 `budget_exceeded` when you submit it. If your tokens run short before it starts, it fails with the harvest code `insufficient_quota`.
The message says how many tokens the request can use, and which Budget is short. For example, the prompt `"The sky is"` with `"max_tokens": 2` can use 13 tokens, as the [Hold counts them](/docs/concepts/budgets#the-hold):
- `This request can use up to 13 tokens, more than your token budget has left. Ask for more tokens at https://v2.infra.acsresearch.org/usage/request.`
- `This request can use up to 13 tokens, more than this API key's token budget has left.` when the key has its own Budget.
- When your monthly allowance is used up too, the message says so, and gives the day in UTC on which it refills.
The address opens the App's form to ask for more tokens ([Ask for more tokens](/docs/app#ask-for-more-tokens)).
[Handle the end of your Budget](/docs/cookbook/budget-cap) handles the error in code.
## Unknown field [#unknown-field]
The API refuses a field that it does not know, so a misspelled field cannot change a result without a warning. An unknown field inside an object, such as a Steering vector, gets 400 `invalid_request` with `param` set to the top-level field. Its message names where the field is, for example `Unknown parameter: 'apply_steering_vectors[0].scael'.`
```bash refused
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "temprature": 0}'
```
```json title="Error"
{
"error": {
"message": "Unknown parameter: 'temprature'.",
"type": "invalid_request_error",
"param": "temprature",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
## Unknown model [#unknown-model]
The API answers a model that it does not serve itself, from its list of served models, before it holds any tokens. A Job or a Loom Branch with such a model gets 400 `model_not_found` too, and the API makes nothing.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "gpt-2", "prompt": "The sky is"}'
```
```json title="Error"
{
"error": {
"message": "The model `gpt-2` does not exist. The API serves llama-8b, llama-405b, trinity-truebase.",
"type": "invalid_request_error",
"param": "model",
"code": "model_not_found"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
## Prompt too long [#prompt-too-long]
The prompt and `max_tokens` together must fit the model's `capabilities.max_model_len` from [`GET /v1/models`](/docs/models). `trinity-truebase` has a `max_model_len` of 8,192 tokens. The prompt `"The sky is"` has 4 tokens with the BOS token, so `"max_tokens": 8190` does not fit:
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "trinity-truebase", "prompt": "The sky is", "max_tokens": 8190}'
```
Inference refuses the request, and the API answers 400 with `param` set to `prompt`:
```json title="Error"
{
"error": {
"message": "This model's maximum context length is 8192 tokens. However, you requested 8190 output tokens and your prompt contains 4 input tokens, for a total of 8194 tokens. Please reduce the length of the input prompt or the number of requested output tokens. (parameter=input_tokens, value=4)",
"type": "invalid_request_error",
"param": "prompt",
"code": "context_length_exceeded"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
## Inference fails [#inference-fails]
When Inference fails, the API answers with the type `upstream_error`, a `code` that names the failure, and `retryable`, as [Answers about Inference](#answers-about-inference) lists. The API does not try a failed request again.
| Status | `code` | Message | When |
| --- | --- | --- | --- |
| 502 | `upstream_unreachable` | `Could not reach Inference: Connection error.` | The API could not reach Inference. |
| 504 | `upstream_deadline` | `Could not reach Inference: Request timed out.` | Inference did not answer within 14 minutes. The answer has `x-should-retry: false`. |
| 502 | `upstream_server_error` | `Inference's answer does not fit the request: ` and what is wrong | Inference's answer did not finish each choice, or its `activations` are missing or have another shape than the request asked for. The answer has `x-should-retry: false`. |
After Inference fails 5 requests to a model in a row, the API sends that model no request for 60 seconds. Each request to the model in that time gets 503 `circuit_open` at once, and its `Retry-After` header and `retry_after_seconds` give the seconds left. Then the next requests try the model again. One that succeeds ends the pause, and one that fails starts it again. A Job's request, a Run, or a Branch of that model fails at once with the same message. Other models are not affected.
A Completion without a stream that runs for more than 150 seconds already has its 200 status. A failure after that comes as the body, in the error object. The `openai` SDK raises no exception for it, and returns a Completion whose `error` field holds the error object. [Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream) explains the wait.
### Inference fails during a stream [#inference-fails-during-a-stream]
If Inference fails after a stream starts, the status is already 200. The error object is then the last event before `[DONE]`:
```text
data: {"error":{"message":"","type":"upstream_error","param":null,"code":"upstream_server_error","retryable":false,"request_id":"req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"},"request_id":"req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"}
data: [DONE]
```
The `openai` SDK raises `openai.APIError` at the error event, after the chunks that came before it. The tokens before the error are real, and your Hold settles to the tokens that Inference used.
A stream from Inference is complete only with its last event. If it ends before that, for example when a connection closes, the API sends an error event before `[DONE]`. Its message is `Inference ended the stream before it was complete.` A cut stream therefore never looks complete. A Loom Node fails in the same way, and keeps its text so far.
## Failed harvests [#failed-harvests]
A harvest that fails after you submit it answers no error status. `GET /v1/harvest/{job_id}` answers the Job with `status` `failed`, and its `error` is a message that says why. [Handle a failed harvest](/docs/harvest#handle-a-failed-harvest) lists each cause and what to do.
A Job whose request fails answers its error at `GET /v1/completion-jobs/{job_id}/result`, with the status that `POST /v1/completions` would have answered ([Fetch the result](/docs/jobs#fetch-the-result)). Its `request_id` is the Job's `request_id`, not the `X-Request-Id` of that answer.
## Errors in the `openai` SDK [#errors-in-the-openai-sdk]
The `openai` SDK raises the error class that matches the status:
| Status | Error class |
| --- | --- |
| 400 | `openai.BadRequestError` |
| 401 | `openai.AuthenticationError` |
| 403 | `openai.PermissionDeniedError` |
| 404 | `openai.NotFoundError` |
| 409 | `openai.ConflictError` |
| 429 | `openai.RateLimitError` |
| 5xx | `openai.InternalServerError` |
| 405, 410, and 413 | `openai.APIStatusError` |
| An error event in a stream | `openai.APIError` |
| The connection failed, or timed out before an answer | `openai.APIConnectionError`, or `openai.APITimeoutError` |
Each error class of a status is a kind of `openai.APIStatusError`. Its `status_code` is the status, `body` holds the error object, and `code`, `param`, and `type` are the fields of the error object. Its `request_id` is the `X-Request-Id` header.
```python
import os
import openai
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
try:
client.completions.create(model="gpt-2", prompt="The sky is")
except openai.BadRequestError as error:
print(error.status_code, error.code, error.param)
```
```text title="Output"
400 model_not_found model
```
The SDK sends some requests again before it raises the error ([Retries in the `openai` SDK](/docs/retries#retries-in-the-openai-sdk)).
# Get access (/docs/get-access)
An admin approves each User of ACS Infra. Apply once, set your password from the approval email, then sign in to the App to make API keys and run prompts.
## Apply for access [#apply-for-access]
1. Open the [Apply form](https://www.v2.infra.acsresearch.org/apply) on the ACS Infra website.
2. Fill in **Email**, **Name**, **Organization**, and **Planned usage**. The other questions are optional, but an admin decides from your answers.
3. Read the platform rules, and select **I agree to these rules.**
4. Select **Apply for access**. The page shows "Application received".
An admin reviews your Application, and you get an email when they decide. When they approve it, you get the email "Your ACS Infra application is approved". The approval gives you a [monthly allowance](/docs/concepts/budgets#your-monthly-allowance) of tokens, so you can send requests at once. If they reject it, the email links to the Apply form, so you can apply again.
If an admin sent you an invite link, open it instead. The form has your email filled in. **Set up my account** approves you at once and opens the page where you set your password. An invite link works once, for 7 days.
## Set your password [#set-your-password]
1. In the approval email, select **Choose your password**. The link works once, for 7 days.
2. The App opens **Set your password**. Enter a **Password** of at least 8 characters.
3. Select **Set the password**. The App shows "Your password is set."
4. Select **Sign in**.
Only your newest password link works. A newer link, from a second approval email or a reset, ends the older ones. If the link is too old or used, the page says "The link did not work". Select **Ask for a new link**, enter your **Email**, and select **Send the link**. You have no password yet, so the email is "Set your password", and its link works for 1 hour.
## Sign in [#sign-in]
1. Open [the App's sign-in page](https://v2.infra.acsresearch.org/sign-in).
2. Enter your **Email** and **Password**, and select **Sign in**.
Until you have an API key, the App opens **API Keys** after sign-in. After that, it opens the **Playground**, or the page that you came from.
When the App shows **Continue with GitHub**, **Continue with Google**, or **Continue with Hugging Face**, you can sign in with that provider instead. The provider must have verified the same email address that you sign in with here:
- Your first sign-in with a provider links it to your User.
- A provider never makes a new User. When no User has the email address that the provider gave, the App says so and links to the Apply form.
- When the provider has not verified that email address, the App says so. Verify it at the provider, then try again.
- ACS Infra keeps none of the provider's tokens, and never calls the provider's API.
After 5 tries to sign in within a minute from one address, the App refuses the next try. It says "Too many tries to sign in." and how many seconds to wait. [Limits](/docs/limits#time-limits) lists the other limits of sign-in.
### Link or unlink a provider [#link-or-unlink-a-provider]
1. Open **Profile** from the menu with your name in the App's navigation.
2. Under **Sign-in methods**, select **Link GitHub**, **Link Google**, or **Link Hugging Face**. The App opens the provider, which sends you back to **Profile**.
3. To take a provider off, select its **Unlink** button, such as **Unlink GitHub**.
**Profile** shows **Sign-in methods** only when the App offers a provider. You cannot unlink your last way to sign in: set a password first, with **Reset your password** on **Profile**. To unlink, your session must be less than a day old, so sign out and sign in again first.
### When your session ends [#when-your-session-ends]
A session lasts until you sign out, or until 30 days pass without a visit to the App. A new password and a suspension also end it. The next page that you open in the App then takes you to sign-in, which says "Your session has ended." After you sign in, the App opens that page again.
## Reset your password [#reset-your-password]
1. Select **Forgot password?** on the sign-in page, or **Reset your password** on **Profile**.
2. Enter your **Email**, and select **Send the link**. The App says "Check your email." for every address, so it never tells whether a User has that address.
3. In the email "Reset your password", select **Choose a new password**. The link works once, for 1 hour.
4. The App opens **Choose a new password**. Enter a **New password** of at least 8 characters, and select **Set the new password**. The App shows "Your password is changed."
5. Select **Sign in**, and sign in with the new password.
A new password signs you out everywhere, on every device. If you have not set a password yet, the email is "Set your password" instead, and the page is **Set your password**.
The same limit of 5 tries a minute from one address holds for asking for a link and for setting a password.
## If your access is suspended [#if-your-access-is-suspended]
An admin can suspend your access, for example when your requests break the platform rules that you agreed to when you applied. While your access is suspended:
- Every request with your API keys gets 401 with the code `account_suspended` and the message `Your access to ACS Infra is suspended. Write to infra@acsresearch.org if you think this is a mistake.`
- Sign-in gets 403 with the same message, and your sessions end.
- The App sends you no password link.
- Your queued Jobs and Loom Nodes fail when their turn comes, and spend no tokens ([Paused keys and suspended access](/docs/concepts/background-work#paused-keys-and-suspended-access)).
If you think the suspension is a mistake, write to `infra@acsresearch.org` from the address that you sign in with. When an admin restores your access, sign in again. Your API keys work again.
# Harvest Activations (/docs/harvest)
A harvest reads the residual stream of a Base model at the Layers that you choose, for each token of many prompts. Inference writes it to files that you download, for example to train a sparse autoencoder or a probe.
Set `ACS_API_BASE` and `ACS_API_KEY` first, as [Set up](/docs/quickstart#set-up) shows. A harvest runs in the background on GPUs of its own, apart from Completions. [Harvests in Limits](/docs/limits#harvests) gives every number on this page. [Download a harvest from a shell script](/docs/cookbook/harvest-script) does each step from one script.
## Submit a harvest [#submit-a-harvest]
1. Send the model, the prompts, and the Layers. This harvest reads Layers 8, 16, and 24 of two prompts:
```bash
curl "$ACS_API_BASE/harvest" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}'
```
The API answers 202 with the harvest. The answer has its `job_id`, `"status": "pending"`, the model, `n_prompts`, the Layers that it reads, and `reserved_tokens`. `tokens`, `started_at`, and `completed_at` are `null` until the harvest starts or ends.
2. Put the harvest's `job_id` in a shell variable for the next commands, for example `export JOB_ID=` with the `job_id` from the answer. The Python programs below read it too.
The request takes these fields. An unknown field gets 400.
| Field | What it does |
| --- | --- |
| `model` | The Base model. [Harvests on each model](/docs/models#harvests-on-each-model) lists the models that harvest. |
| `prompts` | A list of texts, each of 1 or more characters. Token ids are not taken. A text that holds half of a surrogate pair, such as a lone `\ud800` in JSON, gets 400. |
| `layers` | A list of Layers in any order, or `"all"`. The files hold each Layer once, in ascending order. Left out, the harvest reads the quartiles of the model's Layers. |
| `shard_size` | The prompts in each file. |
| `add_special_tokens` | `true` by default: the tokenizer adds the model's start token to each prompt. Set `false` for text that already starts with one. |
| `project_onto` | Directions to project the residual stream onto, in place of the stream itself ([Project onto directions](#project-onto-directions)). |
| `batch_size` | Kept in the manifest only. Inference sizes its own batches. |
The API checks the request and the caps when you submit it. If it refuses the harvest, it creates nothing and answers the error object:
- 400 for a field that is not valid. `param` names the field, such as `prompts`, `layers`, or `project_onto`. A Layer that the model does not have gets a message such as `meta-llama/Llama-3.1-8B has layers 0 to 31: it has no layer 40.`
- 400 `harvest_unsupported` for a model that the API serves but does not harvest.
- 400 `model_not_found` for a model that the API does not serve.
- 413 for a request body over the harvest's own size cap.
- 429 `harvest_quota_exceeded`, `harvest_concurrency_exceeded`, or `harvest_capacity_exceeded` when a cap is full. The message names the cap, such as `You have 3 harvests on small models queued or running, your most. Submit another when one ends, or cancel one.`
- 429 `budget_exceeded` when your tokens cannot hold the prompts.
- 429 `spend_cap` when the API has paused new harvests to limit GPU spending. Its `Retry-After` header says when to try again.
A prompt longer than the model's context keeps its first tokens, and the rest is cut. [Harvests in Limits](/docs/limits#harvests) says how many.
## Check on the harvest [#check-on-the-harvest]
Read the harvest. With `wait`, the API answers when the status changes, or after that many seconds:
```bash
curl "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
-H "Authorization: Bearer $ACS_API_KEY"
```
Read it again until its `status` is `done`, `failed`, or `cancelled`. A harvest is in one of these states:
| `status` | Meaning |
| --- | --- |
| `pending` | The harvest waits for the API to start it. |
| `running` | The API holds the prompts' tokens, and Inference runs the harvest on its GPUs. |
| `done` | The files are ready to download. The harvest has its `result`. |
| `failed` | The harvest did not complete. `error` says why ([Handle a failed harvest](#handle-a-failed-harvest-harvest)). |
| `cancelled` | You or an admin canceled the harvest before it ended. |
The times are ISO 8601 in UTC. `created_at` is when you submitted the harvest, and `started_at` is when its GPU run started. `completed_at` is when it was done, failed, or was canceled.
`reserved_tokens` is what the harvest holds while it runs, and `tokens` is what it was charged, once it ended. [Holds of background work](/docs/concepts/budgets#holds-of-background-work) explains both.
When too many reads wait at once ([Harvests in Limits](/docs/limits#harvests)), the API answers at once without waiting. That answer has the headers `X-Acs-Longpoll: declined` and `Retry-After: 3`. Wait 3 seconds, then read again.
## Download the files [#download-the-files]
1. Wait until the harvest's `status` is `done`.
2. Read the harvest:
```bash
curl "$ACS_API_BASE/harvest/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
Its `result` has a link to each file, and `urls_expire_at` says when the links stop working:
| Field of `result` | What it is |
| --- | --- |
| `shard_urls` | One link for each shard file, in order. |
| `stats_url` | The link to `stats.safetensors`. |
| `manifest_url` | The link to `manifest.json`. |
| `n_shards`, `layer_indices`, `contents` | How many shards there are, the Layers in them, and `activations` or `projections`. |
| `tokens`, `bytes_written` | The token positions in the files, start tokens included, and their size in bytes. |
| `timings` | The seconds of the GPU run: `load_s` to load the model, `forward_s` to read the stream, `save_s` to write the files, and `upload_s` to upload them. |
3. Download the shards and read the first one. This program needs `requests`, `safetensors`, and `torch`:
```python
import os
import requests
from safetensors.torch import load_file
API_BASE = os.environ["ACS_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
JOB_ID = os.environ["JOB_ID"]
answer = requests.get(f"{API_BASE}/harvest/{JOB_ID}", headers=HEADERS, timeout=60)
answer.raise_for_status()
result = answer.json()["result"]
for index, url in enumerate(result["shard_urls"]):
with requests.get(url, stream=True, timeout=600) as shard:
shard.raise_for_status()
with open(f"shard_{index:05d}.safetensors", "wb") as file:
for chunk in shard.iter_content(chunk_size=1 << 20):
file.write(chunk)
tensors = load_file("shard_00000.safetensors")
print(tensors["prompt_0"].dtype, tuple(tensors["prompt_0"].shape))
print(tensors["tokens_0"].tolist())
```
The program prints the type and shape of the first prompt's tensor, then its token ids.
A link needs no API key, and works until `urls_expire_at`. Each answer of `GET /v1/harvest/{job_id}` for a `done` harvest makes new links, and the API keeps the files. [Kept data](/docs/limits#kept-data) gives the times.
Anyone who has a link can download that file until the link expires. Keep the
links as private as your API key, and read the harvest again for new ones
instead of sharing them.
A harvest that is not `done` has no `result` and no `urls_expire_at`.
## Read the shards [#read-the-shards]
Each shard holds `shard_size` prompts, in the order of `prompts`. For the prompt at index `i` of your `prompts`, a shard has two tensors:
| Tensor | Type and shape |
| --- | --- |
| `prompt_` | `bfloat16`, `[Layers, tokens, hidden]`. With `project_onto`, `float32`, `[Layers, tokens, directions]`. |
| `tokens_` | `int32`, `[tokens]`: the token ids, in the order of the token axis of `prompt_`. |
The Layer axis follows `layer_indices`, in ascending order. Layer k is the output of decoder block k, before the model's final norm, as for a Completion ([The residual stream](/docs/concepts/residual-stream)). The last Layer is before the final norm too. `tokens_` starts with the start token when `add_special_tokens` is `true`. The token ids let you match each row to its token without the tokenizer.
## Avoid a second start token [#avoid-a-second-start-token]
The tokenizer adds the model's start token, such as `<|begin_of_text|>`, in front of each prompt. If your text already starts with that token, the prompt gets two, and the API does not warn you. Then `tokens_` is one longer than your own tokenization and repeats its first id.
Send `"add_special_tokens": false` for text that already starts with the start token. The tokenizer then reads each prompt as it is.
## Read the manifest and the statistics [#read-the-manifest-and-the-statistics]
`manifest.json` describes the whole harvest. Inference writes it last, so a harvest whose manifest exists has every file. It has the old platform's file layout, version 2, so programs that read those files read these too. Its main keys:
| Key | What it is |
| --- | --- |
| `run_id` | The harvest's `job_id`. |
| `layout_version` | `2`. |
| `model_id`, `model_identity` | The model, and the Hugging Face revision of the weights that ran. |
| `runtime` | The versions and settings that ran the model. |
| `dtype`, `contents` | `bfloat16` and `activations`, or `float32` and `projections`. |
| `add_special_tokens`, `layer_indices`, `residual_stream_convention` | The request's settings, the Layers, and what Layer k means. |
| `prompts` | The text of every prompt, in order. |
| `shards` | For each shard, its file, the indexes of its prompts, and the name and shape of each tensor. |
| `mean_token_norm` | For each Layer, the mean length of a token's vector. |
`stats.safetensors` holds `mean` and `std` (`float32`, `[Layers, hidden]`), and `layer_indices`. The statistics count every token position but the first of each prompt. With `add_special_tokens: true`, that first position is the start token, whose vectors are outliers.
This program reads both. It needs `requests`, `safetensors`, and `torch`:
```python
import os
import requests
from safetensors.torch import load
API_BASE = os.environ["ACS_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
JOB_ID = os.environ["JOB_ID"]
answer = requests.get(f"{API_BASE}/harvest/{JOB_ID}", headers=HEADERS, timeout=60)
answer.raise_for_status()
result = answer.json()["result"]
manifest = requests.get(result["manifest_url"], timeout=60).json()
print(manifest["layer_indices"], manifest["residual_stream_convention"])
stats = load(requests.get(result["stats_url"], timeout=60).content)
print(stats["mean"].shape, stats["std"].shape)
```
The manifest holds your prompts' text, so keep it as private as the prompts.
## Project onto directions [#project-onto-directions]
With `project_onto`, Inference writes the residual stream projected onto your directions, in place of the stream itself. The files are then much smaller. Inference scales each direction to length 1 first. Each direction has the model's `capabilities.hidden_size`, from [List the models](/docs/models#list-the-models).
1. Put your directions, such as a probe's weights, in a 2-D array of `[directions, hidden_size]`.
2. Send its raw bytes, little-endian, in base64 in `data`, with `dtype` and `shape`. This program submits 8 random directions of `llama-8b`:
```python
import base64
import os
import numpy as np
import requests
API_BASE = os.environ["ACS_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
directions = np.random.default_rng(0).standard_normal((8, 4096)).astype("` is then `float32`, `[Layers, tokens, directions]`. The manifest has `contents: "projections"` and a `projection` key, and the statistics describe the projections.
The API refuses directions that do not fit before any GPU starts. Directions of another hidden size get 400 with `param` `project_onto`. These get 400 too, each with its message:
| Directions | Message |
| --- | --- |
| Bytes that do not match the shape | `project_onto.data is 8 bytes; the shape 1 × 3 of float32 needs 12.` |
| A number that is not finite | `project_onto has numbers that are not finite.` |
| A direction that is all zeros | `project_onto has a direction whose length is 0.` |
| A character in `data` that is not base64, a space, a tab, or a line break | `project_onto.data is not valid base64.` |
## Cancel a harvest [#cancel-a-harvest]
You can cancel a harvest that is `pending` or `running`, as it is while Inference loads the model.
1. Submit the harvest:
```bash
curl "$ACS_API_BASE/harvest" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompts": ["The sky is"], "layers": "all"}'
```
2. Put the new `job_id` in `JOB_ID`, then cancel the harvest:
```bash
curl -X DELETE "$ACS_API_BASE/harvest/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The API answers 200 with the harvest, `"status": "cancelled"`.
The API stops the harvest's GPU run, also one that is still starting. A harvest that you cancel while it is `running` and its GPU run started is charged the tokens that it holds, as `tokens` shows. Any other cancel costs nothing:
- Before its GPU run starts.
- While it waits `pending` again after an API restart.
- After the API process that ran it stopped without warning.
A second cancel answers the canceled harvest again. A cancel of a `done` or `failed` harvest gets 409 `harvest_not_cancellable`, and a `job_id` that you do not have gets 404 `harvest_job_not_found`.
## Handle a failed harvest [#handle-a-failed-harvest]
A failed harvest is charged no tokens. Its `error` is a message that says why:
| Why it failed | What to do |
| --- | --- |
| Inference did not start the harvest. It does not count toward your harvests of the month. | Submit it again later. |
| The GPU run did not end within its time limit ([Time limits](/docs/limits#time-limits)). | Split the prompts into smaller harvests, or read fewer Layers. |
| Inference no longer had the result of the GPU run. | Submit it again. |
| Inference could not confirm that the model's weights are the pinned revision, so it ran nothing. | Submit it again later, and [report it](/docs/support#report-a-bug) if it repeats. |
| Inference refused a value of the request. | Fix what `error` names. |
| When the harvest started, your tokens could no longer hold its prompts. | [Ask for more tokens](/docs/support#ask-for-more-tokens), or submit fewer prompts. |
| The API key was revoked, or your access was suspended, before the harvest started. | Submit it with a key that works. |
| Another failure. | [Report it](/docs/support#report-a-bug) with the harvest's `job_id`. |
If the API restarts while your harvest runs, the harvest goes back to `pending`, and its GPU run goes on. The API then follows the same GPU run, and charges the tokens once. [When the API restarts](/docs/concepts/background-work#when-the-api-restarts) says how.
The [API reference](/docs/reference/harvest/harvest.createHarvest) lists every field of each harvest route.
# Overview (/docs)
ACS Infra serves Base models for research. You send text or token ids through an OpenAI-shaped API or the App, and get each token of the continuation. You can read its log probability and the model's Activations, and add Steering vectors to change what it writes.
To start, [get access](/docs/get-access), then send your first prompt with the [Quickstart](/docs/quickstart).
## Find a page by what you need [#find-a-page-by-what-you-need]
| You want to | Read |
| --- | --- |
| Learn the API by sending real requests | The [Quickstart](/docs/quickstart), then the [Tutorials](/docs/tutorials) |
| Get one task done | The guides, such as [Sample continuations](/docs/sampling) and [Use the App](/docs/app), and the [Cookbook](/docs/cookbook) for whole programs |
| Understand why the API behaves as it does | The Concepts, from [Base models](/docs/concepts/base-models) to [Work in the background](/docs/concepts/background-work) |
| Look up a field, an error, or a limit | [Completions](/docs/completions), [Errors](/docs/errors), [Limits](/docs/limits), and the [API reference](/docs/reference/completions/completions.createCompletion) |
The [Changelog](/docs/changelog) lists what changed in the API and the App. [Support](/docs/support) says how to get help and report a bug.
Send one prompt with curl and with Python, and read its tokens.
Read the residual stream and steer a Base model with a Steering vector.
Run prompts in the Playground, grow a Loom, and read your usage.
Every limit on tokens, traffic, Jobs, harvests, and Looms in one place.
## Read the docs with an AI assistant [#read-the-docs-with-an-ai-assistant]
Each page is also Markdown at its address with `.md` at the end. [`/llms.txt`](/llms.txt) lists every page, and [`/llms-full.txt`](/llms-full.txt) has all of them in one Markdown file, for an assistant to read.
# Run a Job (/docs/jobs)
A Job runs one Completion request on the server. The API runs it to the end, also when your script stops or your connection drops, and keeps the result for you to fetch. Set `ACS_API_BASE` and `ACS_API_KEY` first, as [Set up](/docs/quickstart#set-up) shows.
The App's **Jobs** page leads to [Run a Job from a shell script](/docs/cookbook/job-script), a recipe that does each step below from one script.
## Submit a Job [#submit-a-job]
1. Send a body of `POST /v1/completions` without `stream`, or with `"stream": false`, and an `Idempotency-Key` of your choice:
```bash
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: sky-001" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'
```
The API answers 202 with the queued Job, and its `Location` header is the Job's `status_url`:
```json title="Job"
{
"cancel_requested": false,
"created_at": "2026-11-02T18:35:05.226+00:00",
"expires_at": null,
"finished_at": null,
"id": "01a19d8b-2892-7afa-8b51-35c9fc2f5572",
"model": "llama-8b",
"request_id": "req_job_01a19d8b28927afa8b5135c9fc2f5572",
"result_expired": false,
"result_url": "/v1/completion-jobs/01a19d8b-2892-7afa-8b51-35c9fc2f5572/result",
"started_at": null,
"status": "queued",
"status_url": "/v1/completion-jobs/01a19d8b-2892-7afa-8b51-35c9fc2f5572",
"usage_pending": false
}
```
2. Put the Job's `id` in a shell variable for the next commands, for example `JOB_ID=01a19d8b-2892-7afa-8b51-35c9fc2f5572`.
To resume after a lost connection, send the same request with the same `Idempotency-Key`. The API answers 200 with the Job that the key made, as it is now, and runs nothing again. The same key with another request answers 409 `idempotency_conflict`. Use a new key for each piece of work.
The API checks and admits the request when you submit the Job. If it refuses the request, it creates no Job and answers the error object, as `POST /v1/completions` would:
- 400 for a field that is not valid. `param` names the field, such as `stream` for `"stream": true`.
- 400 `model_not_found` for a model that the API does not serve, with `param` `model`.
- 429 `budget_exceeded` when your tokens cannot hold the request.
- 429 `job_capacity` when you have too many Jobs. [Jobs in Limits](/docs/limits#jobs) lists the caps.
- 429 `spend_cap` while new GPU work is paused ([Spend limit](/docs/limits#spend-limit)).
## Check on the Job [#check-on-the-job]
Read the Job:
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Job"
{
"id": "01a19d8b-2892-7afa-8b51-35c9fc2f5572",
"request_id": "req_job_01a19d8b28927afa8b5135c9fc2f5572",
"model": "llama-8b",
"status": "succeeded",
"cancel_requested": false,
"created_at": "2026-11-02T18:35:05.226+00:00",
"started_at": "2026-11-02T18:35:05.245+00:00",
"finished_at": "2026-11-02T18:35:05.287+00:00",
"expires_at": "2026-12-02T18:35:05.287+00:00",
"result_expired": false,
"status_url": "/v1/completion-jobs/01a19d8b-2892-7afa-8b51-35c9fc2f5572",
"result_url": "/v1/completion-jobs/01a19d8b-2892-7afa-8b51-35c9fc2f5572/result",
"usage_pending": false
}
```
Read the Job again until its `status` is `succeeded`, `failed`, or `cancelled`. A 200 from this route means that the API read the Job. It does not mean that the Completion succeeded. A Job is in one of these states:
| `status` | Meaning |
| --- | --- |
| `queued` | The Job waits to run. |
| `running` | The API sends the request to Inference. `usage_pending` is `true`, as the request holds tokens of your Budget. |
| `succeeded` | The Completion answered 200. Its result is ready. |
| `failed` | The Completion answered an error, or the API failed to keep its result. |
| `cancelled` | You canceled the Job before its Completion ended. |
The times are ISO 8601 in UTC. `created_at` is when you submitted the Job, `started_at` is when the API first started it, and `finished_at` is when it ended. `expires_at` is when its result expires, 30 days after it ended, and `null` until then.
A Job can wait `queued` while other Jobs run. [The order of the work](/docs/concepts/background-work#the-order-of-the-work) says which Job the API starts next.
## Fetch the result [#fetch-the-result]
Fetch the result from the Job's `result_url`:
```bash
curl -i "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The result has the status and the body that `POST /v1/completions` would have answered: 200 and the Completion, or the error object with its own status. When the Completion failed, its error has `request_id` beside `error`: the Job's `request_id`, not the `X-Request-Id` of this answer. Check the status before you read `choices`. The result keeps the status and the body, not the headers of the answer, such as `Retry-After`. The route answers these statuses:
| Status | When |
| --- | --- |
| 200 | The Completion succeeded. The body is the Completion. |
| 202 | The Job is `queued` or `running`. The body is the Job, and `Retry-After: 1` asks you to try again in a second. |
| 400, 429, 502, 503, or 504 | The Completion failed with this status, as `POST /v1/completions` answers it. For example, 429 `budget_exceeded` when your token budget could not hold the request when it ran. |
| 409 `job_cancelled` | You canceled the Job. |
| 410 `job_result_expired` | The result expired, 30 days after the Job ended. The Job itself stays. |
| 500 `job_failed` | The API failed to run the Job or to keep its result. It does not run the Job again. |
The Job takes its Hold when it runs, not when you submit it, as [Holds of background work](/docs/concepts/budgets#holds-of-background-work) explains.
## Ask for Activations in a Job [#ask-for-activations-in-a-job]
A Job can ask for [Activations](/docs/activations) and add [Steering vectors](/docs/steering), as a request to `POST /v1/completions` can. Its result then has `activations`.
1. Submit a Job that reads Layers 0 and 2:
```bash
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: sky-layers-001" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "output_residual_stream": [0, 2]}'
```
2. Put the new `id` in `JOB_ID`, and wait until the Job is `succeeded`.
3. Read the shape of the result's residual stream with `jq`:
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY" | jq -c '.activations.residual_stream.shape'
```
```text title="Shape"
[2,5,4096]
```
4. Decode the result's `activations` as [Decode the tensor](/docs/activations#decode-the-tensor) shows.
## Cancel a Job [#cancel-a-job]
You can cancel a Job that is `queued` or `running`. This Job asks for 2,000 tokens, so it is still running when you cancel it.
1. Submit the Job:
```bash
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: sky-long-001" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2000}'
```
2. Put the new `id` in `JOB_ID`, then cancel the Job:
```bash
curl -X POST "$ACS_API_BASE/completion-jobs/$JOB_ID/cancel" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Job"
{
"id": "01a19d8b-3294-749c-a4f1-ae24ecf281f4",
"request_id": "req_job_01a19d8b3294749ca4f1ae24ecf281f4",
"model": "llama-8b",
"status": "cancelled",
"cancel_requested": true,
"created_at": "2026-11-02T18:35:07.784+00:00",
"started_at": "2026-11-02T18:35:07.820+00:00",
"finished_at": "2026-11-02T18:35:07.859+00:00",
"expires_at": "2026-12-02T18:35:07.859+00:00",
"result_expired": false,
"status_url": "/v1/completion-jobs/01a19d8b-3294-749c-a4f1-ae24ecf281f4",
"result_url": "/v1/completion-jobs/01a19d8b-3294-749c-a4f1-ae24ecf281f4/result",
"usage_pending": false
}
```
The API stops the request that runs, and its result is then 409 `job_cancelled`. A cancel of a Job that has ended changes nothing, and answers the Job as it is.
If the API restarts while your Job runs, the Job runs again later, and its request still spends tokens once. [When the API restarts](/docs/concepts/background-work#when-the-api-restarts) says how.
The [API reference](/docs/reference/completionJobs/completionJobs.createCompletionJob) lists every field of each route of a Job.
# Limits (/docs/limits)
The API and the App limit what one request, one API key, and one User can do. Each table gives the limit, its number, and what the API or the App answers past it. An error with a code is the error object, as [The error object](/docs/errors#the-error-object) shows.
## Tokens [#tokens]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Monthly allowance of a User | 5,000,000 tokens each UTC month, by default. An admin can set another for one User. | 429 `budget_exceeded`. Near the end, `POST /v1/completions` first lowers `max_tokens` to what fits. |
| Generated tokens of one request | 32,000 in all: the number of prompts × `n` × `max_tokens` | 400, `param` `max_tokens` |
| `max_tokens` of a request without `stream` | 15,600. A request without `max_tokens` gets 15,600 at most. | 400 `deadline_infeasible` |
| Prompt and generated tokens of one request | The model's `max_model_len`: 131,072 for `llama-8b` and `llama-405b`, 8,192 for `trinity-truebase` | 400 `context_length_exceeded` |
Your monthly allowance is spent first, then your Budget. [Budgets and Holds](/docs/concepts/budgets) says how a request holds and spends tokens, and when the API lowers `max_tokens`.
The other caps of one Completion, such as `logprobs` and the size of a capture, are in [Size caps](/docs/completions#size-caps). `GET /v1/models` lists them for each model in `limits` ([Models](/docs/models)).
## Traffic [#traffic]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Requests that start work, for each API key | 600 a minute: `POST /v1/completions`, `POST /v1/completion-jobs`, `POST /v1/harvest`, and Branches | 429 `rate_limited`, with `Retry-After` in seconds |
| Other requests to `/v1`, for each API key | 600 a minute | 429 `rate_limited`, with `Retry-After` in seconds |
| Completions that run at once, for each API key | 16 | A Completion waits in the key's line. |
| Completions that wait in the line, for each API key | 64 | 429 `queue_full`, with `Retry-After: 5` |
| Requests in flight to one model, from all keys and the App | 100 | 503 with `Retry-After: 5` for `POST /v1/completions`. A Job, a Run, and a Branch wait for a place. |
| Playground Runs, Completions, Jobs, and Branches that you start in the App | 600 a minute for each User | The App says when to try again. |
| Requests to `/v1` with a missing, wrong, or revoked API key, from one address | 60 a minute | 429 `rate_limited`, with `Retry-After` in seconds |
Each rate limit is a bucket. A full bucket takes a burst of 600 requests, then refills one request every 0.1 seconds. Each API key has its own buckets, and the App's limit is apart from them. The message of a 429 names the limit and says how many seconds to wait.
The line counts only `POST /v1/completions`. A Completion takes its place when it starts and keeps it until it ends, also a stream. A waiting Completion holds no tokens and runs in the order it came. It leaves the line when you close the connection. Jobs, harvests, and Loom Nodes run in the background and take no place.
The `openai` SDK sends a request that got 429 `rate_limited` again by itself. [Retries in the `openai` SDK](/docs/retries#retries-in-the-openai-sdk) says how.
## API keys [#api-keys]
| Limit | Number | Past the limit |
| --- | --- | --- |
| API keys of one User | 10. A paused key counts, and a revoked key does not. | 409 when you make a key. At the limit, the App turns off **New key** and shows `You have 10 API keys, the most that a User may have. Delete one to make another.` |
## Jobs [#jobs]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Requests in one Job | 1 | A body is one request of `POST /v1/completions`. |
| Jobs `queued` or `running` with one API key | 8 | 429 `job_capacity`, with `x-should-retry: false` |
| Jobs `queued` or `running` of one User | 32 | 429 `job_capacity`, with `x-should-retry: false` |
| Jobs `queued` or `running` of all Users | 128 | 429 `job_capacity`, with `x-should-retry: false` |
| Ended Jobs of one User whose results the API keeps | 100 | 429 `job_capacity`, with `x-should-retry: false` |
| Ended Jobs of all Users whose results the API keeps | 1,000 | 429 `job_capacity`, with `x-should-retry: false` |
| Jobs that run at once | 4, for all Users | A Job waits `queued`. |
A Job past a cap is not created, and the message names the cap. A cap of Jobs `queued` or `running` frees a place when a Job ends. A cap of kept results frees one when the result of a Job expires ([Kept data](#kept-data)).
## Harvests [#harvests]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Prompts in one harvest | 1 to 4,096, each of 1 or more characters | 400, `param` `prompts` |
| Tokens of one harvest's prompts, counted as characters / 4 | 2,000,000 | 400, `param` `prompts` |
| Tokens of one prompt in the files | The model's `max_model_len` - 1: 8,191 on each served model | The harvest keeps the first 8,191 tokens of the prompt. |
| Layers in a list in `layers` | 1 to 256, each a Layer of the model | 400, `param` `layers` |
| Prompts in one shard file (`shard_size`) | 1 to 4,096, and 32 when you leave it out | 400, `param` `shard_size` |
| `batch_size` | 1 to 64, or `null` | 400, `param` `batch_size` |
| Directions in `project_onto` | 1 to 64, each of the model's `capabilities.hidden_size` | 400, `param` `project_onto` |
| Harvests that one API key starts in a UTC month | 200. An admin can set another number for your keys. | 429 `harvest_quota_exceeded`, with `x-should-retry: false` |
| Harvests on `llama-8b` `pending` or `running` of one User | 3 | 429 `harvest_concurrency_exceeded`, with `Retry-After: 30` |
| Harvests on `llama-405b` or `trinity-truebase` `pending` or `running` of one User | 1 | 429 `harvest_concurrency_exceeded`, with `Retry-After: 30` |
| Harvests on `llama-405b` or `trinity-truebase` `pending` or `running` of all Users | 2 | 429 `harvest_capacity_exceeded`, with `Retry-After: 30` |
| Reads of a harvest that wait (`wait`) at once | 4 for each API key, and 8 in all | The API answers at once with 200, `X-Acs-Longpoll: declined`, and `Retry-After: 3`. |
A harvest past a cap is not created, and the message names the cap. A cap of harvests `pending` or `running` frees a place when one of them ends. The count of the month leaves out a harvest whose GPU run never started, such as one that you canceled while it was `pending`. It starts again on the first day of each UTC month.
Without `layers`, a harvest reads the quartiles of the model's Layers: `capabilities.n_layers` / 4, / 2, and × 3 / 4. That is Layers 8, 16, and 24 on `llama-8b`. The cap of 2,000,000 counts characters only. The tokens that a harvest holds are counted another way, as [Holds of background work](/docs/concepts/budgets#holds-of-background-work) says. [Harvest Activations](/docs/harvest) shows each refusal.
## Looms [#looms]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Child Nodes of one Branch | 1 to 8 (`n`) | 400, `param` `n` |
| Tokens of one Node | 1,000 (`max_tokens`) | 400, `param` `max_tokens` |
| Nodes of one Loom | 2,000 | 409 for a Branch, a split, or a sibling. 400 for an imported file. |
| Looms in the list | 100: the pinned ones first, then the others by their last change | An older Loom stays, and a read of its id finds it. |
| Nodes that run at once | 8, for all Users | A Node waits `queued`. |
| Characters of a Loom's title | 120 | The API keeps the first 120 characters. |
| Characters of a title from the root | 40, from the root's first line | The API keeps the first 40 characters. |
The 409 for a full Loom says how many more Nodes fit. A Loom's Nodes include each Node that you have not deleted, at any depth.
## Spend limit [#spend-limit]
The API pauses new GPU work when the GPUs cost $3,000 in 24 hours, and an admin can pause it too. Only an admin lifts the pause.
| While the pause is on | What happens |
| --- | --- |
| `POST /v1/completions`, `POST /v1/completion-jobs`, `POST /v1/harvest`, a Branch, and the App's Runs | 429 `spend_cap`, with `Retry-After: 3600` and `x-should-retry: false` |
| A Completion, a Job, a harvest, or a Node that already runs | It finishes. |
| A Job or a Node that is `queued`, or a harvest that is `pending` | It waits, and runs when the pause ends. |
| A Job, a harvest, or a Node that the API puts back in the queue during the pause, as at a restart | It waits too ([When the API restarts](/docs/concepts/background-work#when-the-api-restarts)). |
## Request size [#request-size]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Body of one request, on every route but `POST /v1/harvest` | 8 MiB (8,388,608 bytes) | 413. A body sent in chunks, without `Content-Length`, gets 413 when it passes the cap. The API checks the key first. |
| Body of `POST /v1/harvest` | 50 MiB (52,428,800 bytes) | 413. A body sent in chunks, without `Content-Length`, gets 413 when it passes the cap. The API checks the key first. |
| `Idempotency-Key` | 1 to 255 characters | 400 |
| Fields of one Completion | The caps in [Size caps](/docs/completions#size-caps) | 400, `param` names the field |
## Time limits [#time-limits]
| Limit | Number | Past the limit |
| --- | --- | --- |
| Wait for Inference | 14 minutes for an answer, and 14 minutes between two chunks of a stream | 504 `upstream_deadline`, `Could not reach Inference: Request timed out.`, with `x-should-retry: false`. In a stream, that error is the last event. |
| Wait for Inference's answer to start, for a Job or the App's Runs and Branches | 45 minutes, through a start of a cold model | The request fails with `Could not reach Inference: Request timed out.` |
| Wait for the answer of a Completion without a stream | 150 seconds | The answer starts: 200, then a space every 5 seconds until the Completion's JSON. |
| Tries to sign in, ask for a reset email, or set a new password, from one address | 5 a minute | The App says how many seconds to wait. |
| Password link of the approval email | 7 days, used once. Only the newest link works. | The link no longer works. |
| Password reset link | 1 hour, used once | The link no longer works. |
| Session in the App | 30 days from its last renewal. The App renews it at most once a day while you use it. | The App takes you to sign-in. |
| GPU run of one harvest | 60 minutes on `llama-8b`, and 150 minutes on `llama-405b` and `trinity-truebase` | The harvest fails, its `error` says that it timed out, and it costs nothing. |
| `wait` of a read of a harvest | 0 to 60 seconds | 400 |
[Cold starts and long requests](/docs/concepts/long-requests) explains the 14 minutes, the 45 minutes and the 150 seconds.
## Kept data [#kept-data]
| Data | Kept | After that |
| --- | --- | --- |
| Result of an ended Job | 30 days after the Job ended | The result route answers 410 `job_result_expired`. The Job stays. |
| Activations of a Playground Run | 7 days after the Run ended | The Run shows its text and tokens without them. |
| A deleted Loom, Node, or Playground Run | 14 days after the delete | The API erases it. You cannot read it from the moment you delete it. |
| An `Idempotency-Key` and its answer | 24 hours | A request with the same key runs as a new request. |
| Files of a completed harvest | With no end | The API deletes none of them. |
| A link to a file of a harvest | 7 days from the answer that made it | The link no longer works. Each answer of `GET /v1/harvest/{job_id}` for a `done` Job makes new links. |
# Grow a Loom with the API (/docs/loom)
A Loom is a tree of Nodes that you grow from one prompt. Use it to compare the texts that a Base model writes, and to continue the ones you choose. This guide grows a Loom from a script. Set `ACS_API_BASE` and `ACS_API_KEY` first, as [Set up](/docs/quickstart#set-up) shows. To grow one in the browser, use [the App's Loom page](/docs/app/loom).
The root Node holds the prompt. A Branch makes one or more child Nodes under a Node. Each child is one Completion of the text on the path from the root to its parent. The API runs each Branch in the background, so it finishes when no client is connected.
## Make a Loom [#make-a-loom]
1. Send the prompt:
```bash
curl "$ACS_API_BASE/looms" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "The sky is"}'
```
The API answers the Loom with its root Node, which is `completed`:
```json title="Loom"
{
"id": "01a12e70-9370-7a9b-87d6-4c3d2621969e",
"object": "loom",
"created_at": 1791780492,
"title": "",
"pinned": false,
"nodes": [
{
"id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"object": "loom.node",
"parent_id": null,
"status": "completed",
"text": "The sky is",
"logprobs": null,
"error": null,
"request": null
}
]
}
```
2. Put the Loom's `id` in the shell variable `LOOM_ID`, and the root's `id` in `NODE_ID`.
## Branch under a Node [#branch-under-a-node]
1. Ask for two child Nodes under the root, each with at most 2 tokens and the log probability of each token:
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID/nodes/$NODE_ID/branch" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "max_tokens": 2, "n": 2, "logprobs": 1}'
```
The API answers the new Nodes. They are `queued` and have no text yet:
```json title="Branch"
{
"object": "list",
"data": [
{
"id": "01a12e70-97cc-74a1-bb2d-c199187442ef",
"object": "loom.node",
"parent_id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"status": "queued",
"text": "",
"logprobs": null,
"error": null,
"request": {
"model": "meta-llama/Llama-3.1-8B",
"max_tokens": 2,
"temperature": 1,
"logprobs": 1,
"top_p": 1,
"top_k": 0,
"min_p": 0,
"presence_penalty": 0,
"frequency_penalty": 0,
"repetition_penalty": 1,
"seed": 1739262751,
"stop": null
}
},
{
"id": "01a12e70-97cc-76d3-a601-15fa41388315",
"object": "loom.node",
"parent_id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"status": "queued",
"text": "",
"logprobs": null,
"error": null,
"request": {
"model": "meta-llama/Llama-3.1-8B",
"max_tokens": 2,
"temperature": 1,
"logprobs": 1,
"top_p": 1,
"top_k": 0,
"min_p": 0,
"presence_penalty": 0,
"frequency_penalty": 0,
"repetition_penalty": 1,
"seed": 1739262752,
"stop": null
}
}
]
}
```
2. Follow the new Nodes until they end. [Follow a Branch](#follow-a-branch) shows the two ways.
A Branch takes these fields. Each one has the type and the default of the same field of `POST /v1/completions`, except `max_tokens` and `n`:
| Field | Default | What it does |
| --- | --- | --- |
| `model` | none, required | The model that writes the new Nodes. |
| `n` | `1` | How many child Nodes to make, from 1 to 8. |
| `max_tokens` | `16` | The most tokens of each new Node, at most 1,000. |
| `temperature` | `1` | The sampling temperature. |
| `logprobs` | `null` | The number of top tokens whose log probabilities each Node returns for each of its tokens. |
| `seed` | a new one | Node `i` of the Branch, from 0, samples with `seed + i`. |
| `top_p`, `top_k`, `min_p`, `presence_penalty`, `frequency_penalty`, `repetition_penalty`, `stop` | as in Completions | How to sample each token, and where the text stops. |
[Request fields](/docs/completions#request-fields) says what each sampling field does. Without a `seed`, the API draws one from 0 to 2,147,483,647 for the Branch. With `n` Nodes, the last Node samples with `seed + n - 1`. That number must be at most 9,007,199,254,740,991, the largest integer that JSON and JavaScript hold exactly.
The API checks the Branch when you make it, and makes no Node when it refuses one:
- 400 `invalid_request` for a field that the table does not list.
- 400 for a value that Completions refuses, such as a `top_p` of 0 ([Strict validation](/docs/completions#strict-validation)).
- 400 `model_not_found` for a model that the API does not serve.
- 409 when the Node is `queued`, `running`, or `failed`. Branch only under a `completed` or `cancelled` Node.
- 429 `budget_exceeded` when your tokens cannot hold one Node.
- 429 `spend_cap` while new GPU work is paused ([Spend limit](/docs/limits#spend-limit)).
Each Node takes its own Hold when it runs, and its prompt is the whole text of its path. [Holds of background work](/docs/concepts/budgets#holds-of-background-work) says what that costs.
### Sample a Node again [#sample-a-node-again]
Each child Node keeps its `request`: the Branch fields that it ran with, its own seed included. To sample a Node again, send its `request` as a Branch under the same parent. With the same `seed`, the new Node gets the same tokens, but vLLM can still change a token now and then when it batches requests.
The root's `request` is `null`. So is the `request` of a Node that you wrote as a sibling or a new root. A Node whose fields no longer read as Branch fields has none either. A Node that ran before a range narrowed keeps the value that it ran with, such as a `temperature` of 5. A Branch with that value gets 400 `invalid_request`.
## Follow a Branch [#follow-a-branch]
1. Read the Loom:
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
2. Read it again until each Node of the Branch is `completed`, `failed`, or `cancelled`.
When both new Nodes are completed, the Loom shows their text:
```json title="Loom"
{
"id": "01a12e70-9370-7a9b-87d6-4c3d2621969e",
"object": "loom",
"created_at": 1791780492,
"title": "The sky is",
"pinned": false,
"nodes": [
{
"id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"object": "loom.node",
"parent_id": null,
"status": "completed",
"text": "The sky is",
"logprobs": null,
"error": null,
"request": null
},
{
"id": "01a12e70-97cc-74a1-bb2d-c199187442ef",
"object": "loom.node",
"parent_id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"status": "completed",
"text": " crying,",
"logprobs": {
"text_offset": [0, 7],
"token_logprobs": [-4.989488124847412, -1.722730040550232],
"tokens": [" crying", ","],
"top_logprobs": [
{ " the": -1.4269880056381226, " crying": -4.989488124847412 },
{ ",": -1.722730040550232 }
]
},
"error": null,
"request": {
"model": "meta-llama/Llama-3.1-8B",
"max_tokens": 2,
"temperature": 1,
"logprobs": 1,
"top_p": 1,
"top_k": 0,
"min_p": 0,
"presence_penalty": 0,
"frequency_penalty": 0,
"repetition_penalty": 1,
"seed": 1739262751,
"stop": null
}
},
{
"id": "01a12e70-97cc-76d3-a601-15fa41388315",
"object": "loom.node",
"parent_id": "01a12e70-9373-7799-8f05-0f6104866ad3",
"status": "completed",
"text": " the colour",
"logprobs": {
"text_offset": [0, 4],
"token_logprobs": [-1.4269880056381226, -5.764825344085693],
"tokens": [" the", " colour"],
"top_logprobs": [
{ " the": -1.4269880056381226 },
{ " limit": -0.07732537388801575, " colour": -5.764825344085693 }
]
},
"error": null,
"request": {
"model": "meta-llama/Llama-3.1-8B",
"max_tokens": 2,
"temperature": 1,
"logprobs": 1,
"top_p": 1,
"top_k": 0,
"min_p": 0,
"presence_penalty": 0,
"frequency_penalty": 0,
"repetition_penalty": 1,
"seed": 1739262752,
"stop": null
}
}
]
}
```
The root comes first. Each Node comes after its parent and before its next sibling, and siblings come in the order they were made. A Node's `text` is only its own part. The text of a path is the text of its Nodes from the root down, so a Branch under the second Node continues `"The sky is the colour"`.
The model samples each Node on its own, so with a `temperature` above 0 the two Nodes have different texts. The first Branch also names the Loom: its `title` is now the first line of the root.
A Node is in one of these states:
| `status` | Meaning |
| --- | --- |
| `queued` | The Node waits to run. |
| `running` | Inference writes the Node. `text` holds the tokens so far, and grows. |
| `completed` | The text is complete. With `logprobs` in the Branch, `logprobs` is set. |
| `failed` | The Completion failed. `text` holds the tokens before the failure, and `error` says why. |
| `cancelled` | You stopped the Node while it grew. `text` holds the tokens it grew. |
A Node can wait `queued` while other Nodes run. [Looms in Limits](/docs/limits#looms) says how many Nodes run at once.
### Follow a Loom as a stream [#follow-a-loom-as-a-stream]
To get each change as it happens instead of reading the Loom again, open the Loom's stream:
```bash
curl -N "$ACS_API_BASE/looms/$LOOM_ID/stream" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The stream is Server-Sent Events. Each event is one line that holds `data: ` and a Node object, the same object that a read of the Loom shows:
1. First the API sends each Node of the Loom, in the order of a read.
2. Then, while a Node is `queued` or `running`, the API reads the Loom every 250 ms and sends each Node whose `status` or `text` changed. A running Node's `text` is its whole text so far.
3. `data: [DONE]` ends the stream when no Node is `queued` or `running`.
When the Branch above has ended, the stream sends the three Nodes of [Follow a Branch](#follow-a-branch) and `data: [DONE]` at once.
A stream sees each Branch that starts before the stream ends, also a Branch that another client starts. Open the stream again for a Branch that starts after it ended. If a read of the Loom fails after the stream started, the last event before `[DONE]` is the error object.
## Change the tree [#change-the-tree]
You can write a Loom's text yourself. Each change answers the whole Loom.
| Change | Route | Body |
| --- | --- | --- |
| Edit a Node | `POST /v1/looms/{loom_id}/nodes/{node_id}` | `{"text": "..."}` |
| Split a Node | `POST /v1/looms/{loom_id}/nodes/{node_id}/split` | `{"offset": 5}`, or with `"text"` |
| Add a sibling | `POST /v1/looms/{loom_id}/nodes/{node_id}/sibling` | `{"text": "..."}` |
| Set a new root | `POST /v1/looms/{loom_id}/root` | `{"text": "..."}` |
The API stores a CRLF or a lone CR in your text as LF.
An edit rewrites the Node's text and drops its log probabilities. The same text changes nothing. The Nodes under the edited Node keep their own text, so the text of their paths changes with it.
A split cuts the Node at `offset` characters, counted in Unicode code points. An offset past the end cuts at the end. The Node keeps the text before the cut. A new child gets the text after it, and the Node's children move under that child, so every path keeps its text.
- A split of the Node's own text gives each half the log probabilities of the whole tokens on its side. A token across the cut leaves the new child none.
- With `text`, the API cuts that edited text instead, and neither half keeps log probabilities.
- The new child's `request` has `seed: null`, because part of a sample cannot be sampled again.
A sibling is a Node that you write under the same parent as this Node. A new root replaces the whole tree with one root, your text.
For example, split a Node after its first 7 characters:
1. Put the id of the Node to split in `NODE_ID`, such as the first new Node of [Follow a Branch](#follow-a-branch).
2. Split it:
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID/nodes/$NODE_ID/split" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"offset": 7}'
```
For the Node `" crying,"`, the Node keeps `" crying"` and the log probability of that token. Its new child holds `","` and the log probability of `","`.
The API answers 409 and changes nothing in these cases:
- An edit or a split of a `failed` Node, whose text is what came before the failure.
- A change while a Node that the change touches is `queued` or `running`. For an edit or a split, that is a Node under the changed Node. For a sibling, it is a Node under the parent. For a new root, it is any Node of the Loom. Stop the Branch, or wait until it ends.
- A sibling of the root. A root has no sibling.
- A Branch, a split, or a sibling that would give the Loom more Nodes than it holds ([Looms in Limits](/docs/limits#looms)). The message says how many more Nodes fit.
## Stop a Branch [#stop-a-branch]
Stop every Node of the Loom that is `queued` or `running`:
```bash
curl -X POST "$ACS_API_BASE/looms/$LOOM_ID/cancel" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The API answers the Loom. Each Node that grew is now `cancelled`:
- A running Node keeps the text it grew, and you pay for those tokens, as for a stream that you close.
- A queued Node never runs and holds no tokens.
A stop when no Node grows changes nothing. A stopped Node's text is final, so you can grow a Branch under it. A Branch that you grow at once first waits for the API to save the Node's last tokens.
## Read a failed Node [#read-a-failed-node]
A Node fails when its Completion fails after you made the Branch. The Node's `status` is then `failed`, its `text` holds the tokens that came before the failure, and its `error` is the error object. The `error` is what `POST /v1/completions` answers for the same request, without a `request_id`.
| Cause | `error` |
| --- | --- |
| Your tokens cannot hold the Node when it runs. | `code` `budget_exceeded`, and the Node does not reach Inference. |
| The text of the path and `max_tokens` are longer than the model's context. | `code` `context_length_exceeded` |
| Inference refuses a value, or fails. | `message` says why. |
| The API key that made the Branch was revoked, or your access is suspended. | `code` `invalid_api_key`, and the Node does not reach Inference |
The error's `param` names a field of the Branch, or is `null`. A Branch has no `prompt` field, so a path that is too long has `param: null`.
To continue after a failure, grow a new Branch under the failed Node's parent. A failed Node takes no Branch, edit, or split, but you can delete it.
## List your Looms [#list-your-looms]
```bash
curl "$ACS_API_BASE/looms" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The API answers your newest 100 Looms, each with its title and prompt, without its Nodes:
```json title="Looms"
{
"object": "list",
"data": [
{
"id": "01a124ae-b6ad-7b4b-84cf-e62cb48d9f79",
"object": "loom",
"created_at": 1791616792,
"updated_at": 1791616795,
"title": "The sky is",
"pinned": false,
"prompt": "The sky is"
}
]
}
```
Pinned Looms come first, newest pin first. The other Looms follow by their last change, newest first. Each change of a Loom moves it to the top of the unpinned Looms. A Branch, an edit, a split, a sibling, a new root, a deleted Node, and a rename each change it. A pin does not.
A Loom's `title` is empty until its first Branch names it after the first line of its root, cut to 40 characters.
## Rename or pin a Loom [#rename-or-pin-a-loom]
Send a new `title`, `pinned`, or both:
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"title": "Skies", "pinned": true}'
```
The API answers the Loom with its Nodes, with `"title": "Skies"` and `"pinned": true`.
- The API trims the title and keeps its first 120 characters. An empty title leaves the Loom untitled, so its next Branch names it again.
- `pinned: true` puts the Loom above the unpinned ones, and `pinned: false` unpins it. A pin does not change the Loom's last change, so an unpinned Loom goes back to its place.
## Delete a Node or a Loom [#delete-a-node-or-a-loom]
1. Put the id of the Node to delete in `NODE_ID`.
2. Delete the Node and every Node under it:
```bash
curl -X DELETE "$ACS_API_BASE/looms/$LOOM_ID/nodes/$NODE_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
The API answers the Loom without them:
```json title="Loom"
{
"id": "01a124ae-b6ad-7b4b-84cf-e62cb48d9f79",
"object": "loom",
"created_at": 1791616792,
"title": "The sky is",
"pinned": false,
"nodes": [
{
"id": "01a124ae-b6c2-7415-b99b-cddb466c5a5f",
"object": "loom.node",
"parent_id": null,
"status": "completed",
"text": "The sky is",
"logprobs": null,
"error": null,
"request": null
}
]
}
```
To delete the whole Loom, send:
```bash
curl -X DELETE "$ACS_API_BASE/looms/$LOOM_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Deleted"
{
"id": "01a124ae-b6ad-7b4b-84cf-e62cb48d9f79",
"object": "loom",
"deleted": true
}
```
You cannot undo a delete. A deleted Node or Loom is gone for you at once: no read, list, or Branch finds it.
The API refuses with 409:
- A delete of the root Node. Delete the Loom instead.
- A delete while a Node to delete is `queued` or `running`. Delete it when its Branch ends.
## Export and import a Loom [#export-and-import-a-loom]
1. Download a Loom as a file:
```bash
curl "$ACS_API_BASE/looms/$LOOM_ID/export" \
-H "Authorization: Bearer $ACS_API_KEY" -o loom.json
```
2. Make a new Loom from the file. Send it to `POST /v1/looms/import` with the import command at the end of this section, with `-d @loom.json` in place of its `-d '...'` body.
The file has the shape of the Loom export of the old platform (version 2). So a file from either one imports into the other. It holds the Loom's `title`, the `model` and `last_generation_config` of its last Branch, and each Node, each parent before its children. A Node in the file has these fields:
- `id`, `parent_id`, `text`, and `status`.
- `position`, its place among its siblings.
- `logprobs`, one entry for each token, with `token`, `logprob`, and the likeliest tokens in `top`.
- `generation_config`, the Branch fields it ran with. It also holds the marks `text_modified` after an edit and `context_modified` when a Node above it changed. A part of a split sample has `slice_offset` and `slice_end`.
- `error`, for a failed Node.
You can also write a file yourself. This one makes a Loom of two Nodes:
```bash
curl "$ACS_API_BASE/looms/import" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"loom": {"title": "Skies"}, "nodes": [{"id": "a", "parent_id": null, "text": "The sky is"}, {"id": "b", "parent_id": "a", "text": " blue"}]}'
```
The API answers the new Loom. It reads the file by these rules:
- Each Node gets a new id, and siblings keep the order of their `position`.
- A Node without a `status` is `completed`.
- A Node whose parent is not in the file goes under the file's root when the file has one root. Otherwise it becomes a root, so a Loom can hold several roots.
- A Node whose parents lead back to itself becomes a root.
- The API trims the file's title and cuts it to 120 characters. A file without a title gets `Imported loom`.
- A file of more Nodes than a Loom holds, or with two Nodes of the same `id`, gets 400.
If the API restarts while a Node runs, the Node runs again later. [When the API restarts](/docs/concepts/background-work#when-the-api-restarts) says how.
The [API reference](/docs/reference/looms/looms.createBranch) lists every field of each Loom route.
# Models (/docs/models)
Each request names a Base model in `model`, by the model's `id`: `llama-8b`, `llama-405b`, or `trinity-truebase`, the ids of the old platform. A request without `model` runs on `llama-8b`. Each answer's `model` names the weights that ran it, such as `meta-llama/Llama-3.1-8B`. `GET /v1/models` lists the models that the API serves, with each model's facts and the limits that the API enforces on a request to it.
## List the models [#list-the-models]
```bash tab="curl"
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
for model in client.models.list():
print(model.id, model.default, model.capabilities["n_layers"], model.capabilities["hidden_size"])
```
The curl command prints the list:
```json title="Models"
{
"object": "list",
"data": [
{
"id": "llama-8b",
"object": "model",
"owned_by": "acs",
"served_model_name": "meta-llama/Llama-3.1-8B",
"gpu_shape": "1xL40S",
"status": "live",
"default": true,
"capabilities": {
"max_model_len": 131072,
"max_logprobs": 100,
"max_output_work_tokens": 32000,
"max_inflight_per_key": 16,
"max_queued_per_key": 64,
"rate_limit_per_minute": 600,
"logprobs": true,
"prompt_logprobs": true,
"prompt_logprobs_full_vocab": true,
"full_vocab_max_prompt_tokens": 256,
"activations": true,
"max_steering_vectors": 8,
"max_activation_prompt_tokens": 256,
"n_layers": 32,
"hidden_size": 4096,
"chat_template": null
}
}
]
}
```
The API answers the list from its own settings and asks no Inference, so a list never starts a GPU.
Every User gets the same list, and may run each model in it.
### Model fields [#model-fields]
| Field | Meaning |
| --- | --- |
| `id` | The Model id to send as `model`. |
| `object` | Always `"model"`. |
| `owned_by` | Always `"acs"`. |
| `served_model_name` | The weights that Inference serves, which each answer's `model` names. |
| `gpu_shape` | The GPUs of one container and their type, such as `1xL40S` or `8xH200`. |
| `status` | Always `"live"`. It does not say whether the model is Warm. |
| `default` | `true` on the model that a request without `model` runs on. Set `model` on each request, because the default can change. |
| `capabilities` | The facts of the model and the limits of a request to it, with the keys below. |
### Keys of `capabilities` [#keys-of-capabilities]
| Key | Meaning |
| --- | --- |
| `max_model_len` | The most tokens of prompt and generated text together. |
| `max_logprobs` | The most top tokens of `logprobs`, and of a positive `prompt_logprobs`. |
| `max_output_work_tokens` | The most generated tokens of one request, counted as prompts × `n` × `max_tokens`. |
| `max_inflight_per_key` | Completions that run at once, for each API key. |
| `max_queued_per_key` | Completions that wait for a place, for each API key. |
| `rate_limit_per_minute` | Requests a minute that start work, for each API key, and as many other requests. |
| `logprobs`, `prompt_logprobs`, `prompt_logprobs_full_vocab` | Always `true`: the model takes `logprobs`, `prompt_logprobs`, and `prompt_logprobs: -1`. |
| `full_vocab_max_prompt_tokens` | The most prompt tokens in all × `n` with `prompt_logprobs: -1`. With `output_residual_stream`, the most is 16. |
| `activations` | Always `true`: the model takes `output_residual_stream` and `apply_steering_vectors`. |
| `max_steering_vectors` | The most vectors in `apply_steering_vectors`. |
| `max_activation_prompt_tokens` | The most tokens, of the prompt and `max_tokens`, of a capture of every Layer. A capture of fewer Layers can take more tokens. |
| `n_layers` | The number of the model's Layers, numbered from 0. Send Layers in `output_residual_stream` and `layer_indices`. |
| `hidden_size` | The numbers in one vector of the residual stream. A Steering vector's rows have this length. |
| `chat_template` | Always `null`, because the API applies no chat template. |
[Size caps](/docs/completions#size-caps) explains the caps of one request, and [Limits](/docs/limits) lists every limit with the error past it.
## Read one model [#read-one-model]
`GET /v1/models/{model}` answers one model of the list, with the same fields, as OpenAI's retrieve route does:
```bash tab="curl"
curl "$ACS_API_BASE/models/llama-8b" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
model = client.models.retrieve("llama-8b")
print(model.id, model.served_model_name, model.capabilities["max_model_len"])
```
The path takes the `served_model_name` too, with or without its slash encoded: `/v1/models/meta-llama/Llama-3.1-8B` answers the same model. A model that the list does not have gets 400 with the code `model_not_found`.
## Check whether a model is Warm [#check-whether-a-model-is-warm]
`GET /v1/models/{model}/status` says whether the model's next request starts at once, or which stage its start has reached:
```bash tab="curl"
curl "$ACS_API_BASE/models/llama-405b/status" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Model status"
{
"id": "llama-405b",
"object": "model.status",
"warm": false,
"boot": {
"stage": "weights_loading",
"stage_label": "Loading model weights",
"stage_age_s": 402
},
"activation_boot": {
"stage": "weights_loading",
"stage_label": "Loading model weights",
"stage_age_s": 402
}
}
```
| Field | Meaning |
| --- | --- |
| `id` | The Model id. |
| `object` | Always `"model.status"`. |
| `warm` | `true` when a container of the model serves, so its next request starts at once. `false` when the next request waits for a start. |
| `boot` | The stage of the start that runs, or `null` when none runs. Its fields are in the next table. |
| `activation_boot` | The same as `boot`. |
| `boot` field | Meaning |
| --- | --- |
| `stage` | `waiting_for_gpu`: a request waits, and no container of the model runs yet. `container_started`: a container sets up the model's server. `weights_loading`: it loads the weights onto its GPUs, or restores its memory snapshot. `weights_loaded`: it prepares its memory and kernels. |
| `stage_label` | The stage in words, such as `Loading model weights`. |
| `stage_age_s` | How long the start has run, in seconds. For `waiting_for_gpu`, it counts from the waiting request. For the other stages, it counts from the container's start. |
A model can be Warm and start a second container at the same time. Inference reports each container's start, and the API reads that report, so this route never starts a GPU. If the report is missing, `warm` is `true` for 5 minutes after Inference answered a Completion of the model, and `boot` is `null` unless a request waits. A model that the API does not serve gets 400 with the code `model_not_found`. [Warm and cold models](/docs/concepts/long-requests#warm-and-cold-models) explains why a cold model's first request waits.
## Served models [#served-models]
The API serves three Base models:
| `model` | What it is |
| --- | --- |
| `llama-8b` | `meta-llama/Llama-3.1-8B`, Meta's Llama 3.1 8B Base model, on one L40S GPU. Its context length is 131,072 tokens, the whole context of Llama 3.1. It has 32 Layers, 0 to 31, and a hidden size of 4,096. Its tokenizer adds the BOS token `<\|begin_of_text\|>` in front of a text prompt. |
| `llama-405b` | `meta-llama/Llama-3.1-405B`, Meta's Llama 3.1 405B Base model, on eight H200 GPUs. Its context length is 131,072 tokens, the whole context of Llama 3.1. It has 126 Layers, 0 to 125, and a hidden size of 16,384. Its tokenizer adds the BOS token `<\|begin_of_text\|>` in front of a text prompt. |
| `trinity-truebase` | `arcee-ai/Trinity-Large-TrueBase`, Arcee's Trinity Large TrueBase, a Base model with a mixture of experts, on eight H200 GPUs. Its context length is 8,192 tokens, the model's own. It has 60 Layers, 0 to 59, and a hidden size of 3,072. |
Llama 3.1 is licensed under Meta's [Llama 3.1 Community License](https://dev.meta.ai/llama/llama3_1/license/), and each use of it must follow Meta's [Acceptable Use Policy](https://dev.meta.ai/llama/llama3_1/use-policy/). Built with Llama.
A model that the API does not serve gets 400 with the code `model_not_found`, as [Unknown model](/docs/errors#unknown-model) shows. A prompt and `max_tokens` that together are longer than `max_model_len` get 400 with the code `context_length_exceeded`, as [Prompt too long](/docs/errors#prompt-too-long) shows.
### Harvests on each model [#harvests-on-each-model]
Each served model harvests. A harvest runs on GPUs of its own, apart from the model's Completions:
| `model` | GPUs of one harvest | Harvest caps |
| --- | --- | --- |
| `llama-8b` | One L40S GPU | The caps on Llama 3.1 8B |
| `llama-405b` | Eight H200 GPUs | The caps on Llama 3.1 405B and Trinity Large, which the two models share |
| `trinity-truebase` | Eight H200 GPUs | The caps on Llama 3.1 405B and Trinity Large, which the two models share |
[Harvests in Limits](/docs/limits#harvests) gives each cap. A GPU that ran a harvest stays up 20 minutes after it. A harvest of the same model in that time starts without loading the model, and its `timings.load_s` is 0. A harvest that needs a new GPU is `running` while Inference loads the model, and `timings.load_s` counts that time.
# Quickstart (/docs/quickstart)
In this tutorial, you send one prompt to a Base model with curl and with Python. By the end, you will have read the tokens that it wrote, with the log probability of each, and streamed them as they arrive.
## Set up [#set-up]
1. Apply with the [Apply form](https://www.v2.infra.acsresearch.org/apply). When an admin approves your Application, the approval email lets you set your password. [Get access](/docs/get-access) has each step.
2. With your password set, [sign in to the App](https://v2.infra.acsresearch.org/sign-in) and make an API key, as [Make an API key](/docs/api-keys#make-an-api-key) shows.
3. Put the key and the API's address in shell variables. The curl commands and the Python programs in these docs read both:
```bash
export ACS_API_KEY=sk-acs-...
export ACS_API_BASE=https://api.v2.infra.acsresearch.org/v1
```
4. Install the `openai` Python SDK, which the Python programs use:
```bash
pip install openai
```
## Send a prompt [#send-a-prompt]
Send the prompt `"The sky is"` to `llama-8b`, Meta's Llama 3.1 8B. Ask for at most 2 tokens, and for the log probability of each token with `logprobs`. Run the curl command, the Python program, or both:
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 1}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt="The sky is", logprobs=1, max_tokens=2)
print(completion.choices[0].text)
print(completion.choices[0].logprobs.token_logprobs)
```
The curl command prints the whole Completion:
```json title="Completion"
{
"id": "cmpl-946f88cbcf513663",
"object": "text_completion",
"created": 1791780418,
"model": "meta-llama/Llama-3.1-8B",
"choices": [
{
"index": 0,
"text": " dark with",
"logprobs": {
"text_offset": [0, 5],
"token_logprobs": [-3.864488124847412, -3.125293254852295],
"tokens": [" dark", " with"],
"top_logprobs": [
{ " dark": -3.864488124847412, " the": -1.4269880056381226 },
{ " with": -3.125293254852295, " and": -1.5002931356430054 }
]
},
"finish_reason": "length",
"prompt_logprobs": null
}
],
"usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}
```
In the Completion, you will see these fields:
- `choices[0].text` is the text that the model wrote after your prompt.
- `choices[0].logprobs.tokens` lists each token, and `token_logprobs` has the log probability of each.
- `top_logprobs` has the likeliest tokens at each position. `logprobs: 1` asks for 1 of them, and the token that the model wrote is there too.
- `finish_reason` is `"length"`, because the model stopped at `max_tokens`.
- `usage` counts the tokens of the prompt and of the text. The prompt has 4 tokens, because Llama's tokenizer adds `<|begin_of_text|>` in front of it.
The Python program prints the text and the log probability of each token. The model samples each token, so the Python run is a separate sample, and its tokens differ from those of the curl run:
```text title="Python output"
a beautiful
[-3.176988124847412, -3.1364614963531494]
```
The model continued your text. A Base model does not answer a prompt as a question, and the API applies no chat template. [Base models](/docs/concepts/base-models) explains why.
## Stream the tokens [#stream-the-tokens]
Send the same prompt with `stream: true`. The API sends each token as a Server-Sent Event when Inference makes it:
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
stream = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, stream=True
)
for chunk in stream:
print(chunk.choices[0].text, end="", flush=True)
print()
```
The curl command prints one event for each token, then `[DONE]`. The last token's event has the `finish_reason`:
```text title="Events"
data: {"id":"cmpl-9a33ca90561191f8","object":"text_completion","created":1791780425,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" again","logprobs":null,"finish_reason":null}]}
data: {"id":"cmpl-9a33ca90561191f8","object":"text_completion","created":1791780425,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" filled","logprobs":null,"finish_reason":"length"}]}
data: [DONE]
```
The Python program prints the text as the tokens arrive. It is another separate sample:
```text title="Python output"
usually the
```
You have sent a prompt to a Base model, read its tokens with their log probabilities, and streamed them.
## Next steps [#next-steps]
[Your first steered Completion](/docs/tutorials/first-steered-completion) reads the model's Activations and changes what it writes with a Steering vector. [Sample continuations](/docs/sampling) sets the sampling fields, sends several prompts at once, and handles streams. [Read token probabilities](/docs/token-probabilities) gets the log probabilities of your prompt and of the whole vocabulary. [Jobs](/docs/jobs) runs many requests in the background and keeps the results.
# Retry requests safely (/docs/retries)
A broken connection or a 5xx answer can leave you unsure whether a request ran. This guide shows how to send it again so that the API does the work and charges your Budget once. [Errors](/docs/errors) describes each answer.
## Send a request again safely [#send-a-request-again-safely]
`POST /v1/completions`, `POST /v1/completion-jobs`, `POST /v1/harvest`, and every Loom write take the `Idempotency-Key` header. A cancel of a Job or a harvest needs none: a second cancel answers the Job as it is.
1. Make a new key for each new request, such as a UUID. A key has 1 to 255 characters.
2. Send it in the `Idempotency-Key` header.
3. If the request fails or the connection breaks, send the same request with the same key again.
```bash tab="curl"
KEY=$(uuidgen)
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'
```
```python tab="Python"
import os
import uuid
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
headers = {"Idempotency-Key": str(uuid.uuid4())}
first = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, extra_headers=headers
)
again = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, extra_headers=headers
)
print(first.id == again.id)
```
`uuidgen` makes the key in the shell. Run the `curl` command again with the same `KEY`, and it prints the same Completion. The Python program prints `True`: its second request got the first request's Completion, and Inference ran once. In the `openai` SDK, `extra_headers` sends the header, and the SDK sends it again with each of its own tries.
With a key, the API answers each repeat of a request in this way:
- After the first request finished, the same request gets the first answer. A change to your Budget or to the served models since then does not change it. A Completion comes with the first answer's `X-Acs-Max-Tokens-Clamped` header. A Job, a Loom, or a Branch comes as it is now, and a Job comes with 200 in place of 202.
- While the first request still runs, the same request gets 409 with the code `running` and `Retry-After: 1`.
- A stream runs once. The same stream request again gets 409 `no_answer` with `x-should-retry: false`, and the API does not run it or charge it again. Send a new key to run it again.
- A Completion without a stream runs on when your client goes away, and pays only for its usage. The same request again gets its answer.
- A request that failed before the API charged it, for example with 429 `budget_exceeded` or an error from Inference, frees its key. The same request with the key runs.
- Another request with the same key, or the same key on another route, gets 409 `idempotency_conflict` with `x-should-retry: false`.
The API keeps each key for 24 hours. All your API keys share one set of keys, and another User's keys are apart from yours. A request without the header runs each time that it arrives.
## Which errors to try again [#which-errors-to-try-again]
Send the same request again, with the same `Idempotency-Key`, after these answers:
- 429 `rate_limited` or `queue_full`: wait for the time in the `Retry-After` header. A full Job cap is different: it answers 429 `job_capacity` with `x-should-retry: false`, so submit again when one of your Jobs ends.
- 429 `harvest_concurrency_exceeded`: wait for the time in the `Retry-After` header, or until one of your harvests ends.
- 429 `harvest_capacity_exceeded`: wait for the time in the `Retry-After` header.
- 409 `running`: wait 1 second.
- 503 `db_pool_exhausted` or `circuit_open`: wait for the time in the `Retry-After` header. It is 1 second when the API's database was busy, and up to 60 seconds while the API holds a model back. Nothing ran, and you pay nothing.
- Any error with `retryable: true`, such as 502 `upstream_unreachable` or `vllm_oom`: wait `retry_after_seconds`, the same as the `Retry-After` header.
- 504 `upstream_deadline`: wait a few minutes. Inference did not answer for 14 minutes, so another try at once waits as long, and starts the work on a GPU again.
After these answers, check or change the request before you send it again:
- 500 `internal_error`, with `x-should-retry: false`: the API failed, and the work may have run. If the request had an `Idempotency-Key`, send it again with the same key. It gets the first answer if the work finished, and the API does not charge it twice. Without a key, look for the result first, for example in `GET /v1/looms`. [Report the error](/docs/support#report-a-bug) with its `X-Request-Id`.
- 429 `budget_exceeded`: ask for fewer tokens, with a smaller `max_tokens` or fewer prompts or choices, or [ask for more tokens](/docs/support#ask-for-more-tokens).
- 429 `spend_cap`: new GPU work is paused ([Spend limit](/docs/limits#spend-limit)). Try again after the `Retry-After` header's hour.
- 409 `no_answer`: the stream ran already. Send a new key to run it again.
- 409 `idempotency_conflict`: the key was used with another request. Send a new key with each new request.
- 429 `harvest_quota_exceeded`: the API key started its most harvests this month. Submit the Job in the next UTC month.
- 429 `spend_cap`: the API paused new harvests to limit GPU spending. Wait for the time in the `Retry-After` header.
- Any other 4xx status. Fix what `message` and `param` name.
[Status codes](/docs/errors#status-codes) lists every answer.
## Set the client for long requests [#set-the-client-for-long-requests]
A Completion can wait while Inference starts a cold model, and a long one runs for minutes. The API waits up to 14 minutes for Inference ([Cold starts and long requests](/docs/concepts/long-requests)).
1. Create the client with `timeout=900`. The SDK then waits 900 seconds, longer than the API's 14 minutes.
2. Create it with `max_retries=0`. Each try that the SDK sends after a broken connection starts the work on a GPU again. Without a key, a cut Completion without a stream also pays its whole Hold.
3. Send each request with an `Idempotency-Key`, and send it again yourself after a broken connection.
4. Check the answer for `error`. After 150 seconds, a Completion without a stream gets its 200 status, and a later failure comes as the body.
```python
import os
import uuid
from openai import OpenAI
client = OpenAI(
base_url=os.environ["ACS_API_BASE"],
api_key=os.environ["ACS_API_KEY"],
timeout=900,
max_retries=0,
)
headers = {"Idempotency-Key": str(uuid.uuid4())}
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, extra_headers=headers
)
error = getattr(completion, "error", None)
if error:
print(error["message"])
else:
print(completion.choices[0].text)
```
The SDK raises no exception for an error in the body of a 200 answer. It returns a Completion whose `error` field holds the error object. With curl, set `--max-time` above 840 seconds, or leave it out.
## Retries in the `openai` SDK [#retries-in-the-openai-sdk]
The `openai` SDK sends a request again by itself, two more times by default, after:
- a connection error or a timeout
- 408, 409, and 429 answers, and every 5xx answer
Before each try, it waits as long as the `retry-after-ms` or `Retry-After` header says, up to 2 minutes. Without either header, it waits about half a second, then twice as long each time, up to 8 seconds.
It does not try again after an answer with `x-should-retry: false`, or with a `Retry-After` of more than 2 minutes. The API sends `x-should-retry: false` with these answers, so the SDK raises them at once:
- 409 for a Node that is not in a state for the action, 409 `harvest_not_cancellable`, 409 `idempotency_conflict`, and 409 `no_answer`
- 429 `budget_exceeded`, 429 `job_capacity`, 429 `harvest_quota_exceeded`, and 429 `spend_cap`
- its own 500
- 413 `request_too_large`, 504 `upstream_deadline`, and each error of Inference whose `retryable` is `false`
To raise every error at its first answer, create the client with `max_retries=0`. [Errors in the `openai` SDK](/docs/errors#errors-in-the-openai-sdk) lists the error classes.
# Sample continuations (/docs/sampling)
Use `POST /v1/completions` to sample continuations of your prompts: set the sampling fields, ask for many choices at once, stream the tokens, and repeat a sample. Each example reads `ACS_API_BASE` and `ACS_API_KEY`, which you export in the [Quickstart](/docs/quickstart#set-up).
## Set the sampling fields [#set-the-sampling-fields]
1. Choose `temperature`, from `0` to `2`. `0` takes the likeliest token each time, and the default, `1`, samples from the model's own probabilities. A higher value samples rarer tokens more often.
2. To sample only from the likeliest tokens, set `top_p`, or vLLM's `top_k` or `min_p`.
3. Set `max_tokens` to the most tokens for each choice, and `stop` to the texts that end a choice early.
4. Send the request.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 16, "temperature": 0.7, "top_p": 0.9, "stop": ["\n"]}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=16,
temperature=0.7,
top_p=0.9,
stop=["\n"],
)
print(completion.choices[0].text)
```
A field that you leave out takes OpenAI's default. The model's own settings play no part, so a request samples in the same way on every model. [How to sample](/docs/completions#how-to-sample) lists each sampling field and its default.
## Send several prompts [#send-several-prompts]
1. Put the prompts in a list in `prompt`: texts, or lists of token ids.
2. Set `n` to the number of choices for each prompt.
3. Set `max_tokens`. A request with more than one prompt, or with `n` above 1, must set it.
4. Read choice `i` of prompt `p` at the index `p * n + i`.
Here, `jq` prints the index and the text of each choice:
```bash tab="curl"
curl -s "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}' \
| jq -c '.choices[] | {index, text}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt=["The sky is", "The grass is"], max_tokens=2, n=2
)
for choice in completion.choices:
print(choice.index, repr(choice.text))
```
```jsonl title="Choices"
{ "index": 0, "text": " blue." }
{ "index": 1, "text": " blue because" }
{ "index": 2, "text": " always gre" }
{ "index": 3, "text": " always gre" }
```
```text title="Python output"
0 ' so dark'
1 ' the limit'
2 ' never gre'
3 ' always belonging'
```
Choices 0 and 1 continue `"The sky is"`, and choices 2 and 3 continue `"The grass is"`. The curl command and the Python program sampled apart, so their texts differ.
The `usage` adds up all the prompts and all the choices. One request takes one [Hold](/docs/concepts/budgets#the-hold) for all its choices, and asks for at most 128 choices in all ([Size caps](/docs/completions#size-caps)). [Sample many rollouts](/docs/tutorials/sample-many-rollouts) also sends many requests at once.
## Send vLLM's fields from the `openai` SDK [#send-vllms-fields-from-the-openai-sdk]
vLLM, the engine that runs the Base models in Inference, takes fields that OpenAI's API does not have, such as `top_k`, `min_p`, and `repetition_penalty`. [Request fields](/docs/completions#request-fields) marks each of them. The `openai` SDK has no argument for these fields, and raises `TypeError` if you pass one.
1. Put the vLLM fields in a dictionary.
2. Pass it as `extra_body`. The SDK adds its fields to the top level of the request body.
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
temperature=0.7,
extra_body={"top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1},
)
print(completion.choices[0].text)
```
```text title="Python output"
the limit
```
With curl, put the same fields at the top level of the body. The API refuses a field named `extra_body` with 400 `invalid_request`.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0.7, "top_k": 20, "min_p": 0.05, "repetition_penalty": 1.1}'
```
## Stream a Completion [#stream-a-completion]
1. Set `stream: true`. With curl, add `-N`, so that curl prints each event when it arrives.
2. To get the token counts, add `"stream_options": {"include_usage": true}`.
3. Read the `data:` events until `data: [DONE]`.
```bash tab="curl"
curl -N "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "stream_options": {"include_usage": true}}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
stream = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
stream=True,
stream_options={"include_usage": True},
)
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].text, end="", flush=True)
elif chunk.usage:
print(f"\n[tokens: {chunk.usage.total_tokens}]")
```
The curl command prints the events:
```text title="Events"
data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" deep","logprobs":null,"finish_reason":null}]}
data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" and","logprobs":null,"finish_reason":"length"}]}
data: {"id":"cmpl-a18dc61150cade3a","object":"text_completion","created":1791780733,"model":"meta-llama/Llama-3.1-8B","choices":[],"usage":{"prompt_tokens":4,"completion_tokens":2,"total_tokens":6}}
data: [DONE]
```
The Python program sampled apart from the curl command. It prints the text as the tokens arrive, and then the token count:
```text title="Python output"
teeming
[tokens: 6]
```
Each chunk has the next text of one choice, and its `index`. With several prompts or choices, the chunks of the choices come mixed. `finish_reason` is `null` until the last chunk of a choice.
The last chunk of `include_usage` has `"choices": []`, so `chunk.choices[0]` would fail on it. The check `if chunk.choices:` skips that chunk, and the loop works with or without `include_usage`. Without it, the stream has no token counts, and your Hold still settles to the tokens that Inference wrote.
A stream that ends without `data: [DONE]` did not finish. If Inference fails after the stream starts, the last event before `[DONE]` is an error object, and the `openai` SDK raises `openai.APIError` ([Inference fails during a stream](/docs/errors#inference-fails-during-a-stream)). `logprobs` works in a stream. The API refuses `prompt_logprobs` in a stream with 400, and a [Job](/docs/jobs) cannot stream.
## Stop a stream early [#stop-a-stream-early]
1. Open the stream in a `with` block.
2. Leave the loop when you have the text that you need. The `with` block then closes the connection.
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
with client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=200, stream=True
) as stream:
for chunk in stream:
if chunk.choices:
print(chunk.choices[0].text, end="", flush=True)
if "." in chunk.choices[0].text:
break
print()
```
This program stops at the first token that has a full stop. When the connection closes, the API stops its request to Inference. You pay for the tokens that the stream sent until then, and the Hold gives back the rest ([How a Hold ends](/docs/concepts/budgets#how-a-hold-ends)). With curl, stop the command with Ctrl+C.
## Get the same tokens again [#get-the-same-tokens-again]
1. Set `temperature` to `0`, or set a `seed` of 0 or more.
2. Send the same request again, with every other field the same, to the same model.
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
for _ in range(2):
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=8, seed=7
)
print(repr(completion.choices[0].text))
```
The two requests get the same tokens, except where vLLM's batching changes them. vLLM batches your request with other requests. The batch can change a log probability by a little, and now and then a token where two tokens are almost equally likely. To compare log probabilities between requests, send each request more than once, and read a small difference as no difference. [Why repeats differ](/docs/concepts/base-models#why-repeats-differ) explains it.
# Steer a Completion (/docs/steering)
Add a Steering vector to the residual stream of a Base model at the Layers you choose, to change what it writes. Read the steered Layers with [Activations](/docs/activations) to see what your vector changed. [How a Steering vector changes it](/docs/concepts/residual-stream#how-a-steering-vector-changes-it) explains the addition.
Send Steering vectors in the field `apply_steering_vectors` of `POST /v1/completions`. The field has the name that vllm-lens uses. The `openai` SDK does not know it, so the Python programs send it in `extra_body`. A steered Completion holds and spends tokens as any other Completion does.
The Python programs need Python 3.14 (for `compression.zstd`), NumPy, and the `openai` SDK. They decode the residual stream with the helper from [Decode the tensor](/docs/activations#decode-the-tensor).
## Write a Steering vector [#write-a-steering-vector]
A Steering vector is an object in the list `apply_steering_vectors`. Its `activations` are JSON numbers, or the same numbers encoded as in [Encode a Steering vector](#encode-a-steering-vector). For example, `{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}` adds 2 times `vector` at Layer 1.
- `activations` has one row for each Layer in `layer_indices`. Row `i` goes to Layer `layer_indices[i]`.
- A row is one vector of the model's hidden size. Inference adds it at every prompt token, and at every generated token that runs through the model.
- A row can also be a list of vectors, one for each position that it steers ([Steer chosen positions](#steer-chosen-positions)).
[`GET /v1/models`](/docs/models) lists the hidden size as `capabilities.hidden_size`. `meta-llama/Llama-3.1-8B` has a hidden size of 4096, so each vector has 4096 numbers. To send a NumPy array, convert it with `array.tolist()`. The API sends the numbers to Inference as float32, so each number must be within float32's range.
## Encode a Steering vector [#encode-a-steering-vector]
Encode a large Steering vector to send it in about a tenth of the bytes of JSON numbers. The API checks an encoded vector by the same rules as numbers, and sends it to Inference as it is. This helper encodes a NumPy array of one row for each Layer, as float32:
```python
import base64
from compression import zstd
import numpy as np
def encode(rows):
array = np.asarray(rows, dtype=np.float32)
return {
"data": base64.b64encode(zstd.compress(array.tobytes())).decode(),
"dtype": "float32",
"original_dtype": "torch.float32",
"shape": list(array.shape),
"compression": "zstd",
}
```
Send `{"activations": encode([vector]), "layer_indices": [1], "scale": 2}` in place of `vector.tolist()`. The API also takes bfloat16 numbers as `int16` with `original_dtype` `torch.bfloat16`, as the old platform's `encode_vector` writes them. They take half the bytes of float32. `shape` is `[layers, hidden size]`, or `[layers, positions, hidden size]` for a list of vectors for each Layer. The data must be one zstd frame that states its size, as `zstd.compress` writes it.
## Steer and read in one request [#steer-and-read-in-one-request]
This request adds 2 times a vector at Layer 1, and reads Layer 1 to show the change. The vector is `[1, 2, 3]` and then 4093 zeros:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
vector = np.zeros(4096)
vector[:3] = [1, 2, 3]
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={
"output_residual_stream": [1],
"apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}],
},
)
residual_stream = decode(completion.activations["residual_stream"])
print(residual_stream.shape)
print(residual_stream[0, :, :4].tolist())
```
The program prints the shape of Layer 1, and the first 4 numbers of each token:
```text title="Output"
(1, 5, 4096)
[[2.015625, 4.0, 5.9375, -0.05419921875], [1.9765625, 4.0, 6.0, 0.0030517578125], [1.9921875, 4.0, 6.0, 0.0157470703125], [2.0, 4.0, 6.0, 0.0150146484375], [2.015625, 3.984375, 6.0, -0.01611328125]]
```
The read of a steered Layer includes the steer. Layer 1's own numbers are close to 0 here, so the first 3 numbers of each token are close to 2, 4, and 6. The fourth number stays small.
A request with Steering vectors can also stream, with or without `output_residual_stream`.
## Compare with a zero-scale baseline [#compare-with-a-zero-scale-baseline]
Send the same request at `scale: 0` as the baseline for a steer, not the request without Steering vectors. A request with Steering vectors takes another path through Inference, as [Why compare with scale 0](/docs/concepts/residual-stream#why-compare-with-scale-0) explains.
Set `temperature: 0` in both requests, so that sampling does not change the text between them. A difference between the two answers then comes from your vector. vLLM's batching can still change a token now and then ([How to sample](/docs/completions#how-to-sample)).
This program steers Layer 1 at scale 0 and at scale 2, and reads Layers 0 and 1 each time. It prints both texts, and the first 4 numbers of the difference at each token:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
vector = np.zeros(4096)
vector[:3] = [1, 2, 3]
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=1,
temperature=0,
extra_body={
"output_residual_stream": [0, 1],
"apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 0}],
},
)
steered = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=1,
temperature=0,
extra_body={
"output_residual_stream": [0, 1],
"apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}],
},
)
print(repr(baseline.choices[0].text), repr(steered.choices[0].text))
difference = decode(steered.activations["residual_stream"]) - decode(baseline.activations["residual_stream"])
print(difference[:, :, :4].round(2).tolist())
```
```text title="Output"
' the' ' a'
[[[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]], [[2.0, 3.990000009536743, 6.019999980926514, 0.0], [2.0, 4.0, 6.010000228881836, 0.0], [2.0, 4.0, 6.0, 0.0], [2.0, 4.010000228881836, 6.010000228881836, 0.0]]]
```
Layer 0 comes before the steer, so it does not change. Layer 1 moved by `2 * [1, 2, 3]` at each token, within bfloat16's precision, and its fourth number did not move. Here, the steer also changed the next token from `" the"` to `" a"`.
Equal texts do not show that a steer ran. To see the steer, read the steered Layer as this program does.
## Check a steer on your own vector [#check-a-steer-on-your-own-vector]
The API answers a steer at a Layer that you did not mean as it answers any other request. To check your vector `v` at Layer `L`, read Layers `L - 1` and `L`, with `max_tokens: 1` and `temperature: 0`. With more generated tokens, the steered text goes another way, and every later row differs for that reason alone.
1. Send `scale: 0` as the baseline.
2. Send the same request with a scale other than 0 and `norm_match: false`. Layer `L - 1` must match the baseline. Layer `L` must move by `scale * v` at each token.
3. Send the same request with `norm_match: true`. Subtract the baseline from each row of Layer `L`. The norm of each difference must be about `scale` times the norm of the baseline row.
The program in [Compare with a zero-scale baseline](#compare-with-a-zero-scale-baseline) does steps 1 and 2 for Layer 1.
## Steer chosen positions [#steer-chosen-positions]
To steer some positions only, give each Layer a list of vectors in `activations`, one for each position, and name the positions in `position_indices`. A position counts from 0 at the first prompt token. On `llama-8b`, position 0 of a text prompt is the BOS token. Without `position_indices`, the vectors go to positions 0, 1, 2, and so on.
This program adds a vector at position 2 of Layer 1, first at scale 0 and then at scale 2. It reads Layers 0 and 1 each time, and prints the first 4 numbers of the difference at each token:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
vector = np.zeros(4096)
vector[:3] = [1, 2, 3]
def at_position_2(scale):
return {"activations": [[vector.tolist()]], "layer_indices": [1], "position_indices": [2], "scale": scale}
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=1,
temperature=0,
extra_body={"output_residual_stream": [0, 1], "apply_steering_vectors": [at_position_2(0)]},
)
steered = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=1,
temperature=0,
extra_body={"output_residual_stream": [0, 1], "apply_steering_vectors": [at_position_2(2)]},
)
difference = decode(steered.activations["residual_stream"]) - decode(baseline.activations["residual_stream"])
print(difference[:, :, :4].round(2).tolist())
```
Layer 0 comes before the steer, so it does not move. On Layer 1, only the row of position 2 moves, by `2 * [1, 2, 3]` up to the rounding of bfloat16.
### Steer a span of positions [#steer-a-span-of-positions]
To add one vector at every position of one or more spans, give one vector for each Layer and `position_spans`, a list of `[start, end]` pairs. A span covers `start` to `end - 1`. For example, `{"activations": [vector.tolist()], "layer_indices": [1], "position_spans": [[2, 5]]}` adds `vector` at positions 2, 3, and 4 of Layer 1.
Spans may come in any order, and may not overlap. They cover at most 4,096 positions in all. The API sends Inference one copy of each vector for each position, so the cap on numbers counts each position: [Size caps](/docs/completions#size-caps).
### Positions that a request reaches [#positions-that-a-request-reaches]
Inference adds a vector at a position when a forward pass runs that position. The forward passes run each prompt token, and each generated token but the last, which the model samples and never runs. So a request with a prompt of P tokens and `max_tokens` M reaches positions 0 to P + M - 2. It never reaches the last position of the model's context, `max_model_len` - 1.
The API answers 400 on `apply_steering_vectors` for a position after that, before it holds any tokens. It counts a text prompt's tokens as the Hold does, with the model's tokenizer ([The Hold](/docs/concepts/budgets#the-hold)). Inference counts the tokens too, and refuses a position past them with the same error.
This program asks for position 5 with a prompt of 4 token ids and `max_tokens: 2`, so its passes reach position 4 at most:
```python
import json
import os
from openai import BadRequestError, OpenAI
vector = [0.0] * 4096
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
try:
client.completions.create(
model="llama-8b",
prompt=[128000, 791, 13180, 374],
max_tokens=2,
extra_body={"apply_steering_vectors": [{"activations": [[vector]], "layer_indices": [1], "position_indices": [5]}]},
)
except BadRequestError as error:
print(json.dumps(error.response.json(), indent=2))
```
```json title="Error"
{
"error": {
"message": "apply_steering_vectors steers position 5, but the forward passes of this request reach position 4 at most: the prompt has 4 tokens and max_tokens is 2, and the last generated token never runs through the model. Steer a position up to 4, or raise max_tokens.",
"type": "invalid_request_error",
"param": "apply_steering_vectors",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
A position that the request reaches is steered only when generation gets there. When a stop text or the end-of-text token ends the Completion first, its `finish_reason` is `stop`, and the later positions are not steered.
## Steering vector fields [#steering-vector-fields]
| Field | Default | What it does |
| --- | --- | --- |
| `activations` | Required | One row for each Layer in `layer_indices`. A row is one vector of the model's hidden size, or a list of such vectors, one for each position. Send numbers, or [encode them](#encode-a-steering-vector). |
| `layer_indices` | Required | The Layers to steer, each one once. |
| `scale` | `1.0` | Multiply the vector by this number. [Choose a scale](/docs/concepts/residual-stream#choose-a-scale) says how to pick it. |
| `norm_match` | `false` | First scale the vector to the norm of the residual stream at each token. [Match the norm](/docs/concepts/residual-stream#match-the-norm) explains it. |
| `position_indices` | None | The position of each vector in a row that is a list of vectors, counted from 0 at the first prompt token. Left out, the vectors go to positions 0, 1, 2, and so on. |
| `position_spans` | None | The `[start, end]` spans of positions that the one vector in each row goes to, at most 4,096 positions in all. Not with `position_indices`. |
## Requests that the API refuses [#requests-that-the-api-refuses]
The API answers 400 `invalid_request_error` for these requests before it holds any tokens, and they do not reach Inference. `param` is `apply_steering_vectors`. If the request also has `output_residual_stream`, the errors for `n` and for several prompts name that field instead.
- An empty list of Steering vectors, `n` other than 1, or several prompts.
- More than 8 Steering vectors, or more than 2,097,152 numbers in all of them.
- Rows of `activations` with different lengths, or a number of rows that is not the number of `layer_indices`.
- A repeated or negative Layer.
- A Layer that the model does not have, or a vector whose length is not the model's hidden size. [`GET /v1/models`](/docs/models) lists both as `capabilities.n_layers` and `capabilities.hidden_size`.
- A number outside float32's range, from -3.4028235e+38 to 3.4028235e+38.
- An encoded vector whose data is not base64 or zstd, whose numbers are not float32 or bfloat16, or whose bytes do not fill its `shape`. Its numbers must be finite.
- A field that a Steering vector does not have, such as `scael` for `scale`.
- `position_indices` with one vector for each Layer, or with more or fewer positions than vectors.
- `position_spans` with `position_indices`, or with a list of vectors for each Layer.
- A span that does not end after it starts, spans that overlap, or spans of more than 4,096 positions in all.
- A position that no forward pass of the request reaches, as [Positions that a request reaches](#positions-that-a-request-reaches) says.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "apply_steering_vectors": [{"activations": [[1, 2, 3]], "layer_indices": [1, 2]}]}'
```
```json title="Error"
{
"error": {
"message": "activations needs one row for each layer in layer_indices.",
"type": "invalid_request_error",
"param": "apply_steering_vectors",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
The whole request body is also capped, and a body over the cap gets 413 before the API reads the vectors. [Size caps](/docs/completions#size-caps) lists both caps.
Inference answers only for a served model that `GET /v1/models` does not list. For such a model, Inference checks the Layers and the hidden size itself, and the API passes its 400 on.
## Known limits [#known-limits]
- Vectors that steer the same Layer add up.
- If steering fails inside Inference, every request with Activations or Steering vectors in the same batch fails with 500.
## Next steps [#next-steps]
[Steer with a contrast vector](/docs/cookbook/contrast-steering) builds a Steering vector from the Activations of two sets of prompts, and steers with it at a few scales. [Ask for Activations in a Job](/docs/jobs#ask-for-activations-in-a-job) sends many steered requests in the background.
# Read token probabilities (/docs/token-probabilities)
Read how likely a Base model found each token: the tokens that it generates, the tokens of your prompt, or every token of its vocabulary. Use them to measure surprisal, to score a text, or to see what the model predicted.
Each section sets one field of `POST /v1/completions`. [What to return](/docs/completions#what-to-return) lists these fields and their defaults.
## Get log probabilities for generated tokens [#get-log-probabilities-for-generated-tokens]
Set `logprobs` to a number k from 0 to 100. Each generated token then gets its log probability, and each generated position gets the k likeliest tokens. Leave `logprobs` out to get none.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 5}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, logprobs=5
)
logprobs = completion.choices[0].logprobs
for token, logprob, top in zip(
logprobs.tokens, logprobs.token_logprobs, logprobs.top_logprobs
):
print(repr(token), logprob, top)
```
The curl command and the Python program each send their own request, so their outputs show different samples. The curl command prints the Completion:
```json title="Completion"
{
"id": "cmpl-b3558603c61e9c6e",
"object": "text_completion",
"created": 1791780960,
"model": "meta-llama/Llama-3.1-8B",
"choices": [
{
"index": 0,
"text": " growing clear",
"logprobs": {
"text_offset": [0, 8],
"token_logprobs": [-6.239488124847412, -6.665775299072266],
"tokens": [" growing", " clear"],
"top_logprobs": [
{
" growing": -6.239488124847412,
" the": -1.4269880056381226,
" falling": -2.114488124847412,
" blue": -2.489488124847412,
" not": -2.926988124847412,
" a": -3.176988124847412
},
{
" clear": -6.665775299072266,
" dark": -1.228275179862976,
" darker": -1.353275179862976,
" lighter": -3.0407752990722656,
" brighter": -3.1657752990722656,
" ever": -3.4157752990722656
}
]
},
"finish_reason": "length",
"prompt_logprobs": null
}
],
"usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}
```
The Python program prints one line for each generated token, with its text, its log probability, and its map of likeliest tokens:
```text title="Python output"
' light' -5.864488124847412 {' light': -5.864488124847412, ' the': -1.4269880056381226, ' falling': -2.114488124847412, ' blue': -2.489488124847412, ' not': -2.926988124847412, ' a': -3.176988124847412}
'ening' -0.8648753762245178 {'ening': -0.8648753762245178, ' blue': -1.364875316619873, ',': -2.864875316619873, ' and': -3.114875316619873, ' but': -4.052375316619873}
```
`choices[i].logprobs` has four lists, with one entry for each generated token:
| List | Each entry |
| --- | --- |
| `tokens` | The text of the token. |
| `token_logprobs` | The log probability of the token. |
| `top_logprobs` | A map from the text of each of the k likeliest tokens to its log probability. |
| `text_offset` | Where the token starts in `text`, in characters. |
A map in `top_logprobs` also holds the generated token when it is not one of the k likeliest. So a map has k or k + 1 entries. In the Completion, `" growing"` is not among the 5 likeliest tokens, so its map has 6 entries. In the Python output, `'ening'` is the likeliest token at its position, so its map has 5 entries.
`logprobs: 0` still returns the log probability of each generated token. Each map then holds only the generated token.
The API refuses `true` and `-1` with 400, because `logprobs` takes a number from 0 to 100. To get every token of the vocabulary, follow [Get the whole vocabulary](#get-the-whole-vocabulary).
```bash refused
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": true}'
```
```json title="Error"
{
"error": {
"message": "Invalid type for 'logprobs': expected a number or null.",
"type": "invalid_request_error",
"param": "logprobs",
"code": "invalid_request"
},
"request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}
```
For `-1`, the message is `Invalid value for 'logprobs': expected a value greater than or equal to 0.`
## Get log probabilities for the prompt [#get-log-probabilities-for-the-prompt]
Set `prompt_logprobs` to a number k from 0 to 100. Each prompt position then gets the log probability of the prompt's token and of the k likeliest tokens. `prompt_logprobs` is a field of vLLM that the `openai` SDK does not know, so the Python program sends it in `extra_body`. curl sends it with the other fields.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "prompt_logprobs": 5}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={"prompt_logprobs": 5},
)
for position, tokens in enumerate(completion.choices[0].prompt_logprobs):
if tokens is None:
print(position, None)
continue
for token_id, token in tokens.items():
text = repr(token["decoded_token"])
print(position, token_id, text, token["rank"], token["logprob"])
```
The Python program prints one line for each token at each position. A line has the position, the token id, the token's text, its rank, and its log probability. The curl command prints the same fields in the Completion's JSON.
```text title="Python output"
0 None
1 2 '#' 3 -2.6954822540283203
1 755 'def' 2 -2.1954822540283203
1 791 'The' 4 -3.6329822540283203
1 14924 'Question' 1 -1.1954821348190308
1 16309 'Tags' 5 -4.13298225402832
2 220 ' ' 1 -3.7540297508239746
2 502 ' new' 4 -5.129029750823975
2 1176 ' first' 3 -4.660279750823975
2 2768 ' following' 2 -4.285279750823975
2 7252 ' Best' 5 -5.222779750823975
2 13180 ' sky' 1144 -8.832155227661133
3 374 ' is' 1 -0.9844952821731567
3 574 ' was' 2 -2.109495162963867
3 596 "'s" 4 -3.359495162963867
3 753 '’s' 3 -3.109495162963867
3 3485 ' above' 5 -3.484495162963867
```
`choices[i].prompt_logprobs` has one entry for each prompt token, `usage.prompt_tokens` in all:
- Position 0 is `null`, because no token comes before it. On `meta-llama/Llama-3.1-8B`, position 0 is the BOS token `<|begin_of_text|>`, which Inference adds in front of a text prompt. The first token of your text is at position 1.
- Each later position maps a token id, as a string, to the token's `logprob`, its `rank` among all tokens, and its text in `decoded_token`. Rank 1 is the likeliest token.
- The prompt's own token is always in the map. When it is not one of the k likeliest, vLLM adds it with its rank.
At position 2, the prompt's token `" sky"` has rank 1144, so the model found 1,143 other tokens likelier after `"The"`. At position 3, `" is"` has rank 1, so it was the likeliest token after `"The sky"`.
`prompt_logprobs` does not work with `stream: true`. The API refuses the request with 400 and `param` `prompt_logprobs` before it holds any tokens.
To add up the log probabilities of a text that you already have, follow [Score a text](/docs/cookbook/score-a-text).
## Get the whole vocabulary [#get-the-whole-vocabulary]
Set `prompt_logprobs` to `-1` to get the log probability of every token in the vocabulary at each prompt position. On `meta-llama/Llama-3.1-8B`, that is 128,256 entries for each position. Use it for entropy, for the divergence between two models, or for the tail of the distribution. To rank candidates or to score a text, a number k is enough.
```bash
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 1, "prompt_logprobs": -1}'
```
Inference holds every entry in memory, so the API takes `-1` for at most 256 prompt tokens in all × `n`, over every prompt of the request. With `output_residual_stream`, the limit is 16. The API counts a text's tokens with the model's tokenizer, as the Hold does, and one token for each token id.
A larger request gets 400 with `param` `prompt_logprobs` before it holds any tokens. [Size caps](/docs/completions#size-caps) lists this cap with the others.
### Get the whole vocabulary for generated tokens [#get-the-whole-vocabulary-for-generated-tokens]
`logprobs` returns at most the 100 likeliest tokens at a generated position. To get every token, send the generated text back as part of a prompt:
1. Send the prompt, and keep the generated text and the `usage`.
2. Send the prompt and the generated text together as one new prompt, with `prompt_logprobs: -1` and `max_tokens: 1`. The new prompt must fit the limit above.
3. Check that the new `usage.prompt_tokens` is the first request's `prompt_tokens` plus its `completion_tokens`. Inference tokenizes the text again, and a text can split into other tokens than the ones generated.
4. Read the new `prompt_logprobs` from position `prompt_tokens` of the first request on. Each of these positions holds a generated token, with every token of the vocabulary.
This program does the four steps for two generated tokens, and prints each generated position with the number of tokens in its entry:
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
prompt = "The sky is"
first = client.completions.create(
model="llama-8b", prompt=prompt, max_tokens=2, temperature=0
)
text = prompt + first.choices[0].text
again = client.completions.create(
model="llama-8b",
prompt=text,
max_tokens=1,
extra_body={"prompt_logprobs": -1},
)
start = first.usage.prompt_tokens
end = start + first.usage.completion_tokens
assert again.usage.prompt_tokens == end, "the text split into other tokens"
for position in range(start, end):
print(position, len(again.choices[0].prompt_logprobs[position]))
```
## Return the prompt with the Completion [#return-the-prompt-with-the-completion]
Set `echo: true` to get your prompt back in `choices[i].text`, in front of the generated text. No separator comes between them.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "echo": true}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, echo=True
)
print(repr(completion.choices[0].text))
print(completion.usage.prompt_tokens, completion.usage.completion_tokens)
```
The Python program prints the text, then the prompt tokens and the generated tokens of the usage:
```text title="Python output"
'The sky is often beautiful'
4 2
```
The text starts with your prompt, without the BOS token. `usage.prompt_tokens` counts the BOS token that `meta-llama/Llama-3.1-8B` adds, and `usage.completion_tokens` counts only the generated tokens. For a text prompt, the generated text alone is `text[len(prompt):]`.
With `echo` and `logprobs`, each list in `choices[i].logprobs` starts with one entry for each prompt token, then has one for each generated token. The first prompt token, BOS on `meta-llama/Llama-3.1-8B`, has `null` in `token_logprobs` and in `top_logprobs`, because no token comes before it. These prompt entries count toward the cap on the prompt's log probabilities in [Size caps](/docs/completions#size-caps).
## Next steps [#next-steps]
[Score a text](/docs/cookbook/score-a-text) adds up the log probabilities of a text to get its log-likelihood. [Read Activations](/docs/activations) returns the residual stream at the Layers you choose.
# Changelog (/docs/changelog)
Each entry lists the changes that you can notice as a User of the API or the App, by date, newest first.
## 2026-11-03 [#2026-11-03]
### New [#new]
- The App has a **Request more tokens** page, under **Usage** ([Ask for more tokens](/docs/app#ask-for-more-tokens)). You say how many tokens you need, for which models and which project, and how you will use them. An admin approves the request with a Grant, or rejects it with a note, and the page lists each request with its answer.
- From 90% of your tokens used, a notice under the title of each page of the App says how many are left, with **Request more**. It takes the place of the notes on **Usage** and **API Keys** that told you to email for a Grant.
- **Admin** has a **Token requests** tab, where an admin approves a request with a number of tokens, such as `50K`, or rejects it ([Decide token requests](/docs/app#decide-token-requests)). **Grant tokens**, **Set the Budget**, and **Set the monthly allowance** take a number written the same way.
- The API takes steering at chosen positions on every model. Give each Layer a list of vectors, with `position_indices`, or without them for positions 0, 1, 2, and so on ([Steer chosen positions](/docs/steering#steer-chosen-positions)).
- A Steering vector takes `position_spans`, the old platform's spans of positions. One vector for each Layer goes to each position of the spans ([Steer a span of positions](/docs/steering#steer-a-span-of-positions)).
- An admin can type a User's organization and Discord handle on the User's page ([Open a User's page](/docs/app#open-a-users-page)).
### Changed [#changed]
- When your own tokens cannot hold a request, the message of 429 `budget_exceeded` ends with the address of the form, `Ask for more tokens at https://v2.infra.acsresearch.org/usage/request.` ([Not enough tokens](/docs/errors#not-enough-tokens)). The code stays `budget_exceeded`. In the Playground, the note says how many tokens the request can use and links to **Request more tokens**, without the address.
- A Steering vector position that no forward pass of the request reaches gets 400 on `apply_steering_vectors`, before the request holds any tokens. Before, positions were refused ([Positions that a request reaches](/docs/steering#positions-that-a-request-reaches)).
- `llama-8b` and `llama-405b` take 131,072 tokens of prompt and generated text together, the whole context of Llama 3.1. Before, they took 8,192 ([Served models](/docs/models#served-models)). `trinity-truebase` keeps 8,192, the model's own context.
- With the longer context, a capture of `llama-8b` without `max_tokens` gets 400. Left out, `max_tokens` is the room left in the context, which passes the capture's size cap ([Requests that the API refuses](/docs/activations#requests-that-the-api-refuses)). Set `max_tokens` with every capture.
- `temperature` takes a value from 0 to 2. The old platform's API took up to 100, but Inference refuses a value above 2. So the API refuses it with 400 `invalid_request` before the request holds any tokens ([Request fields](/docs/completions#request-fields)). In the App, **Temperature**'s slider goes from 0 to 2, and the field marks a value outside the range.
- The API refuses the other sampling values that Inference refuses, with 400 `invalid_request` before the request holds any tokens ([How to sample](/docs/completions#how-to-sample)). These are `top_k` below -1, `min_p` outside 0 to 1, `presence_penalty` and `frequency_penalty` outside -2 to 2, and `repetition_penalty` of 0 or less. The API also refuses `n` above 1 at a `temperature` below 0.00001, where Inference samples greedily. Before, such a request woke the model and then failed. The App's fields mark these values too.
- The narrower sampling ranges, `top_p`'s of 2026-10-28 too, apply to new requests only. A Run or a Loom Node saved with a value outside them still lists, opens, and exports with that value. A Run, a Node, or a Job that waits to run with such a value runs, and Inference decides on the value. In the Playground, **Use these settings** puts the value in the form, which marks it ([Find your Runs again](/docs/app/playground#find-your-runs-again)).
- Jobs have the routes and the shape of the old platform's API, so code written for its Jobs runs unchanged ([Run a Job](/docs/jobs)). `POST /v1/completion-jobs` takes one body of `POST /v1/completions` and answers 202 with a `Location` header. `GET /v1/completion-jobs/{job_id}` gives `status` `queued`, `running`, `succeeded`, `failed`, or `cancelled`, ISO times, `status_url`, and `result_url`. `GET /v1/completion-jobs/{job_id}/result` answers the Completion, or the error with the status that `POST /v1/completions` would have answered. A cancel of an ended Job answers the Job. `POST /v1/jobs` and its routes are gone.
- Harvests have the routes and the shape of the old platform's API, so its polling loop runs unchanged ([Harvest Activations](/docs/harvest)). `POST /v1/harvest` answers 202 with `job_id`, `n_prompts`, and `reserved_tokens`. `GET /v1/harvest/{job_id}` gives `status` `pending`, `running`, `done`, `failed`, or `cancelled`, ISO times, `completed_at`, `error` as a message, and, once `done`, `result` with the links of the files. `DELETE /v1/harvest/{job_id}` cancels a harvest, and a `done` or `failed` one gets 409 `harvest_not_cancellable`. `POST /v1/harvests` and its routes are gone.
- An `Idempotency-Key` sent again with another request gets 409 `idempotency_conflict` with `x-should-retry: false`, in place of 400. A full Job cap is 429 `job_capacity`, and an expired Job result is 410 `job_result_expired`.
- The API takes the `X-Request-Id` that you send as the id of the request, as the old platform does ([Headers](/docs/cheatsheet#headers)). It must have 1 to 64 letters, digits, `-`, or `_`.
- A Steering vector's `activations` can be encoded as the old platform's `encode_vector` writes them ([Encode a Steering vector](/docs/steering#encode-a-steering-vector)). The numbers are bfloat16 as `int16`, or float32, in zstd and base64. An encoded vector takes about a tenth of the bytes of JSON numbers, so eight vectors over all 32 Layers of `meta-llama/Llama-3.1-8B` fit the body cap. The API checks it as it checks numbers, and sends it to Inference as it is.
- The API counts a text prompt's tokens with the model's own tokenizer, as Inference does ([The Hold](/docs/concepts/budgets#the-hold)). Before, it counted the UTF-8 bytes. A request without `max_tokens` gets the context that its tokens leave. A 9,000-character prompt to `trinity-truebase` gets about 6,190 tokens, where it got 1. The Hold, the cut near the end of your Budget, and the caps of a capture and of `prompt_logprobs: -1` use this count too.
- `prompt_logprobs: -1` takes several prompts of at most 256 tokens in all × `n`, and keeps 256 with Steering vectors. Only `output_residual_stream` lowers it to 16 ([Size caps](/docs/completions#size-caps)).
- The API's address sends the old platform's page addresses to their pages. `/docs` and its pages and `/llms.txt` lead to these docs, and `/swagger` to the [API reference](/docs/reference). The old example downloads lead to the [Cookbook](/docs/cookbook), and `/usage` to the App's **Usage** page. An old page such as `/docs/api` or `/docs/bulk-harvest` leads to the page that took its place.
### Fixed [#fixed]
- Inference's answer can miss a choice, or have `activations` that are missing or of another shape than the request asked for. The API then answers 502 `upstream_server_error`, and the request costs nothing. Before, such an answer came back as a 200 and cost its tokens. In a stream, the error comes as the last event, and you pay for the tokens before it ([Decode the tensor](/docs/activations#decode-the-tensor)).
- A Loom Node that fails keeps the text that it grew and its own `error`. This is also true when the API could not save its end at the first try. Before, a Node that failed early lost its text, and its `error` said only that the server had an error ([Read a failed Node](/docs/loom#read-a-failed-node)).
- In **Compare**, a model whose Run cannot start shows why in its own place ([Compare models and settings](/docs/app/playground#compare-models-and-settings)). Earlier Runs no longer stay in view beside the new ones.
- A Run says which stop sequence ended it, or that the model ended the text, also when **Top logprobs shown** is 0.
- The likeliest tokens at a token stay on the screen, and the list scrolls when it is long.
- Select a Loom's title to rename it ([Open or start a Loom](/docs/app/loom#open-or-start-a-loom)). **Rename** left the buttons by the title.
- On **Admin**, an Application's unanswered questions show as **Not answered**.
- An email link in the docs and in the App opens your mail app without a blank tab.
## 2026-11-02 [#2026-11-02]
### Changed [#changed-1]
- The API takes the API keys of the old platform, which start with `acs-bm-`, as well as keys that start with `sk-acs-` ([Use API keys](/docs/api-keys#send-the-key)).
- Requests and `GET /v1/models` name each Base model by the id of the old platform: `llama-8b`, `llama-405b`, and `trinity-truebase` ([Models](/docs/models)). A request without `model` runs on `llama-8b`. Each answer's `model` still names the weights that ran it, such as `meta-llama/Llama-3.1-8B`.
- `GET /v1/models` lists each model in the old platform's shape: its `id`, its `served_model_name`, its `gpu_shape`, `default`, and `capabilities` with `n_layers`, `hidden_size`, `max_activation_prompt_tokens`, and the limits of a request ([Model fields](/docs/models#model-fields)). The App keeps its own list. `GET /v1/models/{model}/status` gives `boot` with `stage`, `stage_label`, and `stage_age_s`, and the same in `activation_boot` ([Check whether a model is Warm](/docs/models#check-whether-a-model-is-warm)). Both routes take a `served_model_name` with its slash, such as `/v1/models/meta-llama/Llama-3.1-8B`.
- Each choice of a Completion has `stop_reason`, the `stop` text that ended it, in an answer and in a stream's chunks ([Response fields](/docs/completions#response-fields)). `stream_options.continuous_usage_stats` alone puts the usage so far in each chunk of a stream, without `include_usage`.
- A stream has a `: keepalive` comment line every 5 seconds, as the old platform's streams do. A client's read timeout then does not cut a stream while Inference is silent ([Requests with a stream](/docs/concepts/long-requests#requests-with-a-stream)).
- Every error of the API has the error object of the old platform's API. Code written for that API branches on the same `error.code` ([Status codes](/docs/errors#status-codes)). `request_id` is at the top of the body, beside `error`. A request that the Budget cannot hold is 429 `budget_exceeded`, a rate limit 429 `rate_limited`, and a full line of Completions 429 `queue_full`. A request with no key is 401 `missing_authorization`, and a revoked or paused key is 401 `key_revoked` or `key_disabled`. An unknown model is 400 `model_not_found`, so the `openai` SDK raises `BadRequestError` for it.
- A failure of Inference has the type `upstream_error` and a `code` that names it, such as 502 `upstream_unreachable`, `vllm_oom`, or `upstream_server_error`, with `retryable` and `retry_after_seconds` ([Answers about Inference](/docs/errors#answers-about-inference)). Inference's own 500 is now 502, and its timeout is 504 `upstream_deadline`.
- `POST /v1/completions` takes the values that the old platform's API takes ([Request fields](/docs/completions#request-fields)). `max_tokens: null` is the same as leaving it out, and `output_residual_stream: false` reads no Activations. `output_residual_stream` takes its Layers in any order and with a Layer twice. The Activations hold each Layer once, in ascending order. `priority` takes `null`, a text of digits such as `"5"`, and `true` or `false`.
- The API reads the body of a `/v1` request as JSON whatever its `Content-Type`, as the old platform's API does. `curl -d` without `-H 'Content-Type: application/json'` works, and a body of `text/plain` no longer gets 415. A body sent in chunks, without `Content-Length`, no longer gets 411. The API checks the key first, then reads the body and answers 413 `request_too_large` when it passes the [body cap](/docs/limits#request-size).
- `GET /health` answers in the old platform's shape: `status`, `wrapper` with `uptime_seconds` and the `database`, `backends` with each model's `model_id`, `gpu_shape`, and breaker, `degraded_backends`, and `degraded_reasons`. It still answers 503 only when the database does not answer.
## 2026-10-30 [#2026-10-30]
### New [#new-1]
- Each row of the Playground's **Runs** table ends with a menu, in place of **Delete**. It has **Show**, **Use these settings**, **Copy settings**, **Copy prompt**, **Copy output**, **Copy as API request**, and **Delete**. **Copy as API request** copies a curl command for `POST /v1/completions` that reads your key from `$ACS_API_KEY` ([Find your Runs again](/docs/app/playground#find-your-runs-again)). The menu stays at the right edge of the table when the table scrolls sideways.
- The **Models** tab of **Admin** keeps models Warm ([Keep a model Warm](/docs/app#keep-a-model-warm)). An admin sets hours each week, or keeps a model Warm at once for 1 to 24 hours. The tab shows each model's **Floor** and what its windows cost.
### Changed [#changed-2]
- In the App, **Support** has **Email**, **Discord**, **Feedback**, and **Help**. **Discord** opens the invite to ACS's Discord server in a new tab. Before, it was called **Community** and opened a page with **Join the Discord server**. Old links to that page open the invite ([Discord](/docs/support#discord)).
- **Feedback** under **Support** opens **Send feedback**, which was a button of its own in the navigation ([Send feedback](/docs/app#send-feedback)). **Help** opens [Support](/docs/support) in these docs. Before, it was called **Help and FAQ**.
- **Profile** no longer shows your role or your token budget. **Usage** shows your token budget and monthly allowance ([Read your usage](/docs/app#read-your-usage)).
- **Send feedback** sends your notes to Linear, the team's issue tracker, with your account's email. The team can then answer you. An anonymous note goes without your email ([Send feedback](/docs/app#send-feedback)).
- The **Feedback** tab of **Admin** links each note to its Linear issue.
- **API Keys** lists only your active and paused keys. A revoked key leaves the page, and the line of revoked keys at the bottom is gone. With no such key, the page says **No API keys yet**, also when you once had keys. **By key** on **Usage** still shows a revoked key's tokens under its name ([Revoke a key](/docs/api-keys#revoke-a-key)).
- While **API Keys** loads, it shows the head of its table and three gray rows. With keys, the table stays in place when the list arrives, and **New key** shows in the header. With no key, the page shows **No API keys yet** and **New key** in its middle, with no table. The notes, such as `No tokens left.`, show below the table, or at the bottom of the page with no key. A key's name shows on one line, cut short when it does not fit.
- **Jobs** shows **Run Jobs from a script** in the middle of the page, as **API Keys** does with no key.
- **API Keys** no longer counts your keys beside its title, such as `3 of 10 keys`. At 10 keys, **New key** is still off, and the note below the table says why ([Manage your keys](/docs/api-keys#manage-your-keys)).
- The **Analytics** tab of **Admin** has five tabs: **Users**, **Budget**, **Platform**, **Usage**, and **Spend**. The days at the top of **Analytics** apply to each of them. **Usage** and **Spend** were tabs of **Admin**, and old links to them open their tab in **Analytics**. The **Section** filter is gone ([Read the platform's analytics](/docs/app#read-the-platforms-analytics)).
- On **Admin**, **Applications** shows the day of each Application under **Submitted**, and the day and the admin of its decision under **Reviewed**. **Feedback** shows the day of each note under **Submitted**. Before, these columns were **Received**, **Decided**, and **Sent**.
## 2026-10-29 [#2026-10-29]
### New [#new-2]
- Harvests read the residual stream of up to 4,096 prompts at the Layers that you choose, on GPUs of their own. `POST /v1/harvests` queues one, and `GET /v1/harvests/{harvest_id}` reads it, with `wait` to wait for a change. `GET /v1/harvests/{harvest_id}/results` gives links to its safetensors files, and `POST /v1/harvests/{harvest_id}/cancel` cancels it ([Harvest Activations](/docs/harvest)).
- The files of a harvest have the old platform's layout, version 2, so programs that read those files read these too. Each answer of the results route makes new links that work 7 days, and the API keeps the files ([Kept data](/docs/limits#kept-data)).
- A harvest holds its prompts' tokens while it runs. It pays the tokens that Inference read when it completes, and nothing when it fails ([Holds of background work](/docs/concepts/budgets#holds-of-background-work)). The App's **Usage** page shows its tokens under the source **Harvests**.
- Each API key may start 200 harvests a month. Each User may have 3 harvests on Llama 3.1 8B `queued` or `running`, and 1 on Llama 3.1 405B or Trinity Large ([Harvests](/docs/limits#harvests)).
- The Cookbook's recipes link to their whole programs as files to download. [Run a Job from a shell script](/docs/cookbook/job-script) links to `job.sh`, for example.
- The Cookbook has [Evaluate with Inspect](/docs/cookbook/evaluate-with-inspect). It runs an Inspect eval through `POST /v1/completions`, with a task to download.
- The Cookbook has [Run the workshop notebook](/docs/cookbook/workshop). Its Jupyter notebook goes from a first Completion to a steered Base model.
- An admin's **Admin** page has an **Analytics** tab with the platform's numbers ([Read the platform's analytics](/docs/app#read-the-platforms-analytics)). It counts Users, monthly allowances, refusals, server errors, sources, and hours. The numbers that were above the tabs moved into it.
- **Admin** has a **Spend** tab with the GPU spend of each model and day, and who used each model in 48 hours ([Watch the spend](/docs/app#watch-the-spend)). Its **Lock** and **Clear** pause and resume new GPU work. **Analytics** shows the GPU spend of its days.
- The API pauses new GPU work when the GPUs cost $3,000 in 24 hours, and an admin can pause it too. While the pause is on, a Completion, a Job, or a Branch that you start gets 429 `spend_cap` with `Retry-After: 3600`, and queued work waits. Work that runs finishes ([Spend limit](/docs/limits#spend-limit)).
- An admin can put tags on Users, such as `hiring-2026-08`, and an Invite can give its new User a tag. **Users** on **Admin** lists each User's tags and has a **Tag** filter ([Group Users with tags](/docs/app#group-users-with-tags)).
- **Users** on **Admin** shows each User's organization, engagement, active weeks, and models, and has an **Engagement** filter ([Manage Users](/docs/app#manage-users)).
- An admin can select many Users on **Users** and act on all of them at once ([Act on many Users at once](/docs/app#act-on-many-users-at-once)). The actions are suspend, restore, tag, set the monthly allowance, and delete.
- Each User has a page on **Admin**, which their email on **Users** opens. It has their account, tags, Budget and usage, API keys, Application, and notes ([Open a User's page](/docs/app#open-a-users-page)).
### Changed [#changed-3]
- On **API Keys**, **New key** opens a dialog for the key's name, then shows the key once in **Copy your key**. The page title counts your keys that are not revoked, such as `3 of 10 keys`. At 10, **New key** is off, and the page says why. Before, a form at the top of the page made a key ([Make an API key](/docs/api-keys#make-an-api-key)).
- Each key on **API Keys** has one menu for **Rename**, **Pause** or **Resume**, **View usage**, and **Revoke**. Before, each row had three buttons. A key's tokens this month open **Usage** for that key. Revoked keys fold into one line at the bottom of the page, such as `2 revoked keys`. A time today shows in UTC, such as `05:02 UTC`, and an earlier one shows its date ([Manage your keys](/docs/api-keys#manage-your-keys)).
- **API Keys** no longer shows the tokens of your month and your Budget above the table. **Usage** shows them ([Read your usage](/docs/app#read-your-usage)).
- The App's **Usage** page starts with four numbers. They are the tokens used, the Completions, and the tokens left in the monthly allowance and in the token budget. At the top, choose the days to count, up to 366 UTC days, and one key or one model. The address keeps them. **Tokens per day** charts each day's input and output tokens. **By model** and **By source** split the tokens. Before, the page counted this month and the last 30 days only ([Read your usage](/docs/app#read-your-usage)).
- **Usage** shows each Completion's source and key. Its status is Running, Completed, Failed, or Stopped. Before, it was running or settled. **Usage** no longer shows the tokens that running Completions hold, because the tokens left already leave them out.
- **Usage** shows how each Completion's tokens were counted, **Reported**, **Partial**, or **Held**, and how long the Completion took ([Read your usage](/docs/app#read-your-usage)).
- **By source** on **Usage** counts each source's Completions and its input and output tokens.
- **By key** on **Usage** shows each key's tokens last month and in all, and its own Budget. Select a key's name to count that key alone.
- With one key chosen, **Usage** shows the tokens left in that key's own Budget, when it has one. The meter of the monthly allowance turns orange from 90% used, and red once the allowance is spent.
- **API Keys** shows the tokens left in each key's own **Budget**. It reads its numbers again every minute, and keeps them when a read fails. When you have no tokens left, it says `No tokens left.` and whom to ask for a Grant ([Manage your keys](/docs/api-keys#manage-your-keys)).
- **Copy your key** asks before it closes while you have not copied the key, and before the tab closes. It links to [Set the client for long requests](/docs/retries#set-the-client-for-long-requests).
- **Usage** has a **Refresh** button beside **Updated**. When a read fails, the page keeps the numbers that it read last and says `Could not refresh`. Before, the error took the place of the numbers. The **Usage** and **Analytics** tabs of **Admin** do the same.
- The App says input tokens and output tokens where it said prompt tokens and completion tokens. A Run's line and the **Read Layers** table use the new words. The API's fields `prompt_tokens` and `completion_tokens` keep OpenAI's names.
- The App shows the logo of each model's maker before the model's name. It does so in the Playground's model list, **Models**, **Runs**, a Run's heading, and on **Usage**.
- The App names each model by a short name, such as Llama 3.1 8B, Llama 3.1 405B, or Trinity Large. The model's full id shows when you hover over its name. Exports and the page's address keep the full id.
- A Run in the App's Playground leaves the prompt as you wrote it, and its text shows under the prompt. **Continue the text** is gone. Before, a Run of one choice wrote its text into the prompt as it streamed. To go on from a Run, copy its text into the prompt ([Run a prompt](/docs/app/playground#run-a-prompt)).
## 2026-10-28 [#2026-10-28]
### Changed [#changed-4]
- `top_p` must be more than 0 and at most 1, as OpenAI and vLLM take it. Another value gets 400 with `param` set to `top_p` before the request reaches Inference, and costs nothing ([Strict validation](/docs/completions#strict-validation)). A Loom Branch checks it too, when you make the Branch ([Branch under a Node](/docs/loom#branch-under-a-node)).
- Each Run of the App's Playground keeps the number that it got when it started, after a reload and after a delete. The newest Run shows when you come back, and a delete asks you to confirm. The Runs table shows all its columns, and the page scrolls its rows ([Find your Runs again](/docs/app/playground#find-your-runs-again)).
- **Playground** in the App's navigation goes back to the settings and Compare that this browser used last. It does so until you sign out or sign in again ([Run a prompt](/docs/app/playground#run-a-prompt)).
- The Playground says a setting's limit in plain words, such as `Max tokens: at most 32,000.`. When the prompt and Max tokens do not fit in the model's context, it names the context in tokens.
- A Branch, a split, or a sibling that would give a Loom more than 2,000 Nodes gets a 409. Its message says how many more Nodes fit ([Change the tree](/docs/loom#change-the-tree)).
- **Import a Loom** in the App names the new Loom after the file's title, with ` (imported)` after it. The list then tells it from the Loom it came from. `POST /v1/looms/import` keeps the file's title, as before.
### Fixed [#fixed-1]
- A Loom that you open again sets **Branches** to the number of Nodes of its last Branch. It does so also after you split or delete one of them. Before, it often showed 1.
- A Loom whose last Branch asked for no log probabilities opens with **Top logprobs shown** at 5, so the next Branch has a heatmap. A Branch through the API may ask for none. Before, the box was empty.
- **Import a Loom** in the App says what is wrong with a file that is not a Loom file, such as `"nodes" is missing`. Before, it said `Missing key` and told you to correct a setting and select Branch.
- **Split at the cursor** splits a Node only where you put the cursor or selection inside its text. Before, `s` and then **Split at the cursor** cut at the start. The Node was left empty, and all its text went to the new child.
- A Loom page remembers **Tree** or **Graph** in every browser, also over plain HTTP. Before, a browser without the Cookie Store API opened the next Loom in **Tree**.
- A Loom's **Graph** marks a failed, stopped, or growing Node as the tree does. Its labels fill their column, with the whole first line on hover. Before, a failed Node read **No text yet** with no mark, and labels stopped after 18 characters.
- On a Loom page, a long title shows in full when you hover over it. **Your Looms** scrolls under its heading and fills the column beside **New Loom**. **Tree** and **Graph** stay in sight in a long tree. Before, the title stopped after about 20 characters, and the list was a box of 3 rows on every page. The switch scrolled away with the Nodes.
- An Invite's page says that its form sets you up and approves you at once. It no longer speaks of a review. After you send its form, you choose your password. No approval email follows, because its link would be used already.
- When your session ends, as when your access is suspended, the next page that you open in the App takes you to sign-in. Sign-in says that your session has ended, and opens the page again after you sign in. Before, the page stayed open with only an error box.
- After too many tries to sign in, the sign-in page says how many seconds to wait. It keeps the email that you typed.
- On **Usage** scoped to one key, the monthly allowance and the token budget say that they are for all keys. Their labels read **Your account's monthly allowance** and **Your account's token budget**.
- Each **Copy** in the App, such as a new key's, works when the browser does not let the App copy, as over plain HTTP. The App then shows the text selected, for you to copy by hand. Before, nothing was copied, and nothing on the page said why. The Loom's **Copy text** showed an error page.
- In the App, Cmd+B or Ctrl+B opens and closes only the navigation, not the settings of the Playground or a Loom. On a phone, a new key's dialog keeps a long key inside its edges.
## 2026-10-27 [#2026-10-27]
### New [#new-3]
- Sign in to the App with GitHub, Google, or Hugging Face, when the App shows its button. The provider must have verified the email address that you use for ACS Infra. A provider does not give you access, so apply for access first. On **Profile**, link and unlink each provider ([Sign in](/docs/get-access#sign-in)).
- **Send feedback** in the App's navigation sends the team a bug, a feature request, or another note. It takes up to 5,000 characters and 5 screenshots. Unless you send it anonymously, the App's recent errors and request ids go with it, and the dialog shows them first.
- An admin can invite you by email. The email's link opens the Apply form for your address. You are approved as soon as you send the form, and go straight on to set your password. The link works once, for 7 days.
- Every request that makes or changes something takes an `Idempotency-Key` header. The same request sent again with the key does its work once and is charged once. It gets the first answer, or 409 `no_answer` for a stream ([Send a request again safely](/docs/retries#send-a-request-again-safely)).
- A page in a browser can call `/v1` with your API key. The API answers any origin with CORS, with no cookie. [Call the API from a browser](/docs/api-keys#call-the-api-from-a-browser) says how to keep the key safe.
- Each approved User has a monthly allowance of 5,000,000 tokens. A Completion spends this month's allowance while it fits, then your Budget. The allowance starts again on the first day of each month (UTC), as [Your monthly allowance](/docs/concepts/budgets#your-monthly-allowance) explains.
- `logprobs` takes up to 100 top tokens.
- `"output_residual_stream": true` reads every Layer of the model ([Read Activations](/docs/activations)).
- `stream_options.continuous_usage_stats` with `include_usage` puts the usage of each chunk's choice so far in each chunk.
- `GET /v1/models` lists with each model the `limits` that the API enforces on it ([List the models](/docs/models#list-the-models)).
- A JSON answer of 500 bytes or more comes gzipped when your client asks for it, as the `openai` SDKs do.
- A Completion that is not streamed and runs longer than 150 seconds gets 200 at once. Spaces follow every 5 seconds, then its JSON or its error ([Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream)).
- The App's Playground keeps its settings in its link. The link holds each model of Compare with its settings, Single or Compare, the Heatmap, and Continue the text. A reload, or a link that you send to another User, opens the same Playground. The link never holds your prompt, a Run, or a Steering vector. Your prompt stays in your browser until you sign out.
- In an empty prompt box of the Playground or of a new Loom, Tab puts in the gray example. The next Tab moves on.
- The App remembers when you close the navigation or the Playground's settings.
- The App's **Usage** page shows this month's tokens and your monthly allowance, and what running Completions hold. It has a chart of the last 30 days with its table, each key's tokens, and the tokens of each Surface. It shows the history of your Budgets, with the date and the amount of each Grant. `?key=` shows one key only. The page reads again every minute while it is open.
### Changed [#changed-5]
- The App's Loom page has the tree changes and the keys `g`, `e`, `s`, `i`, `Delete`, and `?`. It has **Stop**, a graph view with zoom and fit, and the likeliest tokens at a token. Its **Export** writes the Loom file shape of the old platform. It also has **Path as text** and **Import a Loom** ([Use the Loom page](/docs/app/loom)).
- The App's **Loom** opens the Loom you changed last, and lists your Looms beside each Loom, pinned first. It has **New Loom**, **Rename**, and **Pin**. A Loom opens with the settings of its last Branch.
- Export a Loom with `GET /v1/looms/{loom_id}/export`, and make one from a file with `POST /v1/looms/import`. The file has the shape of the old platform's Loom export. Files of either then import into the other ([Export and import a Loom](/docs/loom#export-and-import-a-loom)).
- Edit, split, and write Nodes of a Loom, and give it a new root. A Loom holds at most 2,000 Nodes ([Change the tree](/docs/loom#change-the-tree)). A Loom lists its Nodes depth first, each before its next sibling.
- Stop a Loom's growing Nodes with `POST /v1/looms/{loom_id}/cancel`. Each becomes `cancelled`, keeps the text it grew, and pays for it ([Stop a Branch](/docs/loom#stop-a-branch)).
- Rename or pin a Loom with `POST /v1/looms/{loom_id}`. A Loom answers its `title` and `pinned`, and its first Branch names it after its root. `GET /v1/looms` lists pinned Looms first, then the others by their last change ([Rename or pin a Loom](/docs/loom#rename-or-pin-a-loom)).
- A Branch of a Loom without a `seed` draws one. Each Node keeps its own `seed + i` in its `request`, so you can sample any Node again ([Branch under a Node](/docs/loom#branch-under-a-node)).
- A Branch of a Loom takes `n` from 1 to 8 and `max_tokens` up to 1,000. More gets 400 with `param` set to the field ([Branch under a Node](/docs/loom#branch-under-a-node)).
- On **API Keys** you can rename, pause, and resume a key, and a revoked key stays listed. Two of your live keys cannot share a name. A paused key gets 401 `This API key is paused. Resume it on API Keys in the App to use it again.` A revoked one gets `This API key has been revoked.` Before, both got `Incorrect API key provided.`
- The Apply form gives every Application the same answer. While you have an open Application, a second one from your email is not stored, and the first stays as it was. Before, it got 409.
- The API runs up to 4 Jobs at once, not one. While a User has a Job running, a newer Job of another User starts before that User's next one.
- The API waits up to 14 minutes for Inference, not 600 seconds.
- The Playground's heatmap colors a likelier token darker. Before, a less likely token was darker.
- A model that the API does not serve gets 404 `model_not_found` before the API holds any tokens. The message names the models it serves.
- Steering at chosen positions gets 400 on `apply_steering_vectors` until its test on a GPU passes. That is `position_indices`, or a list of vectors for each Layer ([Steer chosen positions](/docs/steering#steer-chosen-positions)).
- `POST /v1/chat/completions` gets 400 with a message that names `POST /v1/completions`. Before, it got 404.
- A negative `seed`, `stream_options` without `stream: true`, and more than 33,554,432 top log probabilities of the prompt in one request get 400 ([Size caps](/docs/completions#size-caps)).
- When Inference refuses the API's own request, the 502 has the header `x-should-retry: false`, so the `openai` SDK does not send the request again.
- Runs, Jobs, and Branches that you start in the App count against a limit of 600 a minute for each User.
- When Inference refuses or fails a Completion before its first token, a timeout included, the API gives its [Hold](/docs/concepts/budgets#the-hold) back. You pay nothing, and the Usage page leaves it out. Before, the Hold was settled at 0 tokens. A Job's request that times out costs nothing.
- A Completion that stops before the API sends it to Inference spends nothing. The running request of a Job that you cancel before it reached Inference spends nothing too.
- A Job or a Branch made with an API key that is paused waits in the queue, holding no tokens. It runs when the key is resumed.
- `POST /v1/completions` without `max_tokens` uses the model's context left after the prompt, at most 32,000. Before, it used 16. A request with more than one prompt, or with `n` above 1, must set it. Jobs and Looms keep 16.
- Near the end of your tokens, `POST /v1/completions` lowers `max_tokens` to what fits and runs, with the header `X-Acs-Max-Tokens-Clamped`. It answers 429 `insufficient_quota` only when not even one token for each choice fits ([Near the end of your tokens](/docs/concepts/budgets#near-the-end-of-your-tokens)).
- `POST /v1/completions` without `stream` and with a `max_tokens` above 15,600 gets 400 `deadline_infeasible` and `X-Should-Retry: false`. It cannot finish within the API's 14-minute wait. A `max_tokens` that you leave out is lowered to 15,600 instead.
- A 429 `insufficient_quota` says which limit refused the request. The message names your monthly allowance and your Budget, with the day the allowance refills, or the API key's own Budget.
- When the API's database is busy or out of reach as a Completion ends, you still get your answer. The API records what it spent a moment later.
- A Job or a Node whose API process crashed runs again. Within about 18 minutes, the next API process gives back the tokens that the crash held, and queues it again. A Job does not pay again for a request that already has its result. Before, the Job or the Node stayed `running` with its tokens held.
- A Job says when the API started it (`started_at`), when it ended (`ended_at`), and when its results expire (`results_expire_at`). Its results answer 202 with the Job and `Retry-After: 1` while it is queued or running. Before, they answered 409. The API keeps the results for 30 days after the Job ended, then answers 410 ([Download the results](/docs/jobs#fetch-the-result)).
- Jobs have caps. An API key may have 8 queued or running, a User 32, and all Users 128. A User may have 100 completed Jobs with kept results, and all Users 1,000. One more gets 429 `rate_limit_exceeded` ([Limits](/docs/limits#jobs)).
- A Job keeps at most 64 MiB of results. Once its lines reach that, its later requests do not run, and each gets an error line ([Limits](/docs/limits#jobs)).
## 2026-10-26 [#2026-10-26]
### New [#new-4]
- Every error answer has the id of its request in `error.request_id`, the same id as in its `X-Request-Id` header ([The error object](/docs/errors#the-error-object)). The error event that ends a stream has it too. A Job's result line and a Node's error have no request id.
- When Inference does not answer within 600 seconds, the 502 `Could not reach Inference: Request timed out.` has the header `x-should-retry: false`. The `openai` SDK then does not send the request again. Your client gets this answer only with a timeout above 600 seconds, for example 660 seconds.
### Changed [#changed-6]
- The API's own 500, `The server had an error while processing your request.`, has the header `x-should-retry: false`. The `openai` SDK then does not send the request again. The API does not know how much of the work it did before the error. Another try could run a Completion or a Job two times. Before you send the request again, look for its result, as [Which errors to try again](/docs/retries#which-errors-to-try-again) says.
### Fixed [#fixed-2]
- When Inference answers with a redirect that the API cannot follow, the API answers 502, as for another refusal of its own request. Before, a Completion got 500, and a stream ended with `Inference ended the stream before it was complete.`
- A text that holds the character U+0000 (NUL) gets 400 `invalid_request_error`, with the message `A text cannot hold the character U+0000 (NUL).` and its field in `param`. The check covers a prompt, a `stop` text, `model`, `user`, a Loom's prompt, a field of the application form, and an email address. Before, a Job, a Loom, a Branch, and the forms answered 500, because the database cannot store the character. A Completion with it gets 400 too, because a Job's requests are Completion requests.
## 2026-10-24 [#2026-10-24]
### Changed [#changed-7]
- `prompt_logprobs: -1` takes one prompt of at most 256 prompt tokens × `n`, or 16 with Activations or Steering vectors. The API counts the tokens as the Hold does. It refuses a larger request, or one with several prompts, with 400 and `param` `prompt_logprobs`, before Inference. Before, only Inference refused it, for a prompt of more than 1,024 tokens, and one request could have several such prompts. Refer to the [size caps](/docs/completions#size-caps).
## 2026-10-12 [#2026-10-12]
### Changed [#changed-8]
- The rate limit counts each API key, and it is 600 requests a minute instead of 60. Each key has two limits of 600 a minute. One is for the requests that start a Completion, a Job, or a Branch, and one is for all other requests. Reading a Job or a Loom then never uses up the limit for Completions. A new key adds requests.
- When an API key has 16 Completions running, the next ones wait for a place instead of getting 429. Up to 64 Completions wait, and they run in the order they came. A waiting Completion holds no tokens until it runs. If you close the connection, the Completion leaves the line and never runs. With 64 waiting, one more gets 429 `rate_limit_exceeded` with `Retry-After: 5`.
- Only Completions count toward the 16 in flight. Jobs, Looms, and the other routes answer at once while 16 Completions run.
- The App's calls have no rate limit, because the limits belong to API keys.
## 2026-10-11 [#2026-10-11]
### New [#new-5]
- An API key has at most 16 requests in flight. One more gets 429 `rate_limit_exceeded`, with no `Retry-After`.
- A User has at most 10 API keys. The App refuses an 11th one with 409.
- Every answer has the header `X-Request-Id`, a new id for each request ([The error object](/docs/errors#the-error-object)). Send it when you [report a bug](/docs/support#report-a-bug).
- A request body is at most 8 MiB, on every route. A larger body gets [413](/docs/errors#status-codes), and a body without a `Content-Length` header gets 411.
- One request has [size caps](/docs/completions#size-caps). It asks for at most 32,000 generated tokens, `prompt_logprobs` from -1 to 100, and at most 4 `stop` texts. It captures at most 2^25 numbers, and has at most 8 Steering vectors of 2^21 numbers. A request over a cap gets 400.
- When an admin approves your Application, you get the email "Your ACS Infra application is approved". Its link to choose your password works for 7 days. Before, approval sent the password reset email, whose link works for 1 hour.
- The App no longer makes a User from its own form. To get access, apply with the form on the ACS Infra website, as [Apply for access](/docs/get-access#apply-for-access) shows.
- An admin can [suspend your access](/docs/get-access#if-your-access-is-suspended). Your API keys then get 401 `invalid_api_key`. Its message is `Your access to ACS Infra is suspended. Write to infra@acsresearch.org if you think this is a mistake.` The Completions of your queued Jobs and Nodes fail with the same error, so they do not reach Inference.
### Changed [#changed-9]
- The rate limit counts each User, not each API key. All your keys and the App share a burst of 60 requests, then 1 more each second. A new key adds no requests.
- [`GET /v1/models`](/docs/models) lists the models from the API's settings, so it never waits for Inference to start. Each model also has its Layers (`num_hidden_layers`) and its `hidden_size`, and its `max_model_len` is never `null`. The list fails only with 401 or 429.
- The App's [Playground](/docs/app#the-apps-pages) replaces its Completions and Activations pages. It runs one Completion with every setting of the API, and colors each token by its probability. It reads the Activations at the Layers that you choose, and adds a Steering vector.
- The approval, password reset, and email verification emails have the look and the words of the ACS Infra website.
### Fixed [#fixed-3]
- The Playground shows the text that the API returns. Before, it could show a stop string that the API had cut from the text.
- When you sign in after a session ended, the App keeps nothing from the User before you in that browser tab. Before, it could show that User's new API key and prompt.
## 2026-10-10 [#2026-10-10]
### New [#new-6]
- Every `/v1` route needs an [API key](/docs/api-keys#send-the-key) in the header `Authorization: Bearer `. A request without a key, or with a wrong or revoked key, gets 401 with the code `invalid_api_key`.
- You [make API keys in the App](/docs/api-keys#make-an-api-key), and every key starts with `sk-acs-`. The **API Keys** page shows each key as `sk-acs-` and its next 4 characters.
- Jobs, Looms, and Nodes belong to the User who made them. The id of another User's Job, Loom, or Node gets [404](/docs/errors#status-codes).
- Each Completion spends tokens from your [Budget](/docs/concepts/budgets). A new User has 0 tokens until an admin adds tokens with a Grant.
- Before Inference runs a request, the API takes a [Hold](/docs/concepts/budgets#the-hold) of the most tokens that the request can use. If your Budget cannot hold them, the API answers 429 with the type and code `insufficient_quota`. The request does not reach Inference.
- A text prompt holds 1 token for each byte of its UTF-8 text, and 1 more for the BOS token that Inference adds. With `add_special_tokens: false`, the prompt does not hold the extra token.
- A Completion that is not streamed and stops before Inference answers spends its whole Hold. The running request of a Job that you cancel also spends its whole Hold.
- An API key can have its own Budget, which limits that key further. An admin fills it with a Grant for that key.
- A Job or a Branch whose Budget cannot hold even one of its requests gets 429 `insufficient_quota` when you submit it, and the API saves nothing. Each request of a Job and each Node still takes [its own Hold](/docs/concepts/budgets#holds-of-background-work) when it runs.
- When you [revoke a key](/docs/api-keys#revoke-a-key), its queued Jobs and Nodes do not reach Inference. Each of their Completions fails with 401 `invalid_api_key` and the message `The API key that made this request was revoked.`
- Each API key has a rate limit on `/v1` of 60 requests a minute, as a burst of 60 and then 1 more each second. Over the limit, the API answers 429 with the type `rate_limit_error` and the code `rate_limit_exceeded`. The answer has a [`Retry-After`](/docs/retries#which-errors-to-try-again) header in seconds.
- To get access, apply with the form at `/apply` on the ACS Infra website. When an admin approves your Application, you get an email with a link to set your password.
- One email can send one Application. The form answers a second one with `An application from this email already exists.`
- If you forget your password, open `/forgot-password` in the App, and follow the link in the email to `/reset-password` ([Reset your password](/docs/get-access#reset-your-password)). A new password ends all your sessions.
- Signing in, and making a User in the App, allow 10 tries a minute from one IP address. More tries get 429.
- The docs have five sections: [API](/docs), [Cookbook](/docs/cookbook), [Tutorials](/docs/tutorials), Changelog, and [Support](/docs/support).
- The API section has new pages for [Authentication and API keys](/docs/api-keys), [Models](/docs/models), [Completions](/docs/completions), [Steering](/docs/steering), [Loom](/docs/loom), and the [Cheatsheet](/docs/cheatsheet).
- Each docs page has a button that copies the page as Markdown, and a menu that opens the page in ChatGPT, Claude, or Cursor.
### Changed [#changed-10]
- [`n`](/docs/completions#strict-validation) is at most 128 in a Completion and in a Branch. A larger `n` gets 400 with `param` set to `n`.
- A [Job](/docs/jobs#submit-a-job) has at most 1,000 requests. A longer list gets 400 with `param` set to `requests`.
- [`priority`](/docs/completions#other-fields) is 0 or more. A negative `priority` gets 400 with `param` set to `priority`.
- A 409, and a 429 with the code `insufficient_quota`, have the header `x-should-retry: false`. The `openai` SDK then raises the error without [trying the request again](/docs/retries#which-errors-to-try-again).
- [When the API stops](/docs/concepts/background-work#when-the-api-restarts), it queues a running Job or Node again. The Completion that it stopped gives back its Hold. When the API starts again, a Job runs only the requests that have no result line. You pay for each request once.
- The `text` of a running Node changes at most every 250 ms while you [follow a Branch](/docs/loom#follow-a-branch).
### Fixed [#fixed-4]
- A Job that you [cancel](/docs/jobs#cancel-a-job) while it starts again after an API restart no longer pays again for a request with a result.
- With `echo` and `max_tokens: 0`, the [Hold](/docs/concepts/budgets#the-hold) counts 1 generated token for each choice, because Inference generates and counts 1. Before, such a request could spend more tokens than its Hold.
- A request with [several prompts](/docs/completions#strict-validation) asks for at most 128 choices in all, `n` for each prompt. More gets 400 with `param` set to `n`, or to `requests` for a request in a Job.
- If Inference answers with a status outside 100 to 599, the API answers [502](/docs/errors#status-codes) with the type `server_error`.
- A model name such as `toString` or `constructor` gets 404 `model_not_found`, as does [any model that Inference does not have](/docs/errors#unknown-model).
## 2026-10-09 [#2026-10-09]
### New [#new-7]
- [`POST /v1/completions`](/docs/completions) continues a prompt with a Base model. It follows OpenAI's completions API, so the `openai` SDK works with it.
- A prompt is text or a list of token ids, and one request can [send several prompts](/docs/sampling#send-several-prompts).
- A field that you leave out gets OpenAI's default, not the model's own. For example, `max_tokens` is 16 and `temperature` is 1.
- [`logprobs`](/docs/token-probabilities#get-log-probabilities-for-generated-tokens) returns the log probabilities of the top tokens at each generated token. [`prompt_logprobs`](/docs/token-probabilities#get-log-probabilities-for-the-prompt) returns them for each token of the prompt.
- With `stream: true`, the API sends the Completion as [Server-Sent Events](/docs/sampling#stream-a-completion) that end with `data: [DONE]`.
- [`GET /v1/models`](/docs/models) lists the Base models, with the context length of each one in `max_model_len`.
- [`output_residual_stream`](/docs/activations) returns the Activations at the Layers that you list, in `activations.residual_stream` of the same response. Its shape is `[layers, tokens, hidden size]`.
- [`apply_steering_vectors`](/docs/steering) adds Steering vectors to the residual stream at the Layers that you choose.
- A [Job](/docs/jobs) runs a list of Completion requests in the background, one after another. You can check on it, cancel it, and download its results as JSON Lines, one line for each request.
- A [Loom](/docs/loom) is a tree of Nodes that you grow from a prompt, one Branch at a time. The API writes the Nodes of a Branch in the background, and `GET /v1/looms/{loom_id}` returns every Node.
- Every error has [OpenAI's error object](/docs/errors#the-error-object), with `message`, `type`, `param`, and `code`.
- The docs site has a [Quickstart](/docs/quickstart), guides, and an [API reference](/docs/reference/completions/completions.createCompletion) made from the API's OpenAPI spec, with a playground. `/llms.txt`, `/llms-full.txt`, and the Markdown of each page give the docs to an AI assistant.
# Handle the end of your Budget (/docs/cookbook/budget-cap)
Keep a script running near the end of your tokens. Detect when the API lowers `max_tokens`, tell the 429 errors apart, and find what holds your tokens. [Near the end of your tokens](/docs/concepts/budgets#near-the-end-of-your-tokens) explains the rules that this recipe acts on.
You need Python and the `openai` SDK, set up as in the [Quickstart](/docs/quickstart#set-up).
## Detect a lowered `max_tokens` [#detect-a-lowered-max_tokens]
Before Inference runs a request, the API holds the most tokens that the request can use. When your tokens cannot hold all of them, `POST /v1/completions` lowers `max_tokens` to the most that fits. It runs the request and says so in the header `X-Acs-Max-Tokens-Clamped`.
1. Send the request with `client.completions.with_raw_response.create`. It takes the arguments of `client.completions.create`, and gives you the response's headers too.
2. Read the header `X-Acs-Max-Tokens-Clamped`. The API sends it only when it lowered `max_tokens`.
3. Read the Completion with `response.parse()`.
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
response = client.completions.with_raw_response.create(
model="llama-8b", prompt="The sky is", max_tokens=200
)
completion = response.parse()
print("X-Acs-Max-Tokens-Clamped:", response.headers.get("x-acs-max-tokens-clamped"))
```
With 25 tokens left, the program prints the header below. The prompt `"The sky is"` holds 4 tokens: 3 for its text and 1 for the BOS token. That leaves room for 21 generated tokens.
```text title="Output"
X-Acs-Max-Tokens-Clamped: requested=200,applied=21,reason=budget,source=client
```
The header has these fields:
- `requested` is the `max_tokens` that you sent, or the default when you left it out.
- `applied` is the `max_tokens` that the request ran with.
- `reason` is `budget` when your tokens set the limit. It is `deadline` when a request without a stream left out `max_tokens` and got 15,600, and `deadline+budget` when both applied.
- `source` is `client` when you set `max_tokens`, and `default` when you left it out. A header with `reason=deadline` has no `source`.
A Completion that reaches the lowered `max_tokens` ends with `finish_reason` `"length"`, as any Completion that reaches its `max_tokens` does.
Only `POST /v1/completions` lowers `max_tokens`. The requests of a Job and the Nodes of a Branch keep theirs, and the Budget refuses them with `budget_exceeded` instead ([Holds of background work](/docs/concepts/budgets#holds-of-background-work)).
## Handle the 429 errors [#handle-the-429-errors]
Four errors have the status 429. Branch on `error.code`:
| `code` | Cause | What to do |
| --- | --- | --- |
| `budget_exceeded` | Not even one generated token for each choice fits in your tokens after the prompt. | Wait for your running requests to end, or [ask for more tokens](/docs/support#ask-for-more-tokens). The same request fails again until then. |
| `rate_limited` | Your API key sent more requests than its [traffic limits](/docs/limits#traffic) allow, or Inference is busy. | Try again after the wait in the `Retry-After` header. |
| `queue_full` | Your API key has its most Completions running and waiting ([Traffic](/docs/limits#traffic)). | Try again after the wait in the `Retry-After` header, and keep fewer requests in flight. |
| `spend_cap` | New GPU work is paused ([Spend limit](/docs/limits#spend-limit)). | Try again after the hour in the `Retry-After` header. |
The `openai` SDK raises `openai.RateLimitError` for each. A 429 `budget_exceeded` or `spend_cap` has the header `x-should-retry: false`, so the SDK raises it at once. For a 429 `rate_limited` or `queue_full`, the SDK first sends the request two more times, and waits as `Retry-After` asks. [Retries in the `openai` SDK](/docs/retries#retries-in-the-openai-sdk) gives the SDK's rules.
1. Create the client with the SDK's default retries, so that the SDK tries a `rate_limited` or `queue_full` again for you.
2. Catch `openai.RateLimitError`, and branch on `error.code`.
3. Print the error's message, which says what refused the request.
```python
import os
import openai
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
try:
client.completions.create(model="llama-8b", prompt="The sky is", max_tokens=200)
except openai.RateLimitError as error:
if error.code == "budget_exceeded":
print("Out of tokens:", error.body["message"])
elif error.code in ("rate_limited", "queue_full"):
print("Still limited after the SDK's tries:", error.body["message"])
elif error.code == "spend_cap":
print("GPU work is paused:", error.body["message"])
else:
raise
```
With 10 tokens left in the API key's own Budget, not even one generated token fits after the prompt's 11. The program prints:
```text title="Output"
Out of tokens: This request can use up to 211 tokens, more than this API key's token budget has left.
```
The end of the message names what refused the request:
- When the key's own Budget refused it, as here, the message ends `more than this API key's token budget has left.`
- When your Budget refused it and you have no monthly allowance, the message says `more than your token budget has left.`
- When your monthly allowance and your Budget refused it, the message says `more than your monthly allowance has left this month and more than your token budget has left.` Then it gives the day the allowance refills.
- When your own tokens refused it, the message ends with the address of the App's form to ask for more: `Ask for more tokens at https://v2.infra.acsresearch.org/usage/request.`
Its number is the Hold of the request, here the prompt's 11 tokens and the 200 of `max_tokens`. Requests that run at the same time can take the room after the API lowered `max_tokens`. The lowered request then gets this error, and its number is the lowered Hold.
`error.body` is the error object, and `error.code` is its `code`. `error.message` is the SDK's own summary, which starts with `Error code: 429`.
## Find what holds your tokens [#find-what-holds-your-tokens]
Each running request holds its tokens until it ends, so the requests that run at the same time share what you have left. The **Usage** page of the App shows the tokens left after their Holds, and the requests that hold them.
1. In the App, open **Usage**.
2. Under **Monthly allowance** and **Token budget**, read the tokens left. The App counts them after the Holds of your running Completions, so a new request fits only in them.
3. Under **Completions**, find the rows with the status **Running**. Each of these Completions keeps its Hold until it ends.
4. To see one API key, choose it in **All keys**.
[Read your usage](/docs/app#read-your-usage) describes each part of the page.
## Next steps [#next-steps]
[Budgets and Holds](/docs/concepts/budgets) explains the monthly allowance, the Budget, and the Hold. [Ask for more tokens](/docs/support#ask-for-more-tokens) says how to get a Grant or a larger monthly allowance. [Errors](/docs/errors#not-enough-tokens) shows the 429 error object.
# Steer with a contrast vector (/docs/cookbook/contrast-steering)
Build a Steering vector from two sets of prompts with the same meanings, one in a pirate style and one plain. The difference of their mean Activations at one Layer points from plain text toward pirate text. Add that difference to the residual stream at a few scales, and compare the texts with the scale-0 baseline.
You need `ACS_API_BASE` and an API key with tokens in `ACS_API_KEY`, as [Set up](/docs/quickstart#set-up) in the Quickstart shows. You also need Python 3.14 (for `compression.zstd`), NumPy, and the `openai` SDK.
## Build the contrast vector and steer with it [#build-the-contrast-vector-and-steer-with-it]
The program below does four steps:
1. Read Layer 15 of each prompt with `max_tokens: 1`. Layer 15 is near the middle of the model's 32 Layers.
2. Take the mean over the tokens of each prompt, without the BOS token.
3. Subtract the mean of the plain set from the mean of the pirate set. The difference is the contrast vector.
4. Add the contrast at Layer 15 to one prompt at the scales 0, 4, and 8, at `temperature: 0`. Scale 0 adds nothing, so it is the baseline ([Why compare with scale 0](/docs/concepts/residual-stream#why-compare-with-scale-0)).
Download `contrast.py`, or save the program below as `contrast.py`. Run it with `python contrast.py`:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
LAYER = 15
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
def mean_over_tokens(prompt):
completion = client.completions.create(
model="llama-8b",
prompt=prompt,
max_tokens=1,
extra_body={"output_residual_stream": [LAYER]},
)
residual_stream = decode(completion.activations["residual_stream"])
return residual_stream[0, 1:].mean(axis=0)
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
pirate = [
"Arr, the wind be fair and the sea be calm.",
"Ahoy, me hearties, hoist the sails!",
"Shiver me timbers, that be a fine ship.",
"Avast, ye scallywags, the treasure be ours!",
]
plain = [
"The wind is fair and the sea is calm.",
"Hello, friends, raise the sails!",
"Goodness, that is a fine ship.",
"Stop, you rascals, the treasure is ours!",
]
contrast = np.mean([mean_over_tokens(p) for p in pirate], axis=0) - np.mean(
[mean_over_tokens(p) for p in plain], axis=0
)
print(contrast.shape, round(float(np.linalg.norm(contrast)), 2))
for scale in [0, 4, 8]:
completion = client.completions.create(
model="llama-8b",
prompt="My advice for your first day at a new job is",
max_tokens=16,
temperature=0,
extra_body={
"apply_steering_vectors": [{"activations": [contrast.tolist()], "layer_indices": [LAYER], "scale": scale}]
},
)
print(scale, repr(completion.choices[0].text))
```
The program prints the shape and the norm of the contrast, and then the scale and the text of each Completion:
```text title="Output"
(4096,) 3.59
0 ' to be yourself. Don’t try to be someone you’re not. You’ll'
4 ' to swag yer’ scollies ye’ ye’ me’ ye’'
8 ' ye Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers Buccaneers'
```
- Each capture asks for `max_tokens: 1`, so its Activations have one row for each prompt token. [Read Activations for text you already have](/docs/activations#read-activations-for-text-you-already-have) explains why.
- `mean_over_tokens` skips row 0, the BOS token. BOS is the same at the start of every prompt, and its Activations are much larger than those of the other tokens.
- `mean_over_tokens` takes the mean of the other rows, so a long prompt and a short prompt count the same.
- The contrast has one number for each dimension of the model's hidden size, 4096.
- `decode` is the helper from [Decode the tensor](/docs/activations#decode-the-tensor).
## Read the result [#read-the-result]
Compare each text with the scale-0 baseline. Here, the contrast has a norm of 3.59. At scale 4, the text moves toward the pirate style. At scale 8, the model repeats one word, so the scale is too large. Lower the scale when the text starts to repeat phrases or break into fragments.
More prompts in each set give a contrast that depends less on any one prompt. [Choose a scale](/docs/concepts/residual-stream#choose-a-scale) helps you pick the next scale to try. To set a scale relative to the residual stream at each token, use `norm_match`, as [Match the norm](/docs/concepts/residual-stream#match-the-norm) explains.
# Evaluate with Inspect (/docs/cookbook/evaluate-with-inspect)
Run an eval on a Base model with [Inspect](https://inspect.aisi.org.uk), the UK AI Security Institute's framework for evals. Its provider `openai-api-completions` sends each sample's input as a raw prompt to `POST /v1/completions`, with no chat template. You need the two shell variables of the [Quickstart](/docs/quickstart#set-up). The commands on this page work with `inspect-ai` 0.3.286 and the `openai` SDK 3.41.0.
## Point Inspect at the API [#point-inspect-at-the-api]
1. Install Inspect and the `openai` SDK:
```bash
pip install inspect-ai openai
```
2. Give Inspect the API's address. For a service named `acs`, Inspect reads the key from `ACS_API_KEY`, which the Quickstart sets, and the address from `ACS_BASE_URL`:
```bash
export ACS_BASE_URL="$ACS_API_BASE"
```
3. Name the model as `openai-api-completions/acs/` and the model's full id, such as `openai-api-completions/acs/llama-8b`. Inspect takes `acs/` off and sends the rest as `model`. [List the models](/docs/models#list-the-models) gives each id.
## Run a task [#run-a-task]
1. Download `capitals.py`, or save this task as `capitals.py`. Each sample is a prompt with two examples, which a Base model continues. `includes()` scores a sample 1 when the model's text contains its target:
```python
from inspect_ai import Task, task
from inspect_ai.dataset import Sample
from inspect_ai.scorer import includes
from inspect_ai.solver import generate
EXAMPLES = "The capital of Italy is Rome.\nThe capital of Japan is Tokyo.\n"
@task
def capitals():
return Task(
dataset=[
Sample(input=EXAMPLES + "The capital of France is", target="Paris"),
Sample(input=EXAMPLES + "The capital of Argentina is", target="Buenos Aires"),
Sample(input=EXAMPLES + "The capital of Canada is", target="Ottawa"),
],
solver=generate(),
scorer=includes(),
)
```
2. Run the eval on Llama 3.1 8B at temperature 0, with up to 4 tokens for each answer:
```bash
inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0 --max-tokens 4
```
Inspect prints the task's accuracy, and writes the log of the eval to `./logs`. `inspect view` shows the logs, with each sample's input and the model's text.
Always set `--max-tokens`. Without it, the provider asks for 1 token, and an answer such as `Buenos Aires` takes two.
## Write prompts for a Base model [#write-prompts-for-a-base-model]
A Base model continues text, and it does not answer a question as a chat model does ([What a Base model does](/docs/concepts/base-models#what-a-base-model-does)). Write each input as the start of a text whose next words are the answer, as the examples in `capitals.py` do.
The provider sends one text. A sample's `input` must be a string or one user message. An input of several messages, or `--system-message`, stops the eval with a `TypeError`.
## Set the request fields [#set-the-request-fields]
The provider sends `model`, `prompt`, `max_tokens`, and `temperature` with each request. It sends `max_tokens: 1` without `--max-tokens`, and `temperature: 0` without `--temperature`. These options of `inspect eval` set the other fields:
| Option | Field |
| --- | --- |
| `--top-p` | `top_p` |
| `--seed` | `seed` |
| `--stop-seqs` | `stop` |
| `--num-choices` | `n` |
| `--logprobs`, and `--top-logprobs` for more than 1 | `logprobs` |
| `--frequency-penalty` and `--presence-penalty` | `frequency_penalty` and `presence_penalty` |
| `--prompt-logprobs` | `prompt_logprobs` |
| `--extra-body` | Each field of its JSON, such as `'{"top_k": 5}'` |
- The scorers read the first choice only, so `--num-choices` adds texts to the log and not to the score.
- `--top-k`, `--best-of`, and `--logit-bias` send nothing, and neither does `--top-logprobs` without `--logprobs`. Send `top_k` in `--extra-body`.
- The provider sends no request with a stream. So `max_tokens` must stay within the limit of a request without a stream ([Tokens](/docs/limits#tokens)).
- The API refuses a field that it does not know, such as vLLM's `return_token_ids`, with 400 `invalid_request` ([Strict validation](/docs/completions#strict-validation)).
## Run a large eval [#run-a-large-eval]
1. Set `--max-connections 16`, the number of Completions that run at once for each API key ([Traffic](/docs/limits#traffic)). Without it, Inspect opens 20 connections at first and up to 100. Past the key's line, the API answers 429 `rate_limited`.
2. Set `--max-retries`, such as `--max-retries 3`. Inspect tries each 429 again with no end. That includes `budget_exceeded` at the end of your tokens and `spend_cap`, which the API sends with `x-should-retry: false`.
3. Run the eval on a Warm model. The first request to a cold model waits for its start. A start of Llama 3.1 405B or Trinity Large takes longer than the API waits for Inference ([Warm and cold models](/docs/concepts/long-requests#warm-and-cold-models)). [Check whether a model is Warm](/docs/models#check-whether-a-model-is-warm) shows the status route.
```bash
inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0 --max-tokens 4 --max-connections 16 --max-retries 3
```
## If the eval fails [#if-the-eval-fails]
| Inspect says | Cause | What to do |
| --- | --- | --- |
| `No ACS_BASE_URL defined in the environment.` | Inspect reads `ACS_BASE_URL`, not `ACS_API_BASE`. | Export `ACS_BASE_URL`, as [Point Inspect at the API](#point-inspect-at-the-api) shows. |
| `Server at does not support /v1/completions.` | Inspect says this for any 404, such as a wrong `ACS_BASE_URL`. | Check that `ACS_BASE_URL` ends in `/v1`. |
| `Error code: 400 - {'error': {'message': 'The model ...` with `model_not_found` | The API does not serve the model. | Check the model's id in `GET /v1/models`. |
| `ValidationError: 1 validation error for ModelOutput` | After 150 seconds, a request without a stream gets its 200, and a later error comes as the body. Inspect cannot read that body ([Requests without a stream](/docs/concepts/long-requests#requests-without-a-stream)). A cold model's start can cause it. | Check that the model is Warm, then run the eval again. |
| `This API serves Base models, which have no chat format` | The provider `openai-api` sends chat requests to `/v1/chat/completions`. | Use `openai-api-completions`. |
## Next steps [#next-steps]
[Score a text](/docs/cookbook/score-a-text) gets the log-likelihood of a text, which an eval with several choices can compare. [Run a Job](/docs/jobs) sends many requests in the background. [Limits](/docs/limits) lists each limit that an eval can meet.
# Download a harvest from a shell script (/docs/cookbook/harvest-script)
Read the Activations of many prompts from one shell script. It submits a harvest, waits until the harvest ends, and downloads its files into a new folder. You need `curl`, `jq`, and the two shell variables of the [Quickstart](/docs/quickstart#set-up). [Harvest Activations](/docs/harvest) explains each route that the script uses.
## Run the script [#run-the-script]
1. Save this script as `harvest.sh`. Put your own prompts and Layers in the body.
```sh
#!/bin/sh
set -eu
harvest=$(curl -sS "$ACS_API_BASE/harvest" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}')
JOB_ID=$(echo "$harvest" | jq -r .job_id)
if [ "$JOB_ID" = null ]; then
echo "$harvest" >&2
exit 1
fi
echo "Submitted harvest job $JOB_ID" >&2
status=pending
while [ "$status" = pending ] || [ "$status" = running ]; do
sleep 3
read=$(curl -sS "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
-H "Authorization: Bearer $ACS_API_KEY")
status=$(printf '%s' "$read" | jq -r .status)
[ "$status" != null ] || { printf '%s\n' "$read" >&2; exit 1; }
done
echo "Harvest job $JOB_ID is $status" >&2
[ "$status" = done ] || exit 1
mkdir "harvest-$JOB_ID"
cd "harvest-$JOB_ID"
curl -sS "$ACS_API_BASE/harvest/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY" \
| jq -r '.result | ([["manifest.json", .manifest_url], ["stats.safetensors", .stats_url]] + [.shard_urls | to_entries[] | ["shard_\(("0000" + (.key | tostring))[-5:]).safetensors", .value]])[] | "url = \"\(.[1])\"\noutput = \"\(.[0])\""' \
| curl -sSf --parallel --config -
echo "Downloaded the files into harvest-$JOB_ID" >&2
```
2. Run the script:
```sh
sh harvest.sh
```
The script writes its messages, such as `Harvest job ... is done`, to standard error. It exits with 1 when the API refuses the harvest or a read of it, and prints the answer. It also exits with 1 when the harvest ends `failed` or `cancelled`, because such a harvest has no files.
3. Read the files in the new folder `harvest-`: `manifest.json`, `stats.safetensors`, and one `shard_.safetensors` for each shard. [Read the shards](/docs/harvest#read-the-shards) says what each file holds.
The script asks the API to wait up to 60 seconds for each change of status, and waits 3 seconds between two reads. So it sends few requests while a long harvest runs. The last `curl` downloads the files in parallel from the links in the harvest's `result`. Those links need no API key.
## If the script stops [#if-the-script-stops]
The harvest runs in the API and on Inference, not in the script, so it keeps running when the script stops. `set -eu` stops the script when a command fails, for example when `curl` cannot reach the API. Put the harvest's `job_id`, from the line `Submitted harvest job ...`, in `JOB_ID`. When the harvest is `done`, read it again with `curl "$ACS_API_BASE/harvest/$JOB_ID" -H "Authorization: Bearer $ACS_API_KEY"` for its links. Each answer has new links, so a link that expired is no problem.
# Cookbook (/docs/cookbook)
A recipe does one task from start to end, with a program or a script to copy. A guide covers one part of the API. Every recipe reads the two shell variables that the [Quickstart](/docs/quickstart#set-up) sets up, and says what else it needs. A recipe with a whole program or notebook links to it as a file to download.
- [Run a Job from a shell script](/docs/cookbook/job-script) submits a Job with `curl`, waits until it ends, and saves its result to a file.
- [Download a harvest from a shell script](/docs/cookbook/harvest-script) submits a harvest with `curl`, waits until it ends, and downloads its files into a folder.
- [Handle the end of your Budget](/docs/cookbook/budget-cap) detects a lowered `max_tokens` and handles the 429 errors in Python.
- [Score a text](/docs/cookbook/score-a-text) adds up the log probabilities of a text's tokens to get its log-likelihood.
- [Evaluate with Inspect](/docs/cookbook/evaluate-with-inspect) runs an Inspect eval on a Base model through `POST /v1/completions`.
- [Steer with a contrast vector](/docs/cookbook/contrast-steering) builds a Steering vector from two sets of prompts, and steers a Base model with it.
- [Run the workshop notebook](/docs/cookbook/workshop) goes from a first Completion to a steered Base model in a Jupyter notebook, with exercises.
Where an output shows text that `meta-llama/Llama-3.1-8B` wrote, it is one sample, and your output differs. [Why repeats differ](/docs/concepts/base-models#why-repeats-differ) explains how to get the same tokens again, and why vLLM's batching can still change one.
# Run a Job from a shell script (/docs/cookbook/job-script)
Run one Completion on the server from a shell script. It submits a Job, waits until the Job ends, and saves the result to a file. You need `curl` 7.76 or later, `jq`, and the two shell variables of the [Quickstart](/docs/quickstart#set-up). [Run a Job](/docs/jobs) explains each route that the script uses.
## Run the script [#run-the-script]
1. Download `job.sh`, or save this script as `job.sh`. Put your own Completion request in the body of the first `curl`.
```bash
#!/bin/sh
set -eu
KEY=${1:-sky-$(date +%s)}
job=$(curl -sS "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}')
JOB_ID=$(echo "$job" | jq -r .id)
echo "Submitted job $JOB_ID with Idempotency-Key $KEY" >&2
status=queued
while [ "$status" = queued ] || [ "$status" = running ]; do
sleep 1
status=$(curl -sS "$ACS_API_BASE/completion-jobs/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY" | jq -r .status)
done
echo "Job $JOB_ID is $status" >&2
curl -sS --fail-with-body "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY"
```
2. Run the script with an `Idempotency-Key` for this piece of work, and send its output to a file:
```sh
sh job.sh sky-001 > result.json
```
The script writes its messages, such as `Job ... is succeeded`, to standard error, so the file holds only the result. It exits with 22 when the result is an error, as `curl --fail-with-body` does, and the file then holds the error object.
3. Read the result in `result.json`. It is the Completion that `POST /v1/completions` would have answered:
```text title="result.json"
{ "id": "cmpl-ab2a9dbbc18c302b", "object": "text_completion", "created": 1791781222, "model": "meta-llama/Llama-3.1-8B", "choices": [{ "index": 0, "text": " falling!", "logprobs": null, "finish_reason": "length", "prompt_logprobs": null }], "usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 } }
```
## If the script stops [#if-the-script-stops]
The Job runs in the API, not in the script, so it keeps running when the script stops. `set -eu` stops the script when a command fails, for example when `curl` cannot reach the API. Run the script again with the same `Idempotency-Key`. The API answers the Job that the key made. The script then saves its result, and the Completion does not run again.
# Score a text (/docs/cookbook/score-a-text)
Get the log-likelihood of a text under a Base model: the sum of the log probabilities of its tokens. You need Python and the `openai` SDK, set up as in the [Quickstart](/docs/quickstart#set-up).
## Score the text [#score-the-text]
1. Find the token ids of your text. Send the text with `prompt_logprobs: 0` and `max_tokens: 1`. The `openai` SDK does not know `prompt_logprobs`, so send it in `extra_body`. With 0, each position has one entry, the prompt's own token, keyed by its id. Ignore the one generated token.
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=1,
extra_body={"prompt_logprobs": 0},
)
for position, tokens in enumerate(completion.choices[0].prompt_logprobs):
print(position, tokens)
```
The program prints the entry of each position:
```text title="Output"
0 None
1 {'791': {'logprob': -3.6329822540283203, 'rank': 4, 'decoded_token': 'The'}}
2 {'13180': {'logprob': -8.832155227661133, 'rank': 1144, 'decoded_token': ' sky'}}
3 {'374': {'logprob': -0.9844952821731567, 'rank': 1, 'decoded_token': ' is'}}
```
- Position 0 is `None`, because no token comes before it. `meta-llama/Llama-3.1-8B` puts the BOS token `<|begin_of_text|>` in front of a text, and its id is 128000. The response does not show it.
- From position 1 on, the key is the id of your token, as a string. Each token of your text has an id and a log probability, because BOS takes position 0.
- `rank` is 1 for the likeliest token at a position. Your token is not always the likeliest one. Here `" sky"` has rank 1144, and the model found 1,143 other tokens likelier after `"The"`.
Match tokens by id, not by text, because two ids can decode to the same text.
2. Send the ids as the prompt, with the BOS id 128000 first. Inference then scores these tokens and no others. It adds no special token, such as BOS, to a prompt of token ids, so put BOS first, as the model does for a text. Position 0 has no entry, and this program reads each of your ids, in `scored`, from position 1 on:
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt=[128000, 791, 13180, 374],
max_tokens=1,
extra_body={"prompt_logprobs": 0},
)
scored = [791, 13180, 374]
positions = completion.choices[0].prompt_logprobs[1:]
for token_id, tokens in zip(scored, positions):
print(token_id, tokens[str(token_id)]["logprob"])
```
```text title="Output"
791 -3.6329822540283203
13180 -8.832155227661133
374 -0.9844952821731567
```
`tokens[str(token_id)]` finds each token by its id. The program reads the log probability of your token, not of another token with the same text.
3. Add up the log probabilities from position 1 on:
```python
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt=[128000, 791, 13180, 374],
max_tokens=1,
extra_body={"prompt_logprobs": 0},
)
scored = [791, 13180, 374]
positions = completion.choices[0].prompt_logprobs[1:]
print(sum(tokens[str(token_id)]["logprob"] for token_id, tokens in zip(scored, positions)))
```
```text title="Log-likelihood"
-13.44963276386261
```
The sum is the log-likelihood of your text after the BOS token. BOS has no log probability, and it is not part of your text.
## Next steps [#next-steps]
[Read token probabilities](/docs/token-probabilities#get-log-probabilities-for-the-prompt) returns the k likeliest tokens at each position of a prompt. [Get the whole vocabulary](/docs/token-probabilities#get-the-whole-vocabulary) returns every token's log probability at each position. [Read Activations for text you already have](/docs/activations#read-activations-for-text-you-already-have) reads the residual stream of the same text.
# Run the workshop notebook (/docs/cookbook/workshop)
Run the workshop notebook to go from a first Completion to a steered Base model in one sitting. It samples Llama 3.1 8B, reads its token probabilities and its Activations, and builds a Steering vector that makes it write like a pirate.
Each section after Set up ends with an exercise to change and run again. You need the two shell variables of the [Quickstart](/docs/quickstart#set-up), and Python 3.14 for `compression.zstd`.
## Open the notebook [#open-the-notebook]
1. Download `workshop.ipynb`.
2. In the folder of the file, in the shell that has `ACS_API_BASE` and `ACS_API_KEY`, start JupyterLab on Python 3.14 with NumPy and the `openai` SDK. With [`uv`](https://docs.astral.sh/uv/), one command does it:
```bash
uvx --python 3.14 --with openai --with numpy --from jupyterlab jupyter-lab workshop.ipynb
```
Without `uv`, install `jupyterlab`, `numpy`, and `openai` with pip on Python 3.14, then run `jupyter lab workshop.ipynb`.
3. Run the cells in order, from the top. When `ACS_API_KEY` is not set, the first code cell, under Set up, asks for your key, and the notebook does not show it.
Each run of the notebook sends about 30 Completions to `meta-llama/Llama-3.1-8B`, and your monthly allowance or your Budget pays for their tokens. The file keeps no outputs, so each text that you see is your own sample.
## Work through the sections [#work-through-the-sections]
Each section of the notebook names the page of these docs that explains it.
| Section | What it does |
| --- | --- |
| Set up | Reads the model's facts, and defines the `decode` helper from [Decode the tensor](/docs/activations#decode-the-tensor). |
| Send your first prompt | Sends one Completion, with the log probability of each token ([Send a prompt](/docs/quickstart#send-a-prompt)). |
| Sample at different temperatures | Repeats a prompt at three temperatures with a fixed `seed` ([Set the sampling fields](/docs/sampling#set-the-sampling-fields)). |
| Read token probabilities | Reads `logprobs` and `prompt_logprobs`, and scores two texts ([Read token probabilities](/docs/token-probabilities)). |
| Read Activations | Reads the residual stream at two Layers, then at every Layer ([Read Activations](/docs/activations)). |
| Steer the model to talk like a pirate | Builds a contrast vector at Layer 15, and steers with it at three scales. Its exercise checks that the steer arrived ([Steer with a contrast vector](/docs/cookbook/contrast-steering)). |
## Next steps [#next-steps]
[Steer with a contrast vector](/docs/cookbook/contrast-steering) is the last section as one program to run. [Choose a scale](/docs/concepts/residual-stream#choose-a-scale) says how to pick a scale for another Layer. [Ask for Activations in a Job](/docs/jobs#ask-for-activations-in-a-job) reads the Activations of many prompts in the background.
# API reference (/docs/reference)
The reference comes from the API's OpenAPI spec, so it changes when the API changes. Each route's page lists the request fields and each response, and its playground sends the request to the API.
| Route | What it does |
| --- | --- |
| [`GET /v1/models`](/docs/reference/models/models.listModels) | List the Base models. |
| [`GET /v1/models/{model_id}`](/docs/reference/models/models.retrieveModel) | Read one Base model. |
| [`GET /v1/models/{model_id}/status`](/docs/reference/models/models.getModelStatus) | Say whether a model is Warm. |
| [`POST /v1/completions`](/docs/reference/completions/completions.createCompletion) | Create a Completion. |
| [`POST /v1/completion-jobs`](/docs/reference/completionJobs/completionJobs.createCompletionJob) | Queue a Job. |
| [`GET /v1/completion-jobs/{job_id}`](/docs/reference/completionJobs/completionJobs.getCompletionJob) | Read a Job. |
| [`GET /v1/completion-jobs/{job_id}/result`](/docs/reference/completionJobs/completionJobs.getCompletionJobResult) | Fetch the result of a Job. |
| [`POST /v1/completion-jobs/{job_id}/cancel`](/docs/reference/completionJobs/completionJobs.cancelCompletionJob) | Cancel a Job. |
| [`POST /v1/harvest`](/docs/reference/harvest/harvest.createHarvest) | Queue a harvest Job of Activations. |
| [`GET /v1/harvest/{job_id}`](/docs/reference/harvest/harvest.getHarvest) | Read a harvest Job and its files, and wait for its status to change. |
| [`DELETE /v1/harvest/{job_id}`](/docs/reference/harvest/harvest.cancelHarvest) | Cancel a harvest Job. |
| [`GET /v1/looms`](/docs/reference/looms/looms.listLooms) | List your Looms, pinned first. |
| [`POST /v1/looms`](/docs/reference/looms/looms.createLoom) | Create a Loom. |
| [`POST /v1/looms/import`](/docs/reference/looms/looms.importLoom) | Make a Loom from a file. |
| [`GET /v1/looms/{loom_id}`](/docs/reference/looms/looms.getLoom) | Read a Loom. |
| [`POST /v1/looms/{loom_id}`](/docs/reference/looms/looms.updateLoom) | Rename or pin a Loom. |
| [`GET /v1/looms/{loom_id}/export`](/docs/reference/looms/looms.exportLoom) | Download a Loom as a file. |
| [`GET /v1/looms/{loom_id}/stream`](/docs/reference/looms/looms.streamLoom) | Follow a Loom as Server-Sent Events. |
| [`POST /v1/looms/{loom_id}/cancel`](/docs/reference/looms/looms.cancelLoom) | Stop the Loom's growing Nodes. |
| [`POST /v1/looms/{loom_id}/root`](/docs/reference/looms/looms.setRoot) | Replace the tree with a new root. |
| [`DELETE /v1/looms/{loom_id}`](/docs/reference/looms/looms.deleteLoom) | Delete a Loom. |
| [`POST /v1/looms/{loom_id}/nodes/{node_id}`](/docs/reference/looms/looms.editNode) | Edit a Node. |
| [`POST /v1/looms/{loom_id}/nodes/{node_id}/split`](/docs/reference/looms/looms.splitNode) | Split a Node at an offset. |
| [`POST /v1/looms/{loom_id}/nodes/{node_id}/sibling`](/docs/reference/looms/looms.addSibling) | Write a Node beside a Node. |
| [`POST /v1/looms/{loom_id}/nodes/{node_id}/branch`](/docs/reference/looms/looms.createBranch) | Branch from a Node of a Loom. |
| [`DELETE /v1/looms/{loom_id}/nodes/{node_id}`](/docs/reference/looms/looms.deleteNode) | Delete a Node and the Nodes under it. |
| [`GET /health`](/docs/reference/health/health.health) | Check that the API answers. |
| [`POST /v1/chat/completions`](/docs/reference/chat/chat.createChatCompletion) | Refuse every chat request with 400 and a message that names `POST /v1/completions`. |
# FAQ (/docs/support/faq)
Each answer is short and links to the page that explains it. If your question is not here, write to [infra@acsresearch.org](mailto:infra@acsresearch.org), as [Get help](/docs/support#get-help) says.
## Why does the model not answer my question? [#why-does-the-model-not-answer-my-question]
Every model is a Base model, which continues your text and does not answer it, and the API applies no chat template. [What a Base model does](/docs/concepts/base-models#what-a-base-model-does) explains how to write a prompt for it.
## Why does my output differ from the output in the docs? [#why-does-my-output-differ-from-the-output-in-the-docs]
A Base model samples each token, so each run of a request writes other text, and each output in the docs is one sample. [Why repeats differ](/docs/concepts/base-models#why-repeats-differ) explains it.
## How do I get the same text twice? [#how-do-i-get-the-same-text-twice]
Set `temperature` to 0, or send the same `seed` to the same model. vLLM's batching can still change a token now and then, as [Get the same tokens again](/docs/sampling#get-the-same-tokens-again) explains.
## Can I use the `openai` SDK? [#can-i-use-the-openai-sdk]
Yes, with `client.completions.create(...)`, since the API follows OpenAI's completions API. The API serves no chat, as [No chat template](/docs/concepts/base-models#no-chat-template) explains.
## How do I send vLLM's fields from the `openai` SDK? [#how-do-i-send-vllms-fields-from-the-openai-sdk]
Put them in `extra_body`, because the SDK raises `TypeError` for an argument that it does not know. [Send vLLM's fields](/docs/sampling#send-vllms-fields-from-the-openai-sdk) shows an example.
## How do I get the whole distribution over the vocabulary? [#how-do-i-get-the-whole-distribution-over-the-vocabulary]
Set vLLM's `prompt_logprobs` to `-1`, within the size caps for one prompt. [Get the whole vocabulary](/docs/token-probabilities#get-the-whole-vocabulary) shows it, and how to get it for generated tokens too.
## Why does a misspelled field get an error? [#why-does-a-misspelled-field-get-an-error]
The API refuses a field that it does not know with 400 `invalid_request`, so that a typo cannot change a result without a warning. [Why the API is strict](/docs/concepts/base-models#why-the-api-is-strict) explains it.
## Why did my Completion stop before `max_tokens`? [#why-did-my-completion-stop-before-max_tokens]
With `finish_reason` `"stop"`, the model wrote its end-of-text token or a `stop` text. With `"length"`, `POST /v1/completions` may have lowered `max_tokens` to fit your tokens, as [Near the end of your tokens](/docs/concepts/budgets#near-the-end-of-your-tokens) explains.
## Why did I get a 429 error? [#why-did-i-get-a-429-error]
The code tells you why:
- `budget_exceeded`: your tokens cannot hold even one token for each choice.
- `rate_limited`: too many requests.
- `spend_cap`: new GPU work is paused ([Spend limit](/docs/limits#spend-limit)).
[Handle the 429 errors](/docs/cookbook/budget-cap#handle-the-429-errors) tells them apart in code.
## How do I get more tokens? [#how-do-i-get-more-tokens]
Ask for them on the App's **Request more tokens** page, under **Usage**. An admin approves the request with a Grant, or rejects it with a note. [Ask for more tokens](/docs/app#ask-for-more-tokens) says how.
## Why was my first request slow? [#why-was-my-first-request-slow]
The model was not Warm, so the request waited while Inference started a container for it. [Warm and cold models](/docs/concepts/long-requests#warm-and-cold-models) explains it.
## What happens to my Job when my script stops? [#what-happens-to-my-job-when-my-script-stops]
The Job keeps running, because it runs in the API and not in your script. [What runs in the background](/docs/concepts/background-work#what-runs-in-the-background) explains it.
## How do I revoke an API key? [#how-do-i-revoke-an-api-key]
On the App's **API Keys** page, open the menu at the end of the key's row and select **Revoke**. Scripts that send the key then get 401 at once. [Revoke a key](/docs/api-keys#revoke-a-key) has the steps.
## Can I share my API key? [#can-i-share-my-api-key]
No, because your access is personal. [Your access is personal](/docs/support/research-preview#your-access-is-personal) says where to point others.
# Support (/docs/support)
ACS Infra is in research preview, and we answer by email. Write to us when you have a question, when something breaks, or when you need more tokens.
## Get help [#get-help]
Write to [infra@acsresearch.org](mailto:infra@acsresearch.org) from the email address that you sign in with, so that we can find your User. Never send an API key.
The [FAQ](/docs/support/faq) answers common questions in a sentence or two, each with a link to the page that explains it.
## Report a bug [#report-a-bug]
Before you write, check [Status codes](/docs/errors#status-codes). The API refuses many requests on purpose. For example, it answers 400 `invalid_request` to a misspelled field, and 400 `model_not_found` to a model that it does not serve. If your error is there, fix the request as that page says.
If the API still does something that you did not expect, write to [infra@acsresearch.org](mailto:infra@acsresearch.org). Send these details, so that we can reproduce the bug:
- The request, as a curl command or as code. Write `$ACS_API_KEY` in place of the key, as the examples in these docs do.
- What you expected the API to do.
- The status code, and the whole error object or the whole output. For a stream, send every event up to `data: [DONE]`.
- The `X-Request-Id` header of the answer, which curl prints with `-i`. An error object has the same id in `request_id`.
- The time of the request, in UTC.
- The `model` of the request, and whether it was a stream.
- The `id` of each Job, Loom, or Node that the request used.
In the App, **Feedback** under **Support** also reaches us. It sends your description, up to five screenshots, and the App's recent errors and request ids, which you see before you send them. [Send feedback](/docs/app#send-feedback) says how.
## Ask for more tokens [#ask-for-more-tokens]
An approval gives you a monthly allowance of tokens. When your research needs more, ask for them on the App's [Request more tokens](https://v2.infra.acsresearch.org/usage/request) page. Say how many tokens you need, for which models and which project, and how you plan to use them. An admin approves the request with a Grant to your Budget, or rejects it with a note. The page shows the answer ([Ask for more tokens](/docs/app#ask-for-more-tokens)). To give one API key its own Budget, which limits what one script spends, write to [infra@acsresearch.org](mailto:infra@acsresearch.org). [Budgets and Holds](/docs/concepts/budgets) explains the difference.
## Discord [#discord]
ACS has a Discord server, where Users share their results and experiments with Base models, talk about research, and hear about updates. In the App, **Discord** under **Support** opens its invite in a new tab. When the App shows no **Discord**, ask us for an invite at [infra@acsresearch.org](mailto:infra@acsresearch.org).
## Next steps [#next-steps]
[Research preview](/docs/support/research-preview) lists the platform rules that you agreed to when you applied. The [FAQ](/docs/support/faq) has short answers, each with a link to its page.
# Research preview (/docs/support/research-preview)
ACS Infra is in research preview. The [Apply form](https://www.v2.infra.acsresearch.org/apply) lists these platform rules, and you agree to them when you apply.
## Your access is personal [#your-access-is-personal]
Do not share your account or your API keys. If someone else needs access, point them to the [Apply form](https://www.v2.infra.acsresearch.org/apply).
## Do not use the platform for harm [#do-not-use-the-platform-for-harm]
Do not use the platform for illegal, harmful, or plainly unethical purposes.
## Expect rough edges [#expect-rough-edges]
The platform is in research preview, so expect rough edges. When something breaks, tell us, as [Report a bug](/docs/support#report-a-bug) says. The [Changelog](/docs/changelog) lists what changed in the API and the App.
## Link to the platform when it helps [#link-to-the-platform-when-it-helps]
When the platform is a significant help to your findings, consider a link to it in the paper or post that presents them. The link goes to the [ACS Infra website](https://infra.acsresearch.org), `https://infra.acsresearch.org`.
## Next steps [#next-steps]
[Support](/docs/support) says how to get help and report a bug. The [FAQ](/docs/support/faq) has short answers to common questions.
# Your first steered Completion (/docs/tutorials/first-steered-completion)
In this tutorial, you steer a Base model for the first time. First, you get access and make an API key. Then you run a Completion, read the residual stream at Layer 1, and add a Steering vector at that Layer. At the end, you will see the residual stream move by your vector.
You need curl, Python 3.14 (for `compression.zstd`), NumPy, and the `openai` SDK. Each output is one sample that `meta-llama/Llama-3.1-8B` wrote.
## Get access [#get-access]
An admin approves each Application to use ACS Infra.
1. Open the [Apply form](https://www.v2.infra.acsresearch.org/apply) on the ACS Infra website.
2. Fill in the form. **Email**, **Name**, **Organization**, and **Planned usage** are required.
3. Read the platform rules, and select the box **I agree to these rules.**
4. Select **Apply for access**. The page shows "Application received".
5. Wait for the email. When an admin approves your Application, you become a User with a monthly allowance of 5,000,000 tokens. The email has the subject "Your ACS Infra application is approved".
6. In the email, select **Choose your password**. The link works once, for 7 days.
7. The App opens the page "Set your password". Enter a password of at least 8 characters, and select **Set the password**. The App shows "Your password is set."
8. Select **Sign in**, and sign in with your email and your password.
9. Open **Usage** in the App. **Monthly allowance** shows the tokens that you have left this month.
[Get access](/docs/get-access) has the details, such as signing in with GitHub, Google, or Hugging Face when the App shows their buttons.
## Make an API key [#make-an-api-key]
1. In the App, open **API Keys**.
2. Select **New key**, enter a name, such as `laptop`, and select **Create key**.
3. The App shows the dialog "Copy your key". Select **Copy**. The App shows the key only once.
4. Put the key and the API's address in shell variables:
```bash
export ACS_API_KEY=sk-acs-...
export ACS_API_BASE=https://api.v2.infra.acsresearch.org/v1
```
Now check the key. List the Base models with it:
```bash
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"
```
```json title="Models"
{
"object": "list",
"data": [
{
"id": "llama-8b",
"object": "model",
"owned_by": "acs",
"served_model_name": "meta-llama/Llama-3.1-8B",
"gpu_shape": "1xL40S",
"status": "live",
"default": true,
"capabilities": {
"max_model_len": 131072,
"max_logprobs": 100,
"max_output_work_tokens": 32000,
"max_inflight_per_key": 16,
"max_queued_per_key": 64,
"rate_limit_per_minute": 600,
"logprobs": true,
"prompt_logprobs": true,
"prompt_logprobs_full_vocab": true,
"full_vocab_max_prompt_tokens": 256,
"activations": true,
"max_steering_vectors": 8,
"max_activation_prompt_tokens": 256,
"n_layers": 32,
"hidden_size": 4096,
"chat_template": null
}
}
]
}
```
Use the `id` of a Base model as `model` in the next steps. This tutorial uses `llama-8b`, the weights `meta-llama/Llama-3.1-8B`. Its `capabilities` give 32 Layers in `n_layers` and a hidden size of 4096 in `hidden_size`.
## Run a Completion [#run-a-completion]
Send the prompt `"The sky is"`, and ask for 2 tokens. Set `temperature` to 0, so that the model writes its likeliest token each time.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0}'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2, temperature=0
)
print(repr(completion.choices[0].text))
```
The curl command prints the Completion:
```json title="Completion"
{
"id": "cmpl-824562de43d1b203",
"object": "text_completion",
"created": 1791780344,
"model": "meta-llama/Llama-3.1-8B",
"choices": [
{
"index": 0,
"text": " the limit",
"logprobs": null,
"finish_reason": "length",
"prompt_logprobs": null
}
],
"usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}
```
The Python program prints the text that the Base model wrote:
```text title="Python output"
' the limit'
```
The API took the tokens of `usage` from your monthly allowance.
## Capture Activations [#capture-activations]
Ask for the residual stream at Layer 1 with `output_residual_stream`. The `openai` SDK does not know the field, so send it in `extra_body`. The API sends the numbers compressed, and Llama's numbers are bfloat16. This program decodes them with base64, zstd, and NumPy, and prints their shape and the first 4 numbers of the first token:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
temperature=0,
extra_body={"output_residual_stream": [1]},
)
residual_stream = decode(completion.activations["residual_stream"])
print(residual_stream.shape)
print(residual_stream[0, 0, :4].tolist())
```
```text title="Output"
(1, 5, 4096)
[0.0145263671875, 0.00982666015625, -0.0791015625, -0.05419921875]
```
The shape is 1 Layer, then 5 rows, then 4096 numbers in each row, the hidden size. The 5 rows are the 4 prompt tokens, with `<|begin_of_text|>` first, and the first of the 2 generated tokens. The last generated token never goes back through the model. [Decode the tensor](/docs/activations#decode-the-tensor) explains each field. Keep the first 4 numbers for the next step.
## Steer Layer 1 [#steer-layer-1]
Add a Steering vector at Layer 1 with `apply_steering_vectors`, and read Layer 1 in the same request. The vector has one number for each of the 4096 dimensions. This one is `[1, 2, 3]` and then zeros, with a `scale` of 2:
```python
import base64
import os
from compression import zstd
import numpy as np
from openai import OpenAI
def decode(tensor):
raw = zstd.decompress(base64.b64decode(tensor["data"]))
array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
if tensor["original_dtype"] == "torch.bfloat16":
array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
return array
vector = np.zeros(4096)
vector[:3] = [1, 2, 3]
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
temperature=0,
extra_body={
"output_residual_stream": [1],
"apply_steering_vectors": [
{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}
],
},
)
residual_stream = decode(completion.activations["residual_stream"])
print(repr(completion.choices[0].text))
print(residual_stream[0, 0, :4].tolist())
```
```text title="Output"
' a '
[2.015625, 4.0, 5.9375, -0.05419921875]
```
The first 3 numbers moved by `2 * [1, 2, 3]`, that is by 2, 4, and 6, within bfloat16's precision. The fourth number did not move. You steered the residual stream at Layer 1.
## Compare with scale 0 [#compare-with-scale-0]
To see what the steer changed in the text, compare it with the same request at `scale: 0`. Do not compare it with the Completion from [Run a Completion](#run-a-completion). A request with a Steering vector takes another path through Inference, and its text can differ from a plain request's for that alone. A scale of 0 adds nothing, so the two requests differ only in `scale`. This program sends the baseline and prints its text:
```python
import os
import numpy as np
from openai import OpenAI
vector = np.zeros(4096)
vector[:3] = [1, 2, 3]
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
temperature=0,
extra_body={
"output_residual_stream": [1],
"apply_steering_vectors": [
{"activations": [vector.tolist()], "layer_indices": [1], "scale": 0}
],
},
)
print(repr(baseline.choices[0].text))
```
Compare its text with the steered text. A difference comes from your vector, or now and then from vLLM's batching ([Why repeats differ](/docs/concepts/base-models#why-repeats-differ)). If the two texts are equal, the numbers that moved by 2, 4, and 6 still show that the steer ran. [Choose a scale](/docs/concepts/residual-stream#choose-a-scale) helps you find a scale that changes the text in the way you want.
## Next steps [#next-steps]
[Read Activations](/docs/activations) explains the encoding of the residual stream, and the checks to run before you use it. [Steer a Completion](/docs/steering) gives every field of a Steering vector, and shows how to check a steer. [Steer with a contrast vector](/docs/cookbook/contrast-steering) builds a Steering vector from two sets of prompts. [Sample many rollouts](/docs/tutorials/sample-many-rollouts) is the next lesson.
# Tutorials (/docs/tutorials)
A tutorial is a lesson. You follow its steps in order, and each step shows the output to expect, so you can check your work as you go. Each lesson links to the guide that explains a step in full.
Start with the [Quickstart](/docs/quickstart), which sends your first Completion. [Your first steered Completion](/docs/tutorials/first-steered-completion) starts with no access at all, and ends with a steered residual stream. [Sample many rollouts](/docs/tutorials/sample-many-rollouts) needs an API key with tokens.
You need curl, and Python with the `openai` SDK. Some steps use `jq` or NumPy, and each lesson says which ones. Each output is one sample that `meta-llama/Llama-3.1-8B` wrote, so your outputs differ. A step at `temperature` 0 gives the likeliest tokens, and vLLM's batching can still change a token now and then.
Send a prompt, read the Completion and its log probabilities, and stream the
tokens.
Get access, make an API key, run a Completion, read its Activations, and
steer the residual stream.
Sample two prompts in one request, then in many requests at once, then in a
Job.
# Sample many rollouts (/docs/tutorials/sample-many-rollouts)
In this tutorial, you sample many rollouts of two prompts. A rollout is one continuation that the model samples. You sample them in one request, then in many requests at once. At the end, you send the same requests as a Job, which runs in the background and keeps the results.
You need an API key with tokens in `ACS_API_KEY`. [Your first steered Completion](/docs/tutorials/first-steered-completion) gets you one. You also need curl, `jq`, Python, and the `openai` SDK.
## Sample two prompts in one request [#sample-two-prompts-in-one-request]
Send a list of prompts, and set `n` to the number of choices for each prompt. Choice `i` of prompt `p` has the index `p * n + i`.
```bash tab="curl"
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}' \
| jq -c '.choices[] | [.index, .text]'
```
```python tab="Python"
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b", prompt=["The sky is", "The grass is"], max_tokens=2, n=2
)
for choice in completion.choices:
prompt, sample = divmod(choice.index, 2)
print(prompt, sample, repr(choice.text))
print(completion.usage.prompt_tokens, completion.usage.completion_tokens)
```
The curl command prints the index and the text of each of the 4 choices:
```text title="curl output"
[0," falling,"]
[1," always blue"]
[2," greener"]
[3," . ."]
```
The Python program prints the number of each choice's prompt, the number of the choice for that prompt, and its text. Then it prints the prompt tokens and the generated tokens of the whole request:
```text title="Python output"
0 0 ' a-going'
0 1 ' the limit'
1 0 ' always gre'
1 1 ' good,'
8 8
```
`usage` adds up the tokens of all the prompts and all the choices. `prompt_tokens` counts each prompt once, whatever `n` is.
At `temperature` 1, the model samples each choice on its own, so the texts usually differ.
The Completion comes back when its last choice ends. One Hold covers all its choices, as [The Hold](/docs/concepts/budgets#the-hold) explains.
## Send many requests at once [#send-many-requests-at-once]
Send each request on its own connection. Each one comes back when its own Completion ends. These examples send the same request 8 times at once.
```bash tab="curl"
for i in 1 2 3 4 5 6 7 8; do
curl -s "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}' \
| jq -c '.choices[0].text' &
done
wait
```
```python tab="Python"
import os
from concurrent.futures import ThreadPoolExecutor
from openai import OpenAI
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
def rollout(_: int) -> str:
completion = client.completions.create(
model="llama-8b", prompt="The sky is", max_tokens=2
)
return completion.choices[0].text
with ThreadPoolExecutor(max_workers=8) as pool:
for text in pool.map(rollout, range(8)):
print(repr(text))
```
The curl commands print the text of each Completion, as `jq` reads it:
```text title="curl output"
" the limit"
" no place"
" no longer"
" blue sometimes"
" the color"
" the limit"
" a wonder"
" the limit"
```
The Python program sends 8 requests of its own, and prints their texts:
```text title="Python output"
' a distinct'
' growing over'
' falling!'
' clear and'
' melty'
' bright,'
' brightly lit'
' falling...'
```
Your API key's [traffic limits](/docs/limits#traffic) set how many requests run at once. Each running request holds its tokens until it ends, so the requests share what you have left. [Handle the end of your Budget](/docs/cookbook/budget-cap) shows what to do when they no longer fit.
## Run the same work as a Job [#run-the-same-work-as-a-job]
A Job runs one Completion request in the API and keeps its result. Your script can stop, and the Job keeps running.
1. Submit the two prompts of the first step as one request with `n: 2`:
```bash
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: rollouts-001" \
-d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}'
```
The API answers 202 with the queued Job:
```json title="Job"
{
"cancel_requested": false,
"created_at": "2026-11-02T18:35:08.037+00:00",
"expires_at": null,
"finished_at": null,
"id": "01a19d8b-338e-7a52-9329-6190679221fa",
"model": "llama-8b",
"request_id": "req_job_01a19d8b338e7a5293296190679221fa",
"result_expired": false,
"result_url": "/v1/completion-jobs/01a19d8b-338e-7a52-9329-6190679221fa/result",
"started_at": null,
"status": "queued",
"status_url": "/v1/completion-jobs/01a19d8b-338e-7a52-9329-6190679221fa",
"usage_pending": false
}
```
2. Put the `id` in a shell variable for the next commands, for example `JOB_ID=01a19d8b-338e-7a52-9329-6190679221fa`.
3. Check on the Job:
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"
```
When the Job ends, its `status` is `succeeded`:
```json title="Job"
{
"id": "01a19d8b-338e-7a52-9329-6190679221fa",
"request_id": "req_job_01a19d8b338e7a5293296190679221fa",
"model": "llama-8b",
"status": "succeeded",
"cancel_requested": false,
"created_at": "2026-11-02T18:35:08.037+00:00",
"started_at": "2026-11-02T18:35:08.065+00:00",
"finished_at": "2026-11-02T18:35:08.132+00:00",
"expires_at": "2026-12-02T18:35:08.132+00:00",
"result_expired": false,
"status_url": "/v1/completion-jobs/01a19d8b-338e-7a52-9329-6190679221fa",
"result_url": "/v1/completion-jobs/01a19d8b-338e-7a52-9329-6190679221fa/result",
"usage_pending": false
}
```
4. Fetch the result, and print the text of each choice with `jq`:
```bash
curl "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY" | jq -c '[.choices[].text]'
```
The result is one Completion with the `n` choices of each prompt, in the order of `prompt`. If the request failed, the result is the error object with its own status, so check for an `error` key before you read `choices`.
The Job takes its Hold when it runs, not when you submit it, as [Holds of background work](/docs/concepts/budgets#holds-of-background-work) explains.
## Next steps [#next-steps]
[Run a Job](/docs/jobs) shows how to check on, cancel, and fetch a Job, and read Activations in one. [Run a Job from a shell script](/docs/cookbook/job-script) submits a Job, waits until it ends, and saves its result. [Handle the end of your Budget](/docs/cookbook/budget-cap) keeps a script running near the end of your tokens.
# List Models (/docs/reference/models/models.listModels)
```json
{
"components": {
"schemas": {
"Capabilities": {
"type": "object",
"properties": {
"max_model_len": {
"type": "integer",
"description": "The most tokens of prompt and generated text together."
},
"max_logprobs": {
"type": "integer",
"description": "The largest `logprobs` or positive `prompt_logprobs`."
},
"max_output_work_tokens": {
"type": "integer",
"description": "Prompts × `n` × `max_tokens` of one request, at most."
},
"max_inflight_per_key": {
"type": "integer",
"description": "Completions that run at once, for each API key."
},
"max_queued_per_key": {
"type": "integer",
"description": "Completions that wait for a place, for each API key."
},
"rate_limit_per_minute": {
"type": "integer",
"description": "Requests a minute that start work, for each API key, and as many others."
},
"logprobs": {
"type": "boolean"
},
"prompt_logprobs": {
"type": "boolean"
},
"prompt_logprobs_full_vocab": {
"type": "boolean",
"description": "The model takes `prompt_logprobs: -1`, the whole vocabulary."
},
"full_vocab_max_prompt_tokens": {
"type": "integer",
"description": "`prompt_logprobs: -1`: at most this many prompt tokens in all × `n`, or 16 with `output_residual_stream`."
},
"activations": {
"type": "boolean",
"description": "The model takes `output_residual_stream` and `apply_steering_vectors`."
},
"max_steering_vectors": {
"type": "integer",
"description": "`apply_steering_vectors`: at most this many vectors."
},
"max_activation_prompt_tokens": {
"type": "integer",
"description": "A capture of every Layer: at most this many tokens of the prompt and `max_tokens`. Fewer Layers take more tokens: Layers × tokens × the hidden size is at most 2^25."
},
"n_layers": {
"type": "integer",
"description": "The model's Layers, from 0 to this number less 1."
},
"hidden_size": {
"type": "integer",
"description": "The numbers in one vector of the residual stream, for one Layer and one token: the width of a steering vector."
},
"chat_template": {
"type": "null",
"description": "Always null: the API applies no chat template."
}
},
"required": [
"max_model_len",
"max_logprobs",
"max_output_work_tokens",
"max_inflight_per_key",
"max_queued_per_key",
"rate_limit_per_minute",
"logprobs",
"prompt_logprobs",
"prompt_logprobs_full_vocab",
"full_vocab_max_prompt_tokens",
"activations",
"max_steering_vectors",
"max_activation_prompt_tokens",
"n_layers",
"hidden_size",
"chat_template"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Model": {
"type": "object",
"properties": {
"id": {
"type": "string",
"description": "The Model id to send as `model`, such as `llama-8b`."
},
"object": {
"type": "string",
"enum": [
"model"
]
},
"owned_by": {
"type": "string",
"enum": [
"acs"
]
},
"served_model_name": {
"type": "string",
"description": "The Checkpoint that Inference serves, which each answer's `model` names."
},
"gpu_shape": {
"type": "string",
"description": "GPUs and their type, such as `1xL40S` or `8xH200`."
},
"status": {
"type": "string",
"enum": [
"live"
]
},
"default": {
"type": "boolean",
"description": "A request without `model` runs on this model."
},
"capabilities": {
"$ref": "#/components/schemas/Capabilities"
}
},
"required": [
"id",
"object",
"owned_by",
"served_model_name",
"gpu_shape",
"status",
"default",
"capabilities"
],
"additionalProperties": false
},
"ModelList": {
"type": "object",
"properties": {
"object": {
"type": "string",
"enum": [
"list"
]
},
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Model"
}
}
},
"required": [
"object",
"data"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/models": {
"get": {
"tags": [
"models"
],
"operationId": "models.listModels",
"parameters": [],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "ModelList",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ModelList"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "List the Base models that the API serves, each with its Model id, the Checkpoint it serves, its GPUs, and its capabilities: its context length, Layers, hidden size, and the limits of a request to it. `default` marks the model of a request without `model`. The API lists them from its settings and asks no Inference, so a list never starts a GPU."
}
}
}
}
```
# Retrieve Model (/docs/reference/models/models.retrieveModel)
```json
{
"components": {
"schemas": {
"Capabilities": {
"type": "object",
"properties": {
"max_model_len": {
"type": "integer",
"description": "The most tokens of prompt and generated text together."
},
"max_logprobs": {
"type": "integer",
"description": "The largest `logprobs` or positive `prompt_logprobs`."
},
"max_output_work_tokens": {
"type": "integer",
"description": "Prompts × `n` × `max_tokens` of one request, at most."
},
"max_inflight_per_key": {
"type": "integer",
"description": "Completions that run at once, for each API key."
},
"max_queued_per_key": {
"type": "integer",
"description": "Completions that wait for a place, for each API key."
},
"rate_limit_per_minute": {
"type": "integer",
"description": "Requests a minute that start work, for each API key, and as many others."
},
"logprobs": {
"type": "boolean"
},
"prompt_logprobs": {
"type": "boolean"
},
"prompt_logprobs_full_vocab": {
"type": "boolean",
"description": "The model takes `prompt_logprobs: -1`, the whole vocabulary."
},
"full_vocab_max_prompt_tokens": {
"type": "integer",
"description": "`prompt_logprobs: -1`: at most this many prompt tokens in all × `n`, or 16 with `output_residual_stream`."
},
"activations": {
"type": "boolean",
"description": "The model takes `output_residual_stream` and `apply_steering_vectors`."
},
"max_steering_vectors": {
"type": "integer",
"description": "`apply_steering_vectors`: at most this many vectors."
},
"max_activation_prompt_tokens": {
"type": "integer",
"description": "A capture of every Layer: at most this many tokens of the prompt and `max_tokens`. Fewer Layers take more tokens: Layers × tokens × the hidden size is at most 2^25."
},
"n_layers": {
"type": "integer",
"description": "The model's Layers, from 0 to this number less 1."
},
"hidden_size": {
"type": "integer",
"description": "The numbers in one vector of the residual stream, for one Layer and one token: the width of a steering vector."
},
"chat_template": {
"type": "null",
"description": "Always null: the API applies no chat template."
}
},
"required": [
"max_model_len",
"max_logprobs",
"max_output_work_tokens",
"max_inflight_per_key",
"max_queued_per_key",
"rate_limit_per_minute",
"logprobs",
"prompt_logprobs",
"prompt_logprobs_full_vocab",
"full_vocab_max_prompt_tokens",
"activations",
"max_steering_vectors",
"max_activation_prompt_tokens",
"n_layers",
"hidden_size",
"chat_template"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Model": {
"type": "object",
"properties": {
"id": {
"type": "string",
"description": "The Model id to send as `model`, such as `llama-8b`."
},
"object": {
"type": "string",
"enum": [
"model"
]
},
"owned_by": {
"type": "string",
"enum": [
"acs"
]
},
"served_model_name": {
"type": "string",
"description": "The Checkpoint that Inference serves, which each answer's `model` names."
},
"gpu_shape": {
"type": "string",
"description": "GPUs and their type, such as `1xL40S` or `8xH200`."
},
"status": {
"type": "string",
"enum": [
"live"
]
},
"default": {
"type": "boolean",
"description": "A request without `model` runs on this model."
},
"capabilities": {
"$ref": "#/components/schemas/Capabilities"
}
},
"required": [
"id",
"object",
"owned_by",
"served_model_name",
"gpu_shape",
"status",
"default",
"capabilities"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/models/{model_id}": {
"get": {
"tags": [
"models"
],
"operationId": "models.retrieveModel",
"parameters": [
{
"name": "model_id",
"in": "path",
"schema": {
"type": "string",
"description": "The Model id from GET /v1/models, such as `llama-8b`, or its `served_model_name`, with its slash as it is or as `%2F`."
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Model",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Model"
}
}
}
},
"400": {
"description": "GET /v1/models does not list the model: `model_not_found`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Return one model of GET /v1/models, as OpenAI's retrieve does. It asks no Inference."
}
}
}
}
```
# Get Model Status (/docs/reference/models/models.getModelStatus)
```json
{
"components": {
"schemas": {
"BootStage": {
"type": "object",
"properties": {
"stage": {
"type": "string",
"enum": [
"waiting_for_gpu",
"container_started",
"weights_loading",
"weights_loaded"
]
},
"stage_label": {
"type": "string"
},
"stage_age_s": {
"type": "integer",
"description": "How long the stage has run, in seconds."
}
},
"required": [
"stage",
"stage_label",
"stage_age_s"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"ModelStatus": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"object": {
"type": "string",
"enum": [
"model.status"
]
},
"warm": {
"type": "boolean",
"description": "A container of this model serves now, so its next request starts at once."
},
"boot": {
"anyOf": [
{
"$ref": "#/components/schemas/BootStage"
},
{
"type": "null"
}
],
"description": "The stage of the start that is under way, or null when none is."
},
"activation_boot": {
"anyOf": [
{
"$ref": "#/components/schemas/BootStage"
},
{
"type": "null"
}
],
"description": "The same as `boot`."
}
},
"required": [
"id",
"object",
"warm",
"boot",
"activation_boot"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/models/{model_id}/status": {
"get": {
"tags": [
"models"
],
"operationId": "models.getModelStatus",
"parameters": [
{
"name": "model_id",
"in": "path",
"schema": {
"type": "string",
"description": "The Model id from GET /v1/models, such as `llama-8b`, or its `served_model_name`, with its slash as it is or as `%2F`."
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "ModelStatus",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ModelStatus"
}
}
}
},
"400": {
"description": "GET /v1/models does not list the model: `model_not_found`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Say whether the model's next request most likely starts at once (`warm`), or which stage its start has reached (`boot`). The API answers from what it saw and asks no Inference, so the route never starts a GPU."
}
}
}
}
```
# Create Completion (/docs/reference/completions/completions.createCompletion)
```json
{
"components": {
"schemas": {
"Activations": {
"type": "object",
"properties": {
"residual_stream": {
"$ref": "#/components/schemas/Tensor"
}
},
"required": [
"residual_stream"
],
"additionalProperties": false
},
"ActivationsEncoded": {
"anyOf": [
{
"$ref": "#/components/schemas/Activations"
},
{
"type": "null"
}
]
},
"Completion": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"object": {
"type": "string",
"enum": [
"text_completion"
]
},
"created": {
"type": "integer"
},
"model": {
"type": "string"
},
"choices": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CompletionChoice"
}
},
"usage": {
"$ref": "#/components/schemas/CompletionUsage"
},
"activations": {
"$ref": "#/components/schemas/ActivationsEncoded"
}
},
"required": [
"id",
"object",
"created",
"model",
"choices",
"usage"
],
"additionalProperties": false
},
"CompletionChoice": {
"type": "object",
"properties": {
"index": {
"type": "integer"
},
"text": {
"type": "string"
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
]
},
"finish_reason": {
"type": "string",
"description": "`stop` or `length`; Inference can add reasons."
},
"stop_reason": {
"anyOf": [
{
"type": "string"
},
{
"type": "integer"
},
{
"type": "null"
}
],
"description": "vLLM: the `stop` text that ended the choice, or the id of a stop token. null when the end-of-sequence token or `max_tokens` ended it, and in each chunk before the choice's last."
},
"prompt_logprobs": {
"anyOf": [
{
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"$ref": "#/components/schemas/PromptLogprob"
}
},
{
"type": "null"
}
]
}
},
{
"type": "null"
}
],
"description": "Per prompt token, keyed by token id. The first is null."
}
},
"required": [
"index",
"text",
"logprobs",
"finish_reason"
],
"additionalProperties": false
},
"CompletionPayload": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`. Left out, the model that GET /v1/models marks `default`."
},
"prompt": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "integer"
}
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
}
},
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "integer"
}
}
}
],
"description": "Text or a list of token ids, or a list of several such prompts. Each prompt gets `n` choices: choice `i` of prompt `p` has the index `p * n + i`."
},
"max_tokens": {
"type": [
"integer",
"null"
],
"maximum": 32000,
"description": "Generate at most this many tokens for each choice, at most 32000 in all: the number of prompts × `n` × `max_tokens`. Without `stream`, at most 15600, what Inference finishes within the API's 14-minute window. Left out, it is the model's context left after the prompt, at most 32000 (15600 without `stream`); a request of more than one prompt, or with `n` above 1, must set it. Near the end of your token budget, the API lowers it to what fits and says so in `X-Acs-Max-Tokens-Clamped`. null is the same as leaving it out."
},
"temperature": {
"type": "number",
"minimum": 0,
"maximum": 2,
"description": "Sample less likely tokens more often; 0 takes the most likely token each time. From 0 to 2, as vLLM takes it."
},
"n": {
"type": "integer",
"maximum": 128,
"description": "How many choices to generate for each prompt; at most 128 in all."
},
"echo": {
"type": "boolean",
"description": "Put the prompt before the generated text."
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"prompt_logprobs": {
"type": [
"integer",
"null"
],
"minimum": -1,
"maximum": 100,
"description": "vLLM: log probabilities of the prompt tokens, and of this many top tokens at each, at most 100, and 33554432 in all: prompt tokens × `n` × (this + 1), and `logprobs` + 1 more with `echo`. -1 for the whole vocabulary, for at most 256 prompt tokens in all × `n` (16 with `output_residual_stream`). The API counts a text's tokens with the model's tokenizer, as Inference does, and token ids exactly. Not with `stream: true`."
},
"top_p": {
"type": "number",
"exclusiveMinimum": 0,
"maximum": 1,
"description": "Sample from the most likely tokens up to this probability; more than 0 and at most 1, as OpenAI and vLLM take it."
},
"top_k": {
"type": "integer",
"minimum": -1,
"description": "vLLM: sample from this many most likely tokens; 0 or -1 for all."
},
"min_p": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "vLLM: leave out tokens less likely than this fraction of the most likely; from 0 to 1."
},
"presence_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token that is already in the text less likely; from -2 to 2."
},
"frequency_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token less likely each time it is already in the text; from -2 to 2."
},
"repetition_penalty": {
"type": "number",
"exclusiveMinimum": 0,
"description": "vLLM: divide the scores of repeated tokens by this, more than 0; 1 for none."
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
},
"stream": {
"type": "boolean",
"description": "Send Server-Sent Events as tokens arrive."
},
"stream_options": {
"anyOf": [
{
"$ref": "#/components/schemas/StreamOptions"
},
{
"type": "null"
}
],
"description": "Only with `stream: true`."
},
"priority": {
"anyOf": [
{
"type": "null"
},
{
"type": "boolean"
},
{
"type": "integer",
"minimum": 0
},
{
"type": "string",
"pattern": "^\\d{1,9}$"
}
],
"description": "vLLM: a lower number runs first, from 0. Requests of equal priority wait in order of arrival. null is 0; a text of digits and true or false count as their number."
},
"user": {
"type": [
"string",
"null"
],
"pattern": "^[^\\u0000]*$",
"description": "Your id for the person who asked."
},
"add_special_tokens": {
"type": "boolean",
"description": "vLLM: add the model's special tokens, such as BOS, to text."
},
"output_residual_stream": {
"anyOf": [
{
"type": "null"
},
{
"type": "boolean",
"enum": [
false
]
},
{
"type": "boolean",
"enum": [
true
]
},
{
"type": "array",
"items": {
"type": "integer",
"minimum": 0
},
"minItems": 1
}
],
"description": "vllm-lens: return the residual stream at these layers, in any order, each once, or `true` for every layer of the model, or `false` for none. Layer 0 is the first, and the last is `capabilities.n_layers` - 1 in GET /v1/models. A capture is at most 33554432 numbers (64 MiB as bfloat16): layers × tokens × the model's hidden size, where the tokens are the prompt's (as the model's tokenizer counts them) and `max_tokens`. So layers × tokens is at most 8192 for llama-8b (hidden size 4096), 2048 for llama-405b (hidden size 16384), 10922 for trinity-truebase (hidden size 3072); a model that the catalogue does not have counts as its largest hidden size, 16384."
},
"apply_steering_vectors": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/SteeringVector"
},
"minItems": 1,
"maxItems": 8
},
{
"type": "null"
}
],
"description": "vllm-lens: add these vectors to the residual stream. At most 8 vectors, and 2097152 numbers in all (8 MiB as float32): layers × positions × the model's hidden size."
}
},
"required": [
"prompt"
],
"additionalProperties": false
},
"CompletionUsage": {
"type": "object",
"properties": {
"prompt_tokens": {
"type": "integer",
"minimum": 0,
"maximum": 2147483647
},
"completion_tokens": {
"type": "integer",
"minimum": 0,
"maximum": 2147483647
},
"total_tokens": {
"type": "integer",
"description": "`prompt_tokens` + `completion_tokens`."
}
},
"required": [
"prompt_tokens",
"completion_tokens",
"total_tokens"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"PromptLogprob": {
"type": "object",
"properties": {
"logprob": {
"type": "number"
},
"rank": {
"type": [
"integer",
"null"
]
},
"decoded_token": {
"type": [
"string",
"null"
]
}
},
"required": [
"logprob",
"rank",
"decoded_token"
],
"additionalProperties": false
},
"SteeringTensor": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The numbers in C order, compressed with zstd in one frame, in base64."
},
"dtype": {
"type": "string",
"description": "`int16` for bfloat16 numbers, with `original_dtype` `torch.bfloat16`, or `float32`."
},
"original_dtype": {
"type": "string",
"description": "`torch.bfloat16` for `int16` numbers."
},
"shape": {
"type": "array",
"items": {
"type": "integer"
},
"description": "`[layers, hidden size]`, or `[layers, positions, hidden size]`."
},
"compression": {
"type": "string",
"enum": [
"zstd"
],
"description": "Always `zstd`."
}
},
"required": [
"data",
"dtype",
"original_dtype",
"shape",
"compression"
],
"additionalProperties": false
},
"SteeringVector": {
"type": "object",
"properties": {
"activations": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
}
},
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
}
}
},
{
"$ref": "#/components/schemas/SteeringTensor"
}
],
"description": "One row for each layer in `layer_indices`. A row is a vector of the model's hidden size, added at every token, or a list of such vectors, one for each position. Send the rows as numbers, or encoded as the old platform's `encode_vector` writes them."
},
"layer_indices": {
"type": "array",
"items": {
"type": "integer",
"minimum": 0
},
"description": "The Layers to add the rows of `activations` at, one for each row. Layer 0 is the first."
},
"scale": {
"type": "number",
"description": "Multiply the vector by this number."
},
"norm_match": {
"type": "boolean",
"description": "First scale the vector to the norm of the residual stream at each token."
},
"position_indices": {
"anyOf": [
{
"type": "array",
"items": {
"type": "integer",
"minimum": 0
}
},
{
"type": "null"
}
],
"description": "For a list of vectors: their token positions, counted from 0 at the first prompt token. Default: 0, 1, 2, and so on. Inference refuses a position after the prompt's tokens + `max_tokens` - 2: the last generated token never runs through the model."
},
"position_spans": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"prefixItems": [
{
"type": "integer",
"minimum": 0
},
{
"type": "integer",
"minimum": 0
}
],
"maxItems": 2,
"minItems": 2
},
"minItems": 1
},
{
"type": "null"
}
],
"description": "For one vector in each row: the spans of positions to add it at, each [start, end) with end after start, counted from 0 at the first prompt token. At most 4096 positions in all, and no span may overlap another. Not with `position_indices`."
}
},
"required": [
"activations",
"layer_indices"
],
"additionalProperties": false
},
"StreamOptions": {
"type": "object",
"properties": {
"include_usage": {
"type": "boolean",
"description": "Send a last chunk with the usage of the whole request and no choices."
},
"continuous_usage_stats": {
"type": "boolean",
"description": "vLLM: with `include_usage`, also put in each chunk the usage of its choice so far."
}
},
"additionalProperties": false
},
"Tensor": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The bytes of the array in C order, compressed with zstd, in base64."
},
"dtype": {
"type": "string",
"description": "The NumPy type of the bytes. A bfloat16 model's numbers come as `int16`."
},
"original_dtype": {
"type": "string",
"description": "The model's type, for example `torch.bfloat16`."
},
"shape": {
"type": "array",
"items": {
"type": "integer"
}
},
"compression": {
"type": "string",
"enum": [
"zstd"
]
}
},
"required": [
"data",
"dtype",
"original_dtype",
"shape",
"compression"
],
"additionalProperties": false,
"description": "Shape [layers, tokens, hidden size]: the output of each requested layer, for each prompt and generated token except the last generated one."
}
}
},
"paths": {
"/v1/completions": {
"post": {
"tags": [
"completions"
],
"operationId": "completions.createCompletion",
"parameters": [
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Completion",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Completion"
}
},
"text/event-stream": {
"schema": {
"anyOf": [
{
"type": "object",
"properties": {
"data": {
"type": "string",
"contentMediaType": "application/json"
},
"event": {
"type": "string",
"enum": [
"message"
]
}
},
"required": [
"data",
"event"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"data": {
"type": "string",
"enum": [
"[DONE]"
]
},
"event": {
"type": "string",
"enum": [
"message"
]
}
},
"required": [
"data",
"event"
],
"additionalProperties": false
}
]
},
"x-effect-stream": {
"encoding": "sse",
"causeSchema": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Fail"
]
},
"error": {
"not": {}
}
},
"required": [
"_tag",
"error"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Die"
]
},
"defect": {}
},
"required": [
"_tag",
"defect"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Interrupt"
]
},
"fiberId": {
"type": [
"number",
"null"
]
}
},
"required": [
"_tag",
"fiberId"
],
"additionalProperties": false
}
]
}
},
"errorSchema": {
"not": {}
},
"failureEvent": "effect/http-api/stream/failure"
}
}
}
},
"400": {
"description": "The API serves no such model: `model_not_found`. Or admission refuses a value, such as a Layer that the model does not have: `invalid_request`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds. Or this API key has its most Completions running and waiting: `queue_full`, `Retry-After` 5. Or Inference is busy, or the token budget cannot hold the request: `budget_exceeded`. Or the Spend stop is on, so the API starts no new GPU work: `spend_cap`, `Retry-After` 3600, `x-should-retry: false`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"502": {
"description": "Inference failed, refused the API's request, or did not answer.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "InferenceError",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"504": {
"description": "InferenceError",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Continue a prompt on a Base model. With `stream: true`, Server-Sent Events: one CompletionChunk per event, then `[DONE]`; with `output_residual_stream`, the last chunk has no choices and has the activations. If Inference fails, the error object with `type` `upstream_error`, the `code` that names the failure, and `retryable`; during a stream, that error is the last event before `[DONE]`.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionPayload"
}
}
},
"required": true
}
}
}
}
}
```
# Create Completion Job (/docs/reference/completionjobs/completionJobs.createCompletionJob)
```json
{
"components": {
"schemas": {
"CompletionJob": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"request_id": {
"type": "string",
"description": "The id to send when you report a problem with this job."
},
"model": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"succeeded",
"failed",
"cancelled"
],
"description": "`queued` and `running` until the job ends. Then `succeeded` when its completion answered 200, `failed` when it answered an error, or `cancelled`."
},
"cancel_requested": {
"type": "boolean",
"description": "Whether the job was cancelled."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When the job started to run. `null` until it starts."
},
"finished_at": {
"type": [
"string",
"null"
],
"description": "When the job ended. `null` until it ends."
},
"expires_at": {
"type": [
"string",
"null"
],
"description": "When the result of the ended job expires: 30 days after it ended. `null` until it ends."
},
"result_expired": {
"type": "boolean"
},
"status_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id} for this job."
},
"result_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id}/result for this job."
},
"usage_pending": {
"type": "boolean",
"description": "Whether the job holds tokens of your budget: `true` while it runs."
}
},
"required": [
"id",
"request_id",
"model",
"status",
"cancel_requested",
"created_at",
"started_at",
"finished_at",
"expires_at",
"result_expired",
"status_url",
"result_url",
"usage_pending"
],
"additionalProperties": false,
"description": "The job is queued. `Location` is The path of this job: `/v1/completion-jobs/{job_id}`."
},
"CompletionJobRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"prompt": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "integer"
}
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
}
},
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "integer"
}
}
}
],
"description": "Text or a list of token ids, or a list of several such prompts. Each prompt gets `n` choices: choice `i` of prompt `p` has the index `p * n + i`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number",
"minimum": 0,
"maximum": 2,
"description": "Sample less likely tokens more often; 0 takes the most likely token each time. From 0 to 2, as vLLM takes it."
},
"n": {
"type": "integer",
"maximum": 128,
"description": "How many choices to generate for each prompt; at most 128 in all."
},
"echo": {
"type": "boolean",
"description": "Put the prompt before the generated text."
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"prompt_logprobs": {
"type": [
"integer",
"null"
],
"minimum": -1,
"maximum": 100,
"description": "vLLM: log probabilities of the prompt tokens, and of this many top tokens at each, at most 100, and 33554432 in all: prompt tokens × `n` × (this + 1), and `logprobs` + 1 more with `echo`. -1 for the whole vocabulary, for at most 256 prompt tokens in all × `n` (16 with `output_residual_stream`). The API counts a text's tokens with the model's tokenizer, as Inference does, and token ids exactly. Not with `stream: true`."
},
"top_p": {
"type": "number",
"exclusiveMinimum": 0,
"maximum": 1,
"description": "Sample from the most likely tokens up to this probability; more than 0 and at most 1, as OpenAI and vLLM take it."
},
"top_k": {
"type": "integer",
"minimum": -1,
"description": "vLLM: sample from this many most likely tokens; 0 or -1 for all."
},
"min_p": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "vLLM: leave out tokens less likely than this fraction of the most likely; from 0 to 1."
},
"presence_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token that is already in the text less likely; from -2 to 2."
},
"frequency_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token less likely each time it is already in the text; from -2 to 2."
},
"repetition_penalty": {
"type": "number",
"exclusiveMinimum": 0,
"description": "vLLM: divide the scores of repeated tokens by this, more than 0; 1 for none."
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
},
"stream": {
"type": "boolean",
"enum": [
false
],
"description": "A completion job is never streamed."
},
"stream_options": {
"anyOf": [
{
"$ref": "#/components/schemas/StreamOptions"
},
{
"type": "null"
}
],
"description": "Only with `stream: true`."
},
"priority": {
"anyOf": [
{
"type": "null"
},
{
"type": "boolean"
},
{
"type": "integer",
"minimum": 0
},
{
"type": "string",
"pattern": "^\\d{1,9}$"
}
],
"description": "vLLM: a lower number runs first, from 0. Requests of equal priority wait in order of arrival. null is 0; a text of digits and true or false count as their number."
},
"user": {
"type": [
"string",
"null"
],
"pattern": "^[^\\u0000]*$",
"description": "Your id for the person who asked."
},
"add_special_tokens": {
"type": "boolean",
"description": "vLLM: add the model's special tokens, such as BOS, to text."
},
"output_residual_stream": {
"anyOf": [
{
"type": "null"
},
{
"type": "boolean",
"enum": [
false
]
},
{
"type": "boolean",
"enum": [
true
]
},
{
"type": "array",
"items": {
"type": "integer",
"minimum": 0
},
"minItems": 1
}
],
"description": "vllm-lens: return the residual stream at these layers, in any order, each once, or `true` for every layer of the model, or `false` for none. Layer 0 is the first, and the last is `capabilities.n_layers` - 1 in GET /v1/models. A capture is at most 33554432 numbers (64 MiB as bfloat16): layers × tokens × the model's hidden size, where the tokens are the prompt's (as the model's tokenizer counts them) and `max_tokens`. So layers × tokens is at most 8192 for llama-8b (hidden size 4096), 2048 for llama-405b (hidden size 16384), 10922 for trinity-truebase (hidden size 3072); a model that the catalogue does not have counts as its largest hidden size, 16384."
},
"apply_steering_vectors": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/SteeringVector"
},
"minItems": 1,
"maxItems": 8
},
{
"type": "null"
}
],
"description": "vllm-lens: add these vectors to the residual stream. At most 8 vectors, and 2097152 numbers in all (8 MiB as float32): layers × positions × the model's hidden size."
}
},
"required": [
"model",
"prompt"
],
"additionalProperties": false
},
"CompletionJob_1": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"request_id": {
"type": "string",
"description": "The id to send when you report a problem with this job."
},
"model": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"succeeded",
"failed",
"cancelled"
],
"description": "`queued` and `running` until the job ends. Then `succeeded` when its completion answered 200, `failed` when it answered an error, or `cancelled`."
},
"cancel_requested": {
"type": "boolean",
"description": "Whether the job was cancelled."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When the job started to run. `null` until it starts."
},
"finished_at": {
"type": [
"string",
"null"
],
"description": "When the job ended. `null` until it ends."
},
"expires_at": {
"type": [
"string",
"null"
],
"description": "When the result of the ended job expires: 30 days after it ended. `null` until it ends."
},
"result_expired": {
"type": "boolean"
},
"status_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id} for this job."
},
"result_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id}/result for this job."
},
"usage_pending": {
"type": "boolean",
"description": "Whether the job holds tokens of your budget: `true` while it runs."
}
},
"required": [
"id",
"request_id",
"model",
"status",
"cancel_requested",
"created_at",
"started_at",
"finished_at",
"expires_at",
"result_expired",
"status_url",
"result_url",
"usage_pending"
],
"additionalProperties": false,
"description": "The `Idempotency-Key` and the request came before: the job they made, as it is now."
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"SteeringTensor": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The numbers in C order, compressed with zstd in one frame, in base64."
},
"dtype": {
"type": "string",
"description": "`int16` for bfloat16 numbers, with `original_dtype` `torch.bfloat16`, or `float32`."
},
"original_dtype": {
"type": "string",
"description": "`torch.bfloat16` for `int16` numbers."
},
"shape": {
"type": "array",
"items": {
"type": "integer"
},
"description": "`[layers, hidden size]`, or `[layers, positions, hidden size]`."
},
"compression": {
"type": "string",
"enum": [
"zstd"
],
"description": "Always `zstd`."
}
},
"required": [
"data",
"dtype",
"original_dtype",
"shape",
"compression"
],
"additionalProperties": false
},
"SteeringVector": {
"type": "object",
"properties": {
"activations": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
}
},
{
"type": "array",
"items": {
"type": "array",
"items": {
"type": "array",
"items": {
"type": "number"
}
}
}
},
{
"$ref": "#/components/schemas/SteeringTensor"
}
],
"description": "One row for each layer in `layer_indices`. A row is a vector of the model's hidden size, added at every token, or a list of such vectors, one for each position. Send the rows as numbers, or encoded as the old platform's `encode_vector` writes them."
},
"layer_indices": {
"type": "array",
"items": {
"type": "integer",
"minimum": 0
},
"description": "The Layers to add the rows of `activations` at, one for each row. Layer 0 is the first."
},
"scale": {
"type": "number",
"description": "Multiply the vector by this number."
},
"norm_match": {
"type": "boolean",
"description": "First scale the vector to the norm of the residual stream at each token."
},
"position_indices": {
"anyOf": [
{
"type": "array",
"items": {
"type": "integer",
"minimum": 0
}
},
{
"type": "null"
}
],
"description": "For a list of vectors: their token positions, counted from 0 at the first prompt token. Default: 0, 1, 2, and so on. Inference refuses a position after the prompt's tokens + `max_tokens` - 2: the last generated token never runs through the model."
},
"position_spans": {
"anyOf": [
{
"type": "array",
"items": {
"type": "array",
"prefixItems": [
{
"type": "integer",
"minimum": 0
},
{
"type": "integer",
"minimum": 0
}
],
"maxItems": 2,
"minItems": 2
},
"minItems": 1
},
{
"type": "null"
}
],
"description": "For one vector in each row: the spans of positions to add it at, each [start, end) with end after start, counted from 0 at the first prompt token. At most 4096 positions in all, and no span may overlap another. Not with `position_indices`."
}
},
"required": [
"activations",
"layer_indices"
],
"additionalProperties": false
},
"StreamOptions": {
"type": "object",
"properties": {
"include_usage": {
"type": "boolean",
"description": "Send a last chunk with the usage of the whole request and no choices."
},
"continuous_usage_stats": {
"type": "boolean",
"description": "vLLM: with `include_usage`, also put in each chunk the usage of its choice so far."
}
},
"additionalProperties": false
}
}
},
"paths": {
"/v1/completion-jobs": {
"post": {
"tags": [
"completionJobs"
],
"operationId": "completionJobs.createCompletionJob",
"parameters": [
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "The `Idempotency-Key` and the request came before: the job they made, as it is now.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJob_1"
}
}
}
},
"202": {
"description": "The job is queued. `Location` is The path of this job: `/v1/completion-jobs/{job_id}`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJob"
}
}
}
},
"400": {
"description": "The API serves no such model: `model_not_found`. Or admission refuses a value, such as a Layer that the model does not have: `invalid_request`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds. Or the token budget cannot hold the request: `budget_exceeded`. Or a job cap is full: 8 jobs queued or running for an API key, 32 for a User, 128 in all; 100 ended jobs whose results the API keeps for a User, 1,000 in all (`job_capacity`, `x-should-retry: false`). Or the Spend stop is on, so the API starts no new GPU work: `spend_cap`, `Retry-After` 3600, `x-should-retry: false`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Queue one completion to run on the server, also after your connection closes. The body is a body of POST /v1/completions that does not stream. The request is admitted first, as POST /v1/completions admits it: a model that the API does not serve answers 400 `model_not_found`, a value that admission refuses answers 400, and if the token budget cannot hold it, 429 `budget_exceeded`; nothing is queued. Send an `Idempotency-Key` so that a retry gets the same job: the same key with another request answers 409 `idempotency_conflict`.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJobRequest"
}
}
},
"required": true
}
}
}
}
}
```
# Get Completion Job (/docs/reference/completionjobs/completionJobs.getCompletionJob)
```json
{
"components": {
"schemas": {
"CompletionJob_2": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"request_id": {
"type": "string",
"description": "The id to send when you report a problem with this job."
},
"model": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"succeeded",
"failed",
"cancelled"
],
"description": "`queued` and `running` until the job ends. Then `succeeded` when its completion answered 200, `failed` when it answered an error, or `cancelled`."
},
"cancel_requested": {
"type": "boolean",
"description": "Whether the job was cancelled."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When the job started to run. `null` until it starts."
},
"finished_at": {
"type": [
"string",
"null"
],
"description": "When the job ended. `null` until it ends."
},
"expires_at": {
"type": [
"string",
"null"
],
"description": "When the result of the ended job expires: 30 days after it ended. `null` until it ends."
},
"result_expired": {
"type": "boolean"
},
"status_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id} for this job."
},
"result_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id}/result for this job."
},
"usage_pending": {
"type": "boolean",
"description": "Whether the job holds tokens of your budget: `true` while it runs."
}
},
"required": [
"id",
"request_id",
"model",
"status",
"cancel_requested",
"created_at",
"started_at",
"finished_at",
"expires_at",
"result_expired",
"status_url",
"result_url",
"usage_pending"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/completion-jobs/{job_id}": {
"get": {
"tags": [
"completionJobs"
],
"operationId": "completionJobs.getCompletionJob",
"parameters": [
{
"name": "job_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "CompletionJob",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJob_2"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Return the job and its status. A 200 here does not mean that the completion succeeded: read GET /v1/completion-jobs/{job_id}/result."
}
}
}
}
```
# Get Completion Job Result (/docs/reference/completionjobs/completionJobs.getCompletionJobResult)
```json
{
"components": {
"schemas": {
"Activations": {
"type": "object",
"properties": {
"residual_stream": {
"$ref": "#/components/schemas/Tensor"
}
},
"required": [
"residual_stream"
],
"additionalProperties": false
},
"ActivationsEncoded": {
"anyOf": [
{
"$ref": "#/components/schemas/Activations"
},
{
"type": "null"
}
]
},
"CompletionChoice": {
"type": "object",
"properties": {
"index": {
"type": "integer"
},
"text": {
"type": "string"
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
]
},
"finish_reason": {
"type": "string",
"description": "`stop` or `length`; Inference can add reasons."
},
"stop_reason": {
"anyOf": [
{
"type": "string"
},
{
"type": "integer"
},
{
"type": "null"
}
],
"description": "vLLM: the `stop` text that ended the choice, or the id of a stop token. null when the end-of-sequence token or `max_tokens` ended it, and in each chunk before the choice's last."
},
"prompt_logprobs": {
"anyOf": [
{
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"$ref": "#/components/schemas/PromptLogprob"
}
},
{
"type": "null"
}
]
}
},
{
"type": "null"
}
],
"description": "Per prompt token, keyed by token id. The first is null."
}
},
"required": [
"index",
"text",
"logprobs",
"finish_reason"
],
"additionalProperties": false
},
"CompletionJob_3": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"request_id": {
"type": "string",
"description": "The id to send when you report a problem with this job."
},
"model": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"succeeded",
"failed",
"cancelled"
],
"description": "`queued` and `running` until the job ends. Then `succeeded` when its completion answered 200, `failed` when it answered an error, or `cancelled`."
},
"cancel_requested": {
"type": "boolean",
"description": "Whether the job was cancelled."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When the job started to run. `null` until it starts."
},
"finished_at": {
"type": [
"string",
"null"
],
"description": "When the job ended. `null` until it ends."
},
"expires_at": {
"type": [
"string",
"null"
],
"description": "When the result of the ended job expires: 30 days after it ended. `null` until it ends."
},
"result_expired": {
"type": "boolean"
},
"status_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id} for this job."
},
"result_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id}/result for this job."
},
"usage_pending": {
"type": "boolean",
"description": "Whether the job holds tokens of your budget: `true` while it runs."
}
},
"required": [
"id",
"request_id",
"model",
"status",
"cancel_requested",
"created_at",
"started_at",
"finished_at",
"expires_at",
"result_expired",
"status_url",
"result_url",
"usage_pending"
],
"additionalProperties": false,
"description": "The job is queued or running: the job, with `Retry-After: 1`."
},
"CompletionUsage": {
"type": "object",
"properties": {
"prompt_tokens": {
"type": "integer",
"minimum": 0,
"maximum": 2147483647
},
"completion_tokens": {
"type": "integer",
"minimum": 0,
"maximum": 2147483647
},
"total_tokens": {
"type": "integer",
"description": "`prompt_tokens` + `completion_tokens`."
}
},
"required": [
"prompt_tokens",
"completion_tokens",
"total_tokens"
],
"additionalProperties": false
},
"Completion_1": {
"type": "object",
"properties": {
"id": {
"type": "string"
},
"object": {
"type": "string",
"enum": [
"text_completion"
]
},
"created": {
"type": "integer"
},
"model": {
"type": "string"
},
"choices": {
"type": "array",
"items": {
"$ref": "#/components/schemas/CompletionChoice"
}
},
"usage": {
"$ref": "#/components/schemas/CompletionUsage"
},
"activations": {
"$ref": "#/components/schemas/ActivationsEncoded"
}
},
"required": [
"id",
"object",
"created",
"model",
"choices",
"usage"
],
"additionalProperties": false,
"description": "The completion succeeded."
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"PromptLogprob": {
"type": "object",
"properties": {
"logprob": {
"type": "number"
},
"rank": {
"type": [
"integer",
"null"
]
},
"decoded_token": {
"type": [
"string",
"null"
]
}
},
"required": [
"logprob",
"rank",
"decoded_token"
],
"additionalProperties": false
},
"Tensor": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The bytes of the array in C order, compressed with zstd, in base64."
},
"dtype": {
"type": "string",
"description": "The NumPy type of the bytes. A bfloat16 model's numbers come as `int16`."
},
"original_dtype": {
"type": "string",
"description": "The model's type, for example `torch.bfloat16`."
},
"shape": {
"type": "array",
"items": {
"type": "integer"
}
},
"compression": {
"type": "string",
"enum": [
"zstd"
]
}
},
"required": [
"data",
"dtype",
"original_dtype",
"shape",
"compression"
],
"additionalProperties": false,
"description": "Shape [layers, tokens, hidden size]: the output of each requested layer, for each prompt and generated token except the last generated one."
}
}
},
"paths": {
"/v1/completion-jobs/{job_id}/result": {
"get": {
"tags": [
"completionJobs"
],
"operationId": "completionJobs.getCompletionJobResult",
"parameters": [
{
"name": "job_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "The completion succeeded.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Completion_1"
}
}
}
},
"202": {
"description": "The job is queued or running: the job, with `Retry-After: 1`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJob_3"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The job was cancelled: `job_cancelled`, `x-should-retry: false`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"410": {
"description": "The result expired, 30 days after the job ended: `job_result_expired`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"502": {
"description": "Inference failed, refused the API's request, or did not answer.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "InferenceError",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"504": {
"description": "InferenceError",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Return the answer of an ended job, with the status that POST /v1/completions gave it: 200 and the Completion, or the error object with its own status, such as 429 `budget_exceeded` or 502. A cancelled job answers 409 `job_cancelled`, and a job that the server failed to run answers 500 `job_failed`. While the job is queued or running, 202 with the job and `Retry-After: 1`. 30 days after the job ended, 410 `job_result_expired`."
}
}
}
}
```
# Cancel Completion Job (/docs/reference/completionjobs/completionJobs.cancelCompletionJob)
```json
{
"components": {
"schemas": {
"CompletionJob_2": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"request_id": {
"type": "string",
"description": "The id to send when you report a problem with this job."
},
"model": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"succeeded",
"failed",
"cancelled"
],
"description": "`queued` and `running` until the job ends. Then `succeeded` when its completion answered 200, `failed` when it answered an error, or `cancelled`."
},
"cancel_requested": {
"type": "boolean",
"description": "Whether the job was cancelled."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When the job started to run. `null` until it starts."
},
"finished_at": {
"type": [
"string",
"null"
],
"description": "When the job ended. `null` until it ends."
},
"expires_at": {
"type": [
"string",
"null"
],
"description": "When the result of the ended job expires: 30 days after it ended. `null` until it ends."
},
"result_expired": {
"type": "boolean"
},
"status_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id} for this job."
},
"result_url": {
"type": "string",
"description": "The path of GET /v1/completion-jobs/{job_id}/result for this job."
},
"usage_pending": {
"type": "boolean",
"description": "Whether the job holds tokens of your budget: `true` while it runs."
}
},
"required": [
"id",
"request_id",
"model",
"status",
"cancel_requested",
"created_at",
"started_at",
"finished_at",
"expires_at",
"result_expired",
"status_url",
"result_url",
"usage_pending"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/completion-jobs/{job_id}/cancel": {
"post": {
"tags": [
"completionJobs"
],
"operationId": "completionJobs.cancelCompletionJob",
"parameters": [
{
"name": "job_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "CompletionJob",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/CompletionJob_2"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Cancel a queued or running job, and answer the job. A running job stops its Inference request. A job that ended stays as it is, and the answer is the same job."
}
}
}
}
```
# Create Harvest (/docs/reference/harvest/harvest.createHarvest)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"HarvestJob": {
"type": "object",
"properties": {
"job_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"run_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"status": {
"type": "string",
"enum": [
"pending",
"running",
"done",
"failed",
"cancelled"
],
"description": "`pending` while it waits for the Runner, then `running`, then `done`, `failed` or `cancelled`."
},
"model": {
"type": "string"
},
"n_prompts": {
"type": "integer",
"description": "How many prompts it has."
},
"reserved_tokens": {
"type": "integer",
"description": "The tokens it holds against your budget while it runs: each prompt's UTF-8 bytes, and 1 more for its start token."
},
"layers": {
"type": "array",
"items": {
"type": "integer"
},
"description": "The layers it reads, in ascending order."
},
"tokens": {
"type": [
"integer",
"null"
],
"description": "The tokens it was charged: what the engine read when it is done, the reserved tokens when it was cancelled after its GPU run started, else 0. `null` until it ends."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When its GPU run started. `null` until it starts."
},
"completed_at": {
"type": [
"string",
"null"
],
"description": "When it ended. `null` until it ends."
},
"result": {
"$ref": "#/components/schemas/HarvestResult"
},
"urls_expire_at": {
"type": "string",
"description": "With `result`: when its links stop working, 7 days after this answer."
},
"error": {
"type": "string",
"description": "Why it failed. Only on a failed job."
}
},
"required": [
"job_id",
"run_id",
"status",
"model",
"n_prompts",
"reserved_tokens",
"layers",
"tokens",
"created_at",
"started_at",
"completed_at"
],
"additionalProperties": false
},
"HarvestRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models whose harvest is on (the API docs list them)."
},
"prompts": {
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"minLength": 1
},
"minItems": 1,
"maxItems": 4096,
"description": "From 1 to 4,096 texts, and at most 2,000,000 tokens counted as characters divided by 4. Token ids are not taken. A prompt longer than the model's context keeps its first context length - 1 tokens."
},
"layers": {
"anyOf": [
{
"type": "string",
"enum": [
"all"
]
},
{
"type": "array",
"items": {
"type": "integer",
"minimum": 0,
"maximum": 512
},
"minItems": 1,
"maxItems": 256
}
],
"description": "The layers to read: a list of up to 256 (in any order; each is read once), or `\"all\"`. Left out: the quartiles, capabilities.n_layers / 4, / 2 and × 3 / 4 (8, 16 and 24 on llama-8b). Layer k is the residual stream after decoder block k; the last layer is before the final norm."
},
"shard_size": {
"type": "integer",
"minimum": 1,
"maximum": 4096,
"description": "Prompts in each shard file."
},
"batch_size": {
"type": [
"integer",
"null"
],
"minimum": 1,
"maximum": 64,
"description": "Kept in the manifest, as Production; the engine sizes its own batches."
},
"add_special_tokens": {
"type": "boolean",
"description": "Add the model's start token to each prompt. Set false for text that already starts with one."
},
"project_onto": {
"anyOf": [
{
"$ref": "#/components/schemas/Projection"
},
{
"type": "null"
}
],
"description": "Write the residual stream projected onto these directions, as float32, instead of the stream itself. The engine scales each direction to length 1."
}
},
"required": [
"model",
"prompts"
],
"additionalProperties": false
},
"HarvestResult": {
"type": "object",
"properties": {
"n_shards": {
"type": "integer"
},
"layer_indices": {
"type": "array",
"items": {
"type": "integer"
}
},
"manifest_url": {
"type": "string"
},
"shard_urls": {
"type": "array",
"items": {
"type": "string"
},
"description": "One link for each shard, in order."
},
"stats_url": {
"type": "string"
},
"contents": {
"type": "string",
"enum": [
"activations",
"projections"
]
},
"tokens": {
"type": "integer",
"description": "The token positions the files hold, after the cut to the context, start tokens included."
},
"bytes_written": {
"type": "integer"
},
"timings": {
"type": "object",
"properties": {
"load_s": {
"type": "number"
},
"forward_s": {
"type": "number"
},
"save_s": {
"type": "number"
},
"upload_s": {
"type": "number"
}
},
"required": [
"load_s",
"forward_s",
"save_s",
"upload_s"
],
"additionalProperties": false,
"description": "Seconds of its GPU run: loading the model (0 when a container had it), reading the stream, writing the files and uploading them."
}
},
"required": [
"n_shards",
"layer_indices",
"manifest_url",
"shard_urls",
"stats_url",
"contents",
"tokens",
"bytes_written",
"timings"
],
"additionalProperties": false,
"description": "The files of a done harvest job. Each link works 7 days and needs no API key. Each answer makes new links; the files are kept."
},
"Projection": {
"type": "object",
"properties": {
"data": {
"type": "string",
"description": "The directions' values, raw little-endian, in base64. Spaces and line breaks are left out. Each value is finite, and no direction is all zeros."
},
"dtype": {
"type": "string",
"enum": [
"float32",
"float16",
"bfloat16"
],
"description": "The type of each value."
},
"shape": {
"type": "array",
"prefixItems": [
{
"type": "integer",
"minimum": 1,
"maximum": 64,
"description": "The directions: from 1 to 64."
},
{
"type": "integer",
"minimum": 1,
"description": "The model's hidden size, as GET /v1/models gives it."
}
],
"maxItems": 2,
"minItems": 2,
"description": "The directions and the model's hidden size."
},
"compression": {
"type": "string",
"enum": [
"none"
],
"description": "Only `none`."
}
},
"required": [
"data",
"dtype",
"shape"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/harvest": {
"post": {
"tags": [
"harvest"
],
"operationId": "harvest.createHarvest",
"parameters": [
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"202": {
"description": "HarvestJob",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HarvestJob"
}
}
}
},
"400": {
"description": "The model has no harvest: `harvest_unsupported`. A layer that the model does not have, or directions of another hidden size, answers 400 with its `param`. | The API serves no such model: `model_not_found`. Or admission refuses a value, such as a Layer that the model does not have: `invalid_request`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds. Or a harvest cap is full: this API key's harvest jobs this month (200 by default, `harvest_quota_exceeded`, `x-should-retry: false`), your 3 harvest jobs on small models or 1 on a big model queued or running (`harvest_concurrency_exceeded`, `Retry-After: 30`), or 2 harvest jobs on big models for all Users (`harvest_capacity_exceeded`, `Retry-After: 30`). Or the token budget cannot hold the prompts' tokens: `budget_exceeded`. Or the Spend stop is on: `spend_cap`, `Retry-After: 3600`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Queue a harvest job: the residual stream of a Base model at the chosen layers, for each token of up to 4,096 prompts, written as files that you download by the links in its `result`. It holds the prompts' tokens while it runs and is charged the tokens the engine read. Only this route takes a body of up to 50 MiB. Answers the job, `pending`.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HarvestRequest"
}
}
},
"required": true
}
}
}
}
}
```
# Get Harvest (/docs/reference/harvest/harvest.getHarvest)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"HarvestJob": {
"type": "object",
"properties": {
"job_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"run_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"status": {
"type": "string",
"enum": [
"pending",
"running",
"done",
"failed",
"cancelled"
],
"description": "`pending` while it waits for the Runner, then `running`, then `done`, `failed` or `cancelled`."
},
"model": {
"type": "string"
},
"n_prompts": {
"type": "integer",
"description": "How many prompts it has."
},
"reserved_tokens": {
"type": "integer",
"description": "The tokens it holds against your budget while it runs: each prompt's UTF-8 bytes, and 1 more for its start token."
},
"layers": {
"type": "array",
"items": {
"type": "integer"
},
"description": "The layers it reads, in ascending order."
},
"tokens": {
"type": [
"integer",
"null"
],
"description": "The tokens it was charged: what the engine read when it is done, the reserved tokens when it was cancelled after its GPU run started, else 0. `null` until it ends."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When its GPU run started. `null` until it starts."
},
"completed_at": {
"type": [
"string",
"null"
],
"description": "When it ended. `null` until it ends."
},
"result": {
"$ref": "#/components/schemas/HarvestResult"
},
"urls_expire_at": {
"type": "string",
"description": "With `result`: when its links stop working, 7 days after this answer."
},
"error": {
"type": "string",
"description": "Why it failed. Only on a failed job."
}
},
"required": [
"job_id",
"run_id",
"status",
"model",
"n_prompts",
"reserved_tokens",
"layers",
"tokens",
"created_at",
"started_at",
"completed_at"
],
"additionalProperties": false
},
"HarvestResult": {
"type": "object",
"properties": {
"n_shards": {
"type": "integer"
},
"layer_indices": {
"type": "array",
"items": {
"type": "integer"
}
},
"manifest_url": {
"type": "string"
},
"shard_urls": {
"type": "array",
"items": {
"type": "string"
},
"description": "One link for each shard, in order."
},
"stats_url": {
"type": "string"
},
"contents": {
"type": "string",
"enum": [
"activations",
"projections"
]
},
"tokens": {
"type": "integer",
"description": "The token positions the files hold, after the cut to the context, start tokens included."
},
"bytes_written": {
"type": "integer"
},
"timings": {
"type": "object",
"properties": {
"load_s": {
"type": "number"
},
"forward_s": {
"type": "number"
},
"save_s": {
"type": "number"
},
"upload_s": {
"type": "number"
}
},
"required": [
"load_s",
"forward_s",
"save_s",
"upload_s"
],
"additionalProperties": false,
"description": "Seconds of its GPU run: loading the model (0 when a container had it), reading the stream, writing the files and uploading them."
}
},
"required": [
"n_shards",
"layer_indices",
"manifest_url",
"shard_urls",
"stats_url",
"contents",
"tokens",
"bytes_written",
"timings"
],
"additionalProperties": false,
"description": "The files of a done harvest job. Each link works 7 days and needs no API key. Each answer makes new links; the files are kept."
}
}
},
"paths": {
"/v1/harvest/{job_id}": {
"get": {
"tags": [
"harvest"
],
"operationId": "harvest.getHarvest",
"parameters": [
{
"name": "job_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "wait",
"in": "query",
"schema": {
"type": "string"
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "HarvestJob",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HarvestJob"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Return the harvest job and its status. A `done` job has its `result`: a link for its manifest, its stats and each shard, which works 7 days and needs no API key; each answer makes new links. With `wait`, wait up to that many seconds for its status to change. An API key may have 4 such waits at once; past that, the answer comes at once with `X-Acs-Longpoll: declined` and `Retry-After: 3`."
}
}
}
}
```
# Cancel Harvest (/docs/reference/harvest/harvest.cancelHarvest)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"HarvestJob": {
"type": "object",
"properties": {
"job_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"run_id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"status": {
"type": "string",
"enum": [
"pending",
"running",
"done",
"failed",
"cancelled"
],
"description": "`pending` while it waits for the Runner, then `running`, then `done`, `failed` or `cancelled`."
},
"model": {
"type": "string"
},
"n_prompts": {
"type": "integer",
"description": "How many prompts it has."
},
"reserved_tokens": {
"type": "integer",
"description": "The tokens it holds against your budget while it runs: each prompt's UTF-8 bytes, and 1 more for its start token."
},
"layers": {
"type": "array",
"items": {
"type": "integer"
},
"description": "The layers it reads, in ascending order."
},
"tokens": {
"type": [
"integer",
"null"
],
"description": "The tokens it was charged: what the engine read when it is done, the reserved tokens when it was cancelled after its GPU run started, else 0. `null` until it ends."
},
"created_at": {
"type": "string",
"description": "An ISO 8601 time in UTC, such as `2026-11-03T09:30:00.123+00:00`."
},
"started_at": {
"type": [
"string",
"null"
],
"description": "When its GPU run started. `null` until it starts."
},
"completed_at": {
"type": [
"string",
"null"
],
"description": "When it ended. `null` until it ends."
},
"result": {
"$ref": "#/components/schemas/HarvestResult"
},
"urls_expire_at": {
"type": "string",
"description": "With `result`: when its links stop working, 7 days after this answer."
},
"error": {
"type": "string",
"description": "Why it failed. Only on a failed job."
}
},
"required": [
"job_id",
"run_id",
"status",
"model",
"n_prompts",
"reserved_tokens",
"layers",
"tokens",
"created_at",
"started_at",
"completed_at"
],
"additionalProperties": false
},
"HarvestResult": {
"type": "object",
"properties": {
"n_shards": {
"type": "integer"
},
"layer_indices": {
"type": "array",
"items": {
"type": "integer"
}
},
"manifest_url": {
"type": "string"
},
"shard_urls": {
"type": "array",
"items": {
"type": "string"
},
"description": "One link for each shard, in order."
},
"stats_url": {
"type": "string"
},
"contents": {
"type": "string",
"enum": [
"activations",
"projections"
]
},
"tokens": {
"type": "integer",
"description": "The token positions the files hold, after the cut to the context, start tokens included."
},
"bytes_written": {
"type": "integer"
},
"timings": {
"type": "object",
"properties": {
"load_s": {
"type": "number"
},
"forward_s": {
"type": "number"
},
"save_s": {
"type": "number"
},
"upload_s": {
"type": "number"
}
},
"required": [
"load_s",
"forward_s",
"save_s",
"upload_s"
],
"additionalProperties": false,
"description": "Seconds of its GPU run: loading the model (0 when a container had it), reading the stream, writing the files and uploading them."
}
},
"required": [
"n_shards",
"layer_indices",
"manifest_url",
"shard_urls",
"stats_url",
"contents",
"tokens",
"bytes_written",
"timings"
],
"additionalProperties": false,
"description": "The files of a done harvest job. Each link works 7 days and needs no API key. Each answer makes new links; the files are kept."
}
}
},
"paths": {
"/v1/harvest/{job_id}": {
"delete": {
"tags": [
"harvest"
],
"operationId": "harvest.cancelHarvest",
"parameters": [
{
"name": "job_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "HarvestJob",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/HarvestJob"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The job is done or failed: only a pending or running job can be cancelled. `harvest_not_cancellable`, `x-should-retry: false`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Cancel a pending or running harvest job, and stop its GPU run, also one that is still starting. A job that is running and whose GPU run started is charged the tokens it holds; any other cancel costs nothing. Answers the cancelled job, also for a second cancel."
}
}
}
}
```
# List Looms (/docs/reference/looms/looms.listLooms)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"LoomList": {
"type": "object",
"properties": {
"object": {
"type": "string",
"enum": [
"list"
]
},
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomSummary"
},
"description": "At most 100 Looms: the pinned ones first, newest pin first, then the others by their last change, newest first. A rename, a Branch and a deleted Node change a Loom; a pin does not."
}
},
"required": [
"object",
"data"
],
"additionalProperties": false
},
"LoomSummary": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"updated_at": {
"type": "integer",
"description": "When the Loom last changed, in Unix seconds: a rename, a Branch or a deleted Node."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"prompt": {
"type": "string",
"description": "The text of the root Node."
}
},
"required": [
"id",
"object",
"created_at",
"updated_at",
"title",
"pinned",
"prompt"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms": {
"get": {
"tags": [
"looms"
],
"operationId": "looms.listLooms",
"parameters": [],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "LoomList",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoomList"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "List your Looms as Production does: the pinned ones first, then the others by their last change, newest first, at most 100, each with its name and prompt. Read a Loom for its Nodes."
}
}
}
}
```
# Create Loom (/docs/reference/looms/looms.createLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"LoomRequest": {
"type": "object",
"properties": {
"prompt": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The text of the root Node."
}
},
"required": [
"prompt"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.createLoom",
"parameters": [
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Make a Loom. Its root Node is the prompt. A retry with the same `Idempotency-Key` answers the Loom as it is now, or 404 if it was deleted since.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoomRequest"
}
}
},
"required": true
}
}
}
}
}
```
# Import Loom (/docs/reference/looms/looms.importLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"LoomFile": {
"type": "object",
"properties": {
"version": {
"description": "The file's version, 2. Production's export writes it; the API does not keep it."
},
"node_order": {
"description": "How the Nodes are ordered. Production's export writes it; the API does not keep it."
},
"loom": {
"type": "object",
"properties": {
"title": {
"type": [
"string",
"null"
],
"description": "The Loom's name, at most 120 characters; none is `Imported loom`."
},
"model": {
"description": "The model of its last Branch. Production's export writes it; the API does not keep it."
},
"last_generation_config": {
"description": "The settings of its last Branch. Production's export writes it; the API does not keep it."
}
},
"additionalProperties": false,
"description": "The Loom's title and last settings."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileNode"
},
"description": "At most 2000 Nodes, each with a unique id. A Node whose parent is not in the file goes under the file's root when it has one root, else it is a root; a cycle is broken at a Node that becomes a root.",
"maxItems": 2000
}
},
"required": [
"nodes"
],
"additionalProperties": false
},
"LoomFileNode": {
"type": "object",
"properties": {
"id": {
"type": "string",
"minLength": 1,
"description": "The Node's id in the file. An import gives each Node a new one."
},
"parent_id": {
"type": [
"string",
"null"
],
"description": "The id of its parent in the file; null or none for a root."
},
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's own text; none is empty."
},
"status": {
"type": [
"string",
"null"
],
"enum": [
"completed",
"cancelled",
"failed"
],
"description": "How the Node ended; none is `completed`."
},
"position": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 2147483647,
"description": "Its place among its siblings, from 0; none is its place in the file."
},
"model": {
"type": [
"string",
"null"
],
"description": "The model that wrote it, when no `generation_config` says."
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "The seed it ran with, when no `generation_config` says."
},
"generation_config": {
"anyOf": [
{
"type": "object"
},
{
"type": "null"
}
],
"description": "The request it ran with: the API keeps the options that a Branch takes, and Production's marks `text_modified`, `context_modified`, `slice_offset` and `slice_end`."
},
"logprobs": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileToken"
}
},
{
"type": "null"
}
],
"description": "Each token of its text, with its log probability."
},
"error": {
"description": "Why a failed Node failed: the error object, as the API's export writes it."
},
"generation_id": {
"description": "The run that wrote it. Production's export writes it; the API does not keep it."
},
"choice_index": {
"description": "Its place in that run. Production's export writes it; the API does not keep it."
},
"created_at": {
"description": "When it was made. Production's export writes it; the API does not keep it."
},
"finish_reason": {
"description": "Why its run ended. Production's export writes it; the API does not keep it."
},
"stop_reason": {
"description": "The stop that ended it. Production's export writes it; the API does not keep it."
},
"n_tokens": {
"description": "How many tokens it has. Production's export writes it; the API does not keep it."
},
"usage_source": {
"description": "Where its usage came from. Production's export writes it; the API does not keep it."
},
"api_key_id": {
"description": "The API key it ran with. Production's export writes it; the API does not keep it."
}
},
"required": [
"id"
],
"additionalProperties": false
},
"LoomFileToken": {
"type": "object",
"properties": {
"token": {
"type": "string",
"description": "The token's text."
},
"logprob": {
"type": [
"number",
"null"
],
"description": "The token's log probability."
},
"top": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileTopToken"
}
},
{
"type": "null"
}
],
"description": "The likeliest tokens at this place."
}
},
"required": [
"token",
"logprob"
],
"additionalProperties": false
},
"LoomFileTopToken": {
"type": "object",
"properties": {
"token": {
"type": "string",
"description": "A likely token."
},
"logprob": {
"type": "number",
"description": "Its log probability."
}
},
"required": [
"token",
"logprob"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/import": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.importLoom",
"parameters": [
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Make a Loom from a file that GET /v1/looms/{loom_id}/export or Production's Loom export wrote, and answer it. Its Nodes get new ids; a Node keeps its text, its status (none is `completed`), its log probabilities and the Branch options it ran with.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoomFile"
}
}
},
"required": true
}
}
}
}
}
```
# Export Loom (/docs/reference/looms/looms.exportLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"LoomFile": {
"type": "object",
"properties": {
"version": {
"description": "The file's version, 2. Production's export writes it; the API does not keep it."
},
"node_order": {
"description": "How the Nodes are ordered. Production's export writes it; the API does not keep it."
},
"loom": {
"type": "object",
"properties": {
"title": {
"type": [
"string",
"null"
],
"description": "The Loom's name, at most 120 characters; none is `Imported loom`."
},
"model": {
"description": "The model of its last Branch. Production's export writes it; the API does not keep it."
},
"last_generation_config": {
"description": "The settings of its last Branch. Production's export writes it; the API does not keep it."
}
},
"additionalProperties": false,
"description": "The Loom's title and last settings."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileNode"
},
"description": "At most 2000 Nodes, each with a unique id. A Node whose parent is not in the file goes under the file's root when it has one root, else it is a root; a cycle is broken at a Node that becomes a root.",
"maxItems": 2000
}
},
"required": [
"nodes"
],
"additionalProperties": false
},
"LoomFileNode": {
"type": "object",
"properties": {
"id": {
"type": "string",
"minLength": 1,
"description": "The Node's id in the file. An import gives each Node a new one."
},
"parent_id": {
"type": [
"string",
"null"
],
"description": "The id of its parent in the file; null or none for a root."
},
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's own text; none is empty."
},
"status": {
"type": [
"string",
"null"
],
"enum": [
"completed",
"cancelled",
"failed"
],
"description": "How the Node ended; none is `completed`."
},
"position": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 2147483647,
"description": "Its place among its siblings, from 0; none is its place in the file."
},
"model": {
"type": [
"string",
"null"
],
"description": "The model that wrote it, when no `generation_config` says."
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "The seed it ran with, when no `generation_config` says."
},
"generation_config": {
"anyOf": [
{
"type": "object"
},
{
"type": "null"
}
],
"description": "The request it ran with: the API keeps the options that a Branch takes, and Production's marks `text_modified`, `context_modified`, `slice_offset` and `slice_end`."
},
"logprobs": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileToken"
}
},
{
"type": "null"
}
],
"description": "Each token of its text, with its log probability."
},
"error": {
"description": "Why a failed Node failed: the error object, as the API's export writes it."
},
"generation_id": {
"description": "The run that wrote it. Production's export writes it; the API does not keep it."
},
"choice_index": {
"description": "Its place in that run. Production's export writes it; the API does not keep it."
},
"created_at": {
"description": "When it was made. Production's export writes it; the API does not keep it."
},
"finish_reason": {
"description": "Why its run ended. Production's export writes it; the API does not keep it."
},
"stop_reason": {
"description": "The stop that ended it. Production's export writes it; the API does not keep it."
},
"n_tokens": {
"description": "How many tokens it has. Production's export writes it; the API does not keep it."
},
"usage_source": {
"description": "Where its usage came from. Production's export writes it; the API does not keep it."
},
"api_key_id": {
"description": "The API key it ran with. Production's export writes it; the API does not keep it."
}
},
"required": [
"id"
],
"additionalProperties": false
},
"LoomFileToken": {
"type": "object",
"properties": {
"token": {
"type": "string",
"description": "The token's text."
},
"logprob": {
"type": [
"number",
"null"
],
"description": "The token's log probability."
},
"top": {
"anyOf": [
{
"type": "array",
"items": {
"$ref": "#/components/schemas/LoomFileTopToken"
}
},
{
"type": "null"
}
],
"description": "The likeliest tokens at this place."
}
},
"required": [
"token",
"logprob"
],
"additionalProperties": false
},
"LoomFileTopToken": {
"type": "object",
"properties": {
"token": {
"type": "string",
"description": "A likely token."
},
"logprob": {
"type": "number",
"description": "Its log probability."
}
},
"required": [
"token",
"logprob"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/export": {
"get": {
"tags": [
"looms"
],
"operationId": "looms.exportLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "LoomFile",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoomFile"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Download the Loom as a file in Production's export shape (version 2): its title and last Branch settings, and each Node with its settings and marks, parents first. POST /v1/looms/import makes a Loom from it."
}
}
}
}
```
# Get Loom (/docs/reference/looms/looms.getLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}": {
"get": {
"tags": [
"looms"
],
"operationId": "looms.getLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Return the Loom: each Node with its status and text. Read it again to follow a Branch."
}
}
}
}
```
# Update Loom (/docs/reference/looms/looms.updateLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"LoomChange": {
"type": "object",
"properties": {
"title": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The new name. The API trims it and keeps its first 120 characters; an empty one leaves the Loom untitled, so its next Branch names it."
},
"pinned": {
"type": "boolean",
"description": "true pins the Loom to the top of the list; false unpins it."
}
},
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.updateLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Rename or pin the Loom, and answer it. A rename moves it to the top of its unpinned Looms; a pin puts it above them.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/LoomChange"
}
}
},
"required": true
}
}
}
}
}
```
# Delete Loom (/docs/reference/looms/looms.deleteLoom)
```json
{
"components": {
"schemas": {
"DeletedLoom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"deleted": {
"type": "boolean",
"enum": [
true
]
}
},
"required": [
"id",
"object",
"deleted"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}": {
"delete": {
"tags": [
"looms"
],
"operationId": "looms.deleteLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "DeletedLoom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/DeletedLoom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Delete the Loom and all its Nodes. You cannot undo it. While a Node of the Loom is `queued` or `running`, 409 and the error object."
}
}
}
}
```
# Stream Loom (/docs/reference/looms/looms.streamLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/stream": {
"get": {
"tags": [
"looms"
],
"operationId": "looms.streamLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"text/event-stream": {
"schema": {
"anyOf": [
{
"type": "object",
"properties": {
"data": {
"type": "string",
"contentMediaType": "application/json"
},
"event": {
"type": "string",
"enum": [
"message"
]
}
},
"required": [
"data",
"event"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"data": {
"type": "string",
"enum": [
"[DONE]"
]
},
"event": {
"type": "string",
"enum": [
"message"
]
}
},
"required": [
"data",
"event"
],
"additionalProperties": false
}
]
},
"x-effect-stream": {
"encoding": "sse",
"causeSchema": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Fail"
]
},
"error": {
"not": {}
}
},
"required": [
"_tag",
"error"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Die"
]
},
"defect": {}
},
"required": [
"_tag",
"defect"
],
"additionalProperties": false
},
{
"type": "object",
"properties": {
"_tag": {
"type": "string",
"enum": [
"Interrupt"
]
},
"fiberId": {
"type": [
"number",
"null"
]
}
},
"required": [
"_tag",
"fiberId"
],
"additionalProperties": false
}
]
}
},
"errorSchema": {
"not": {}
},
"failureEvent": "effect/http-api/stream/failure"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Follow a Loom as Server-Sent Events: each Node of the Loom, then each Node again when its status or text changes, then `[DONE]` when no Node is `queued` or `running`. A running Node's text is its whole text so far, at most every 250 ms."
}
}
}
}
```
# Cancel Loom (/docs/reference/looms/looms.cancelLoom)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/cancel": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.cancelLoom",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Stop each Node of the Loom that is `queued` or `running`, and answer the Loom. Each becomes `cancelled`: a running Node keeps the text it grew, and its tokens so far are charged; a queued Node never runs and holds nothing."
}
}
}
}
```
# Create Branch (/docs/reference/looms/looms.createBranch)
```json
{
"components": {
"schemas": {
"Branch": {
"type": "object",
"properties": {
"object": {
"type": "string",
"enum": [
"list"
]
},
"data": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
}
}
},
"required": [
"object",
"data"
],
"additionalProperties": false
},
"BranchRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 1000,
"description": "Generate at most this many tokens for each Node; at most 1000."
},
"temperature": {
"type": "number",
"minimum": 0,
"maximum": 2,
"description": "Sample less likely tokens more often; 0 takes the most likely token each time. From 0 to 2, as vLLM takes it."
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number",
"exclusiveMinimum": 0,
"maximum": 1,
"description": "Sample from the most likely tokens up to this probability; more than 0 and at most 1, as OpenAI and vLLM take it."
},
"top_k": {
"type": "integer",
"minimum": -1,
"description": "vLLM: sample from this many most likely tokens; 0 or -1 for all."
},
"min_p": {
"type": "number",
"minimum": 0,
"maximum": 1,
"description": "vLLM: leave out tokens less likely than this fraction of the most likely; from 0 to 1."
},
"presence_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token that is already in the text less likely; from -2 to 2."
},
"frequency_penalty": {
"type": "number",
"minimum": -2,
"maximum": 2,
"description": "Make a token less likely each time it is already in the text; from -2 to 2."
},
"repetition_penalty": {
"type": "number",
"exclusiveMinimum": 0,
"description": "vLLM: divide the scores of repeated tokens by this, more than 0; 1 for none."
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
},
"n": {
"type": "integer",
"minimum": 1,
"maximum": 8,
"description": "How many child Nodes to make, from 1 to 8."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/nodes/{node_id}/branch": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.createBranch",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "node_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Branch",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Branch"
}
}
}
},
"400": {
"description": "The API serves no such model: `model_not_found`. Or admission refuses a value, such as a Layer that the model does not have: `invalid_request`. | The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds. Or the token budget cannot hold the request: `budget_exceeded`. Or the Spend stop is on, so the API starts no new GPU work: `spend_cap`, `Retry-After` 3600, `x-should-retry: false`.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Grow a Branch under a completed Node: `n` child Nodes, each one Completion of its text. The text of a Node is the text on the path from the root to it. The server writes the new Nodes as their tokens arrive, also when no client is connected. Answers the new Nodes, `queued`. For a Node that is not completed, 409 and the error object. For a model that the API does not serve, 400 `model_not_found`, and no Node is made. If the token budget cannot hold one Node's Completion, 429 `budget_exceeded` and no Node is made.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/BranchRequest"
}
}
},
"required": true
}
}
}
}
}
```
# Set Root (/docs/reference/looms/looms.setRoot)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"NodeText": {
"type": "object",
"properties": {
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's text. A CRLF or a lone CR is stored as LF, as Production does."
}
},
"required": [
"text"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/root": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.setRoot",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`). | The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Replace the Loom's tree with one root, the text you write, and answer the Loom. While a Node of the Loom is `queued` or `running`, 409.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/NodeText"
}
}
},
"required": true
}
}
}
}
}
```
# Edit Node (/docs/reference/looms/looms.editNode)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"NodeText": {
"type": "object",
"properties": {
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's text. A CRLF or a lone CR is stored as LF, as Production does."
}
},
"required": [
"text"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/nodes/{node_id}": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.editNode",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "node_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`). | The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Rewrite the Node's text, and answer the Loom. A new text drops the Node's log probabilities, as they were of the old one; the same text changes nothing. For a failed Node, or while a Node under it is `queued` or `running`, 409.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/NodeText"
}
}
},
"required": true
}
}
}
}
}
```
# Delete Node (/docs/reference/looms/looms.deleteNode)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/nodes/{node_id}": {
"delete": {
"tags": [
"looms"
],
"operationId": "looms.deleteNode",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "node_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Delete the Node and every Node under it, and answer the Loom without them. You cannot undo it. For the root, or while a Node to delete is `queued` or `running`, 409 and the error object."
}
}
}
}
```
# Split Node (/docs/reference/looms/looms.splitNode)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"SplitRequest": {
"type": "object",
"properties": {
"offset": {
"type": "integer",
"minimum": 0,
"description": "Where to cut, in characters (Unicode code points) from the start of the text. Past the end, the cut is at the end."
},
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's edited text, cut at `offset`. Without it, the Node's own text is cut, and each half keeps its log probabilities."
}
},
"required": [
"offset"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/nodes/{node_id}/split": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.splitNode",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "node_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`). | The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Cut the Node at `offset`, and answer the Loom. The Node keeps the text before the cut; a new child gets the text after it, and the Node's children move under the child, so every path keeps its text. A cut of the Node's own text gives each half the log probabilities of its whole tokens; the child's `request` has no seed, as a part of a sample cannot be sampled again. For a failed Node, a Loom of 2,000 Nodes, or while a Node under it is `queued` or `running`, 409.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/SplitRequest"
}
}
},
"required": true
}
}
}
}
}
```
# Add Sibling (/docs/reference/looms/looms.addSibling)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Logprobs": {
"type": "object",
"properties": {
"text_offset": {
"type": "array",
"items": {
"type": "integer"
}
},
"token_logprobs": {
"type": "array",
"items": {
"type": [
"number",
"null"
]
}
},
"tokens": {
"type": "array",
"items": {
"type": "string"
}
},
"top_logprobs": {
"type": "array",
"items": {
"anyOf": [
{
"type": "object",
"additionalProperties": {
"type": "number"
}
},
{
"type": "null"
}
]
}
}
},
"required": [
"text_offset",
"token_logprobs",
"tokens",
"top_logprobs"
],
"additionalProperties": false
},
"Loom": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom"
]
},
"created_at": {
"type": "integer",
"description": "When the Loom was made, in Unix seconds."
},
"title": {
"type": "string",
"description": "The Loom's name. Empty until its first Branch names it after the first line of its root, or a rename names it."
},
"pinned": {
"type": "boolean",
"description": "A pinned Loom comes first in the list."
},
"nodes": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Node"
},
"description": "Depth first from each root: each Node comes after its parent and before its next sibling, and siblings come in the order they were made."
}
},
"required": [
"id",
"object",
"created_at",
"title",
"pinned",
"nodes"
],
"additionalProperties": false
},
"Node": {
"type": "object",
"properties": {
"id": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"object": {
"type": "string",
"enum": [
"loom.node"
]
},
"parent_id": {
"type": [
"string",
"null"
],
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid",
"description": "null for the root, whose text is the prompt."
},
"status": {
"type": "string",
"enum": [
"queued",
"running",
"completed",
"failed",
"cancelled"
],
"description": "The root is `completed`. A `cancelled` Node was stopped while it grew, and keeps the text it grew."
},
"text": {
"type": "string",
"description": "The text so far. It grows while the Node is `running`."
},
"logprobs": {
"anyOf": [
{
"$ref": "#/components/schemas/Logprobs"
},
{
"type": "null"
}
],
"description": "With `logprobs` in the Branch: the log probabilities of the Node's tokens, when the Node is `completed`."
},
"error": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorObject"
},
{
"type": "null"
}
],
"description": "Why the Completion failed, when `failed`."
},
"request": {
"anyOf": [
{
"$ref": "#/components/schemas/NodeRequest"
},
{
"type": "null"
}
],
"description": "The Branch options that the Node ran with, its own seed included. A Branch with them under its parent samples it again, unless one of them is outside a range that narrowed after the Node ran. null for the root, and for a Node whose options do not read as Branch options."
}
},
"required": [
"id",
"object",
"parent_id",
"status",
"text",
"logprobs",
"error",
"request"
],
"additionalProperties": false
},
"NodeRequest": {
"type": "object",
"properties": {
"model": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Base model: an `id` from GET /v1/models, such as `llama-8b`."
},
"max_tokens": {
"type": "integer",
"maximum": 32000,
"description": "Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × `n` × `max_tokens`."
},
"temperature": {
"type": "number"
},
"logprobs": {
"type": [
"integer",
"null"
],
"minimum": 0,
"maximum": 100,
"description": "Return the log probabilities of this many top tokens; at most 100."
},
"top_p": {
"type": "number"
},
"top_k": {
"type": "integer"
},
"min_p": {
"type": "number"
},
"presence_penalty": {
"type": "number"
},
"frequency_penalty": {
"type": "number"
},
"repetition_penalty": {
"type": "number"
},
"seed": {
"type": [
"integer",
"null"
],
"minimum": 0,
"description": "From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests."
},
"stop": {
"anyOf": [
{
"type": "string",
"pattern": "^[^\\u0000]*$"
},
{
"type": "array",
"items": {
"type": "string",
"pattern": "^[^\\u0000]*$"
},
"maxItems": 4
},
{
"type": "null"
}
],
"description": "Stop before any of these texts; at most 4."
}
},
"required": [
"model"
],
"additionalProperties": false
},
"NodeText": {
"type": "object",
"properties": {
"text": {
"type": "string",
"pattern": "^[^\\u0000]*$",
"description": "The Node's text. A CRLF or a lone CR is stored as LF, as Production does."
}
},
"required": [
"text"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/looms/{loom_id}/nodes/{node_id}/sibling": {
"post": {
"tags": [
"looms"
],
"operationId": "looms.addSibling",
"parameters": [
{
"name": "loom_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "node_id",
"in": "path",
"schema": {
"type": "string",
"pattern": "^([0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[1-8][0-9a-fA-F]{3}-[89abAB][0-9a-fA-F]{3}-[0-9a-fA-F]{12}|00000000-0000-0000-0000-000000000000|[fF]{8}-[fF]{4}-[fF]{4}-[fF]{4}-[fF]{12})$",
"format": "uuid"
},
"required": true
},
{
"name": "idempotency-key",
"in": "header",
"schema": {
"type": "string",
"minLength": 1,
"maxLength": 255,
"description": "A key of your choice, 1 to 255 characters, such as a UUID. A retry with the same key and the same request does the work once: it gets the first answer, and is never charged twice. Kept 24 hours, one set for each User across all their API keys."
},
"required": false
}
],
"security": [
{
"bearer": []
}
],
"responses": {
"200": {
"description": "Loom",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Loom"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"401": {
"description": "No API key, or a wrong or revoked one.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"404": {
"description": "NotFound",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"409": {
"description": "The status of the Node does not allow the request (`x-should-retry: false`). Or: The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`). | The `Idempotency-Key` was used already: with another request (`idempotency_conflict`), or a request with it still runs (`Retry-After` 1), or it ran and its answer cannot be sent again, as a stream (`x-should-retry: false`).",
"content": {
"application/json": {
"schema": {
"anyOf": [
{
"$ref": "#/components/schemas/ErrorResponse"
},
{
"$ref": "#/components/schemas/ErrorResponse"
}
]
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"429": {
"description": "Too many requests with this API key in a minute, or from one address with a wrong API key, or while lookups of its other keys hold every token that is left: `rate_limited`. Try again after `Retry-After` seconds.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Write a Node beside this one, under the same parent, and answer the Loom. For a root, a Loom of 2,000 Nodes, or while a Node under the parent is `queued` or `running`, 409.",
"requestBody": {
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/NodeText"
}
}
},
"required": true
}
}
}
}
}
```
# Health (/docs/reference/health/health.health)
```json
{
"components": {
"schemas": {
"Backend": {
"type": "object",
"properties": {
"model_id": {
"type": "string"
},
"status": {
"type": "string",
"enum": [
"live"
]
},
"gpu_shape": {
"type": "string"
},
"breaker_state": {
"type": "string",
"enum": [
"closed",
"open",
"half_open"
]
},
"breaker_consecutive_failures": {
"type": "integer"
},
"breaker_open_seconds": {
"type": [
"integer",
"null"
]
},
"in_flight": {
"type": "integer"
}
},
"required": [
"model_id",
"status",
"gpu_shape",
"breaker_state",
"breaker_consecutive_failures",
"breaker_open_seconds",
"in_flight"
],
"additionalProperties": false
},
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
},
"Health": {
"type": "object",
"properties": {
"status": {
"type": "string",
"enum": [
"ok",
"degraded",
"down"
]
},
"wrapper": {
"type": "object",
"properties": {
"uptime_seconds": {
"type": "integer"
},
"database": {
"type": "object",
"properties": {
"ok": {
"type": "boolean"
},
"latency_ms": {
"type": [
"integer",
"null"
]
}
},
"required": [
"ok",
"latency_ms"
],
"additionalProperties": false
},
"db_pool": {
"type": "object",
"properties": {
"saturated": {
"type": "boolean"
}
},
"required": [
"saturated"
],
"additionalProperties": false
},
"runners": {
"type": "array",
"items": {
"$ref": "#/components/schemas/RunnerHealth"
}
}
},
"required": [
"uptime_seconds",
"database",
"db_pool",
"runners"
],
"additionalProperties": false
},
"backends": {
"type": "array",
"items": {
"$ref": "#/components/schemas/Backend"
}
},
"degraded_backends": {
"type": "array",
"items": {
"type": "string"
}
},
"degraded_reasons": {
"type": "array",
"items": {
"type": "string"
}
}
},
"required": [
"status",
"wrapper",
"backends",
"degraded_backends",
"degraded_reasons"
],
"additionalProperties": false
},
"RunnerHealth": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"running": {
"type": "integer"
},
"seconds_since_poll": {
"type": [
"integer",
"null"
]
}
},
"required": [
"name",
"running",
"seconds_since_poll"
],
"additionalProperties": false
}
}
},
"paths": {
"/health": {
"get": {
"tags": [
"health"
],
"operationId": "health.health",
"parameters": [],
"security": [],
"responses": {
"200": {
"description": "Health",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Health"
}
}
}
},
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"503": {
"description": "Health",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/Health"
}
}
}
}
},
"description": "Return the state of the API in v1's shape: `ok`, `degraded` with its `degraded_reasons`, or `down`. It checks the database on its own connection, whether a pooled connection is free, and each runner, and never asks Inference. `backends` gives each served model's breaker and requests in flight, and `degraded_backends` the models whose breaker is open. 503 only when the database does not answer within `HEALTH_TIMEOUT`; a degraded API answers 200."
}
}
}
}
```
# Create Chat Completion (/docs/reference/chat/chat.createChatCompletion)
```json
{
"components": {
"schemas": {
"ErrorObject": {
"type": "object",
"properties": {
"message": {
"type": "string"
},
"type": {
"type": "string",
"description": "One of invalid_request_error, rate_limit_error, upstream_error, service_unavailable, internal_error."
},
"param": {
"type": [
"string",
"null"
]
},
"code": {
"type": [
"string",
"null"
],
"description": "What went wrong, the field to branch on: for example `invalid_request`, `model_not_found`, `budget_exceeded`, `rate_limited`, or `upstream_unreachable`. A Node or a Job line that failed before 2026-11-02 keeps the code of its day."
},
"retryable": {
"type": "boolean",
"description": "On an error of the model server, and on each error after the answer started: whether the same request can succeed when you send it again."
},
"retry_after_seconds": {
"type": "integer",
"description": "With `retryable: true`: wait this many seconds before you send the request again, as `Retry-After` says."
},
"upstream_status": {
"type": "integer",
"description": "The status that the model server answered with."
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"message",
"type",
"param",
"code"
],
"additionalProperties": false
},
"ErrorResponse": {
"type": "object",
"properties": {
"error": {
"$ref": "#/components/schemas/ErrorObject"
},
"request_id": {
"type": "string",
"pattern": "^[\\w-]{1,64}$",
"description": "The id of the request, as in its `X-Request-Id` header. Send it when you report a problem."
}
},
"required": [
"error",
"request_id"
],
"additionalProperties": false
}
}
},
"paths": {
"/v1/chat/completions": {
"post": {
"tags": [
"chat"
],
"operationId": "chat.createChatCompletion",
"parameters": [],
"security": [],
"responses": {
"400": {
"description": "The request is invalid: `param` names the field at fault, or Inference refused a value.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"413": {
"description": "The request body is over the body cap, `MAX_BODY_SIZE` (8 MiB by default), or `HARVEST_MAX_BODY_SIZE` (50 MiB by default) for POST /v1/harvest. It does not reach Inference.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
},
"500": {
"description": "The API failed.",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/ErrorResponse"
}
}
}
}
},
"description": "Not served: Base models have no chat format. Always answers 400, naming POST /v1/completions.",
"deprecated": true
}
}
}
}
```