Cheatsheet
Each section gives the smallest request for one task, and links to the page that explains it. Each curl command reads the API's address from ACS_API_BASE and your key from ACS_API_KEY. The commands that also need JOB_ID, LOOM_ID, or NODE_ID say where the value comes from.
Send a first Completion
The continuation comes back in choices[0].text. The Quickstart shows the response.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'Set the sampling fields
A field that you leave out gets OpenAI's default, such as temperature 1. The same request with the same seed samples the same tokens, and vLLM's batching can still change one now and then. stop ends the text where one of its strings starts. Set the sampling fields shows each field.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0.7, "top_p": 0.9, "seed": 7, "stop": ["\n"]}'Sample many continuations
A list of prompts and n return one choice for each prompt and each sample. Choice i of prompt p has the index p * n + i. Send several prompts shows more.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": ["The sky is", "The grass is"], "max_tokens": 2, "n": 2}'Read log probabilities
logprobs: 5 returns the log probability of each generated token, and of the 5 likeliest tokens at its position. Get log probabilities for generated tokens explains the lists.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 5}'Read the prompt's log probabilities
prompt_logprobs: 5 returns the log probability of each prompt token, and of the 5 likeliest tokens at its position. -1 returns the whole vocabulary, for at most 256 prompt tokens in all × n. The openai SDK sends this field only in extra_body. Get log probabilities for the prompt shows both.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 1, "prompt_logprobs": 5}'Echo the prompt
echo: true puts the prompt in front of the generated text, with no separator. Return the prompt with the Completion shows it.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "echo": true}'Stream a Completion
stream: true sends Server-Sent Events that end with data: [DONE]. include_usage adds a last chunk with the token counts. Stream a Completion shows the events.
curl -N "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "stream_options": {"include_usage": true}}'Read Activations
output_residual_stream returns the residual stream at the Layers in the list, in activations.residual_stream. Decode the tensor shows how to read it.
curl "$ACS_API_BASE/completions" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "output_residual_stream": [0, 2]}'Steer a Completion
apply_steering_vectors adds each vector to the residual stream at the Layers in its layer_indices. A vector has one number for each dimension of the model's hidden size, 4096 for meta-llama/Llama-3.1-8B. This example is a Python program, because it builds the vector with NumPy. Steer a Completion explains the fields.
import os
import numpy as np
from openai import OpenAI
vector = np.zeros(4096)
vector[0] = 0.5
client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
model="llama-8b",
prompt="The sky is",
max_tokens=2,
extra_body={"apply_steering_vectors": [{"activations": [vector.tolist()], "layer_indices": [1], "scale": 2.0}]},
)
print(repr(completion.choices[0].text))Run a Job
A Job runs one Completion request on the server. The first command submits a Job with an Idempotency-Key. The other two need JOB_ID, the id in the first command's answer. The second reads the Job's status. The third fetches the result once the Job has ended, with the status that the Completion ended with. Run a Job explains each step.
curl "$ACS_API_BASE/completion-jobs" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: sky-001" \
-d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2}'curl "$ACS_API_BASE/completion-jobs/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"curl "$ACS_API_BASE/completion-jobs/$JOB_ID/result" \
-H "Authorization: Bearer $ACS_API_KEY"Harvest Activations
A harvest reads the residual stream of many prompts at the Layers that you choose, and writes it to files. The first command submits a harvest. The other two need JOB_ID, the job_id in the first command's answer. The second waits up to 60 seconds for the harvest's status to change. Once the harvest is done, its result has links to the files. The third cancels a harvest that is pending or running. Harvest Activations explains each step.
curl "$ACS_API_BASE/harvest" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}'curl "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
-H "Authorization: Bearer $ACS_API_KEY"curl -X DELETE "$ACS_API_BASE/harvest/$JOB_ID" \
-H "Authorization: Bearer $ACS_API_KEY"Grow a Loom
A Loom is a tree of Nodes that grows from a prompt. The first command makes a Loom. The other two need LOOM_ID, the Loom's id in its answer. The second also needs NODE_ID, the id of the root Node in nodes[0]. It asks for a Branch of 2 Nodes under the root, and the third reads the Loom with the new Nodes. Grow a Loom with the API explains each step.
curl "$ACS_API_BASE/looms" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"prompt": "The sky is"}'curl "$ACS_API_BASE/looms/$LOOM_ID/nodes/$NODE_ID/branch" \
-H "Authorization: Bearer $ACS_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model": "llama-8b", "max_tokens": 2, "n": 2}'curl "$ACS_API_BASE/looms/$LOOM_ID" \
-H "Authorization: Bearer $ACS_API_KEY"List the models
GET /v1/models returns the id to send as model, and the context length of each model in capabilities.max_model_len. List the models describes each field.
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"Tokens
Before Inference runs a request, the API holds the most tokens that the request can use, as The Hold explains. When your tokens cannot hold all of them, POST /v1/completions lowers max_tokens to what fits and sends X-Acs-Max-Tokens-Clamped. It answers 429 budget_exceeded only when not even one token for each choice fits. Without stream, a max_tokens above 15,600 gets 400 deadline_infeasible, and a max_tokens that you leave out is at most 15,600.
Headers
| Header | Set by | What it does |
|---|---|---|
Authorization: Bearer $ACS_API_KEY | You | Sends your API key. Every /v1 route needs it. |
Content-Type: application/json | You | Says that the body is JSON. The API reads the body of a /v1 request as JSON whatever its type, so curl -d without this header works too. |
Idempotency-Key | You | Runs a request that writes only once. The same request again with the same key gets the first answer (Send a request again safely). |
Accept-Encoding: gzip | You | Asks for a JSON answer of 500 bytes or more compressed, with Content-Encoding: gzip. The openai SDKs and curl --compressed send it and decompress the answer. A stream is not compressed. |
Content-Type: text/event-stream | The API | Marks a stream. |
Location | The API | The path of a Job, on the answer of POST /v1/completion-jobs. |
X-Acs-Max-Tokens-Clamped | The API | Says that POST /v1/completions lowered max_tokens, from which value to which, and why (Detect a lowered max_tokens). |
Allow | The API | Lists the methods of a route, with a 405 answer. |
Retry-After, retry-after-ms | The API | Give the wait before the next try. The API sets Retry-After on a 429 of its traffic limits, a 429 harvest_concurrency_exceeded or harvest_capacity_exceeded, a 429 spend_cap (an hour), and a 409 running. A 202 of a Job's result has it too, a second. It sets it on a 503, and on a 502 that retryable says to try again. When Inference is busy, the API passes on both headers as Inference sent them, with a 429 rate_limited. |
x-should-retry: false | The API | Tells the openai SDK not to send the request again at once (Which errors to try again). It comes with 400 deadline_infeasible, 409 no_answer, 409 idempotency_conflict, 409 harvest_not_cancellable, and a 409 about the status of a Node. It comes with 429 budget_exceeded, harvest_quota_exceeded, job_capacity, and spend_cap. It comes with the API's own 500, with 413 request_too_large, with 504 upstream_deadline, and with each error of Inference whose retryable is false. |
X-Acs-Longpoll: declined | The API | Says that a read of a harvest with wait answered at once, because too many reads wait. It comes with Retry-After: 3 (Check on the Job). |
X-Request-Id | You, or the API | The id of the request, on every answer. An error has the same id in request_id at the top of its body. Send it when you report a problem. If you send an X-Request-Id of 1 to 64 letters, digits, -, or _, the API uses yours. |
Errors
Every error has the error object of the old platform's API. Two errors can have the same status, so tell them apart by code. Errors lists every status.
| Status | code | More |
|---|---|---|
| 400 | invalid_request | Unknown field |
| 400 | bad_json | Status codes |
| 400 | model_not_found | Unknown model |
| 400 | context_length_exceeded | Prompt too long |
| 400 | deadline_infeasible | Requests without a stream |
| 400 | harvest_unsupported | Harvests on each model |
| 401 | missing_authorization, invalid_api_key, key_revoked, key_disabled, or account_suspended | Send the key |
| 404 | job_not_found or harvest_job_not_found | Status codes |
| 409 | null | Status codes |
| 409 | running or no_answer | Send a request again safely |
| 429 | budget_exceeded | Not enough tokens |
| 429 | rate_limited | Traffic or Jobs |
| 429 | queue_full | Traffic |
| 429 | harvest_quota_exceeded, harvest_concurrency_exceeded, or harvest_capacity_exceeded | Harvests |
| 429 | spend_cap | Spend limit |
| 500 | internal_error | Status codes |
| 502, 503, or 504 | upstream_unreachable, upstream_server_error, vllm_oom, vllm_engine_dead, circuit_open, or upstream_deadline | Inference fails |
Limits
Limits gives every number for tokens, traffic, API keys, Jobs, harvests, Looms, request size, time limits, and kept data.
Next steps
Completions describes every field of the request and of the response. The Cookbook has programs for whole tasks.