Completions

POST /v1/completions continues one prompt, or several prompts, with a Base model. It follows OpenAI's completions API, so the openai SDK calls it with client.completions.create(...). Sample continuations shows the tasks, and the API reference has the schema of every field.

Request fieldsCopy Anchor Link

prompt is required. Apart from model and max_tokens, a field that you leave out, or sets to null, gets OpenAI's default, or the value that turns a vLLM field off. The model's own generation_config.json never sets a default.

A field marked vLLM comes from vLLM, the engine that runs the Base models in Inference, and is not in OpenAI's API. A field marked vllm-lens comes from vllm-lens, the vLLM plugin that reads and changes the residual stream. The openai SDK sends both kinds only in extra_body (Send vLLM's fields).

What to continueCopy Anchor Link

The rules of max_tokens:

• Left out or null, it is the model's max_model_len less the prompt's tokens as the Hold counts them. It is at least 1 and at most 32,000, and at most 15,600 without stream.
• A request with more than one prompt, or with n above 1, must set it.
• A request writes at most 32,000 tokens in all: the prompts × n × max_tokens (Size caps).
• Without stream, a value above 15,600 gets 400 deadline_infeasible. Inference cannot write more within the API's 14-minute wait (Requests without a stream).
• The prompt and max_tokens together must fit the model's max_model_len (Prompt too long).
• Near the end of your tokens, the API lowers it to what your Budget can hold. The header X-Acs-Max-Tokens-Clamped then says so (Near the end of your tokens).
• In a Job's request and in a Loom Branch, it is 16 when you leave it out.

How to sampleCopy Anchor Link

At temperature 0, or with the same seed, the same request on the same model gets the same tokens and log probabilities again, with one exception. vLLM batches your request with other requests. The batch can change a log probability by a little, and now and then a token where two tokens are almost equally likely.

What to returnCopy Anchor Link

Read token probabilities, Read Activations, and Steer a Completion show these fields in use.

Other fieldsCopy Anchor Link

Strict validationCopy Anchor Link

The API checks the type of every field before the request reaches Inference, and it refuses what it does not know. A wrong value never falls back to a default (Why the API is strict).

• A field that the API does not know gets 400 with the code invalid_request (Unknown field).
• A value of the wrong type gets 400, with param set to the field. A number in a string, such as "2", is not a number, except for priority. 1 and "true" are not booleans. A whole number such as 1 is valid for a decimal field such as temperature.
• The API checks the ranges of logprobs, prompt_logprobs, seed, priority, and the sampling fields in How to sample. It also checks the shape of the Activations and Steering vector fields, and the size caps.
• The API refuses n above 1 at a temperature below 0.00001, where Inference picks the likeliest token each time.
• The API refuses stream_options without stream: true, and prompt_logprobs with stream: true.
• Inference checks the other values, for example that each token id of a prompt is in the model's vocabulary. The API passes on Inference's 400, with Inference's message and param.

This request sends max_tokens as a string, so the API refuses it:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": "2"}'

{
  "error": {
    "message": "Invalid type for 'max_tokens': expected a number.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

Size capsCopy Anchor Link

Each cap keeps one request from using too much memory in the API or in Inference. A request over a cap gets 400 invalid_request_error, with param set to the field, and does not reach Inference. The caps count a prompt's tokens as the Hold counts them, with the model's tokenizer.

llama-8b has a hidden size of 4,096, so one capture is at most 8,192 Layers × tokens. That is all 32 Layers for 256 tokens, or 1 Layer for 8,192 tokens.

The whole request body is at most 8 MiB on every route, and a larger body gets 413 (Status codes). The body can cap the Steering vectors first: in JSON, 8 MiB holds about 699,000 numbers of 12 characters each. Encoded Steering vectors take about a tenth of those bytes, so eight vectors over all 32 Layers of meta-llama/Llama-3.1-8B fit.

This request asks for 2 choices of 16,001 tokens, which is 32,002 tokens in all:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 16001, "n": 2}'

{
  "error": {
    "message": "At most 32000 generated tokens in all: n = 2 times max_tokens = 16001 for each of 1 prompts is 32002.",
    "type": "invalid_request_error",
    "param": "max_tokens",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

Response fieldsCopy Anchor Link

The Quickstart shows a whole Completion.

Each choice has these fields:

In a stream, each data: event is a chunk with the same id, object, created, and model. Its choices has the next text of one choice, with finish_reason and stop_reason null until the last chunk of that choice. With include_usage, a last chunk has "choices": [] and the usage of the whole request. The Activations come in a last chunk of their own, and data: [DONE] ends the stream. Every 5 seconds the stream also has a : keepalive comment line, which SSE clients skip (Requests with a stream).

Not in the APICopy Anchor Link

The API does not take these fields and routes of OpenAI's API:

• best_of. vLLM accepts it and ignores it, so the API refuses it as an unknown field. n returns each choice instead.
• suffix. vLLM supports it only for models that fill in the middle of a text.
• logit_bias.
• Chat. The API applies no chat template (No chat template). POST /v1/chat/completions gets 400 invalid_request_error with the message This API serves Base models, which have no chat format: send POST /v1/completions with a prompt instead of POST /v1/chat/completions with messages.