Create Completion

The Base model: an id from GET /v1/models, such as llama-8b. Left out, the model that GET /v1/models marks default.

Text or a list of token ids, or a list of several such prompts. Each prompt gets n choices: choice i of prompt p has the index p * n + i.

Generate at most this many tokens for each choice, at most 32000 in all: the number of prompts × n × max_tokens. Without stream, at most 15600, what Inference finishes within the API's 14-minute window. Left out, it is the model's context left after the prompt, at most 32000 (15600 without stream); a request of more than one prompt, or with n above 1, must set it. Near the end of your token budget, the API lowers it to what fits and says so in X-Acs-Max-Tokens-Clamped. null is the same as leaving it out.

Sample less likely tokens more often; 0 takes the most likely token each time. From 0 to 2, as vLLM takes it.

How many choices to generate for each prompt; at most 128 in all.

Put the prompt before the generated text.

Return the log probabilities of this many top tokens; at most 100.

vLLM: log probabilities of the prompt tokens, and of this many top tokens at each, at most 100, and 33554432 in all: prompt tokens × n × (this + 1), and logprobs + 1 more with echo. -1 for the whole vocabulary, for at most 256 prompt tokens in all × n (16 with output_residual_stream). The API counts a text's tokens with the model's tokenizer, as Inference does, and token ids exactly. Not with stream: true.

Sample from the most likely tokens up to this probability; more than 0 and at most 1, as OpenAI and vLLM take it.

vLLM: sample from this many most likely tokens; 0 or -1 for all.

vLLM: leave out tokens less likely than this fraction of the most likely; from 0 to 1.

Make a token that is already in the text less likely; from -2 to 2.

Make a token less likely each time it is already in the text; from -2 to 2.

vLLM: divide the scores of repeated tokens by this, more than 0; 1 for none.

From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests.

Stop before any of these texts; at most 4.

Send Server-Sent Events as tokens arrive.

Only with stream: true.

vLLM: a lower number runs first, from 0. Requests of equal priority wait in order of arrival. null is 0; a text of digits and true or false count as their number.

Your id for the person who asked.

vLLM: add the model's special tokens, such as BOS, to text.

vllm-lens: return the residual stream at these layers, in any order, each once, or true for every layer of the model, or false for none. Layer 0 is the first, and the last is capabilities.n_layers - 1 in GET /v1/models. A capture is at most 33554432 numbers (64 MiB as bfloat16): layers × tokens × the model's hidden size, where the tokens are the prompt's (as the model's tokenizer counts them) and max_tokens. So layers × tokens is at most 8192 for llama-8b (hidden size 4096), 2048 for llama-405b (hidden size 16384), 10922 for trinity-truebase (hidden size 3072); a model that the catalogue does not have counts as its largest hidden size, 16384.

vllm-lens: add these vectors to the residual stream. At most 8 vectors, and 2097152 numbers in all (8 MiB as float32): layers × positions × the model's hidden size.