Create Completion Job

The Base model: an id from GET /v1/models, such as llama-8b.

Text or a list of token ids, or a list of several such prompts. Each prompt gets n choices: choice i of prompt p has the index p * n + i.

Generate at most this many tokens for each choice. A request generates at most 32000 tokens in all: the number of prompts × n × max_tokens.

Sample less likely tokens more often; 0 takes the most likely token each time. From 0 to 2, as vLLM takes it.

How many choices to generate for each prompt; at most 128 in all.

Put the prompt before the generated text.

Return the log probabilities of this many top tokens; at most 100.

vLLM: log probabilities of the prompt tokens, and of this many top tokens at each, at most 100, and 33554432 in all: prompt tokens × n × (this + 1), and logprobs + 1 more with echo. -1 for the whole vocabulary, for at most 256 prompt tokens in all × n (16 with output_residual_stream). The API counts a text's tokens with the model's tokenizer, as Inference does, and token ids exactly. Not with stream: true.

Sample from the most likely tokens up to this probability; more than 0 and at most 1, as OpenAI and vLLM take it.

vLLM: sample from this many most likely tokens; 0 or -1 for all.

vLLM: leave out tokens less likely than this fraction of the most likely; from 0 to 1.

Make a token that is already in the text less likely; from -2 to 2.

Make a token less likely each time it is already in the text; from -2 to 2.

vLLM: divide the scores of repeated tokens by this, more than 0; 1 for none.

From 0. The same seed samples the same tokens; vLLM can still change one now and then when it batches requests.

Stop before any of these texts; at most 4.

A completion job is never streamed.

Only with stream: true.

vLLM: a lower number runs first, from 0. Requests of equal priority wait in order of arrival. null is 0; a text of digits and true or false count as their number.

Your id for the person who asked.

vLLM: add the model's special tokens, such as BOS, to text.

vllm-lens: return the residual stream at these layers, in any order, each once, or true for every layer of the model, or false for none. Layer 0 is the first, and the last is capabilities.n_layers - 1 in GET /v1/models. A capture is at most 33554432 numbers (64 MiB as bfloat16): layers × tokens × the model's hidden size, where the tokens are the prompt's (as the model's tokenizer counts them) and max_tokens. So layers × tokens is at most 8192 for llama-8b (hidden size 4096), 2048 for llama-405b (hidden size 16384), 10922 for trinity-truebase (hidden size 3072); a model that the catalogue does not have counts as its largest hidden size, 16384.

vllm-lens: add these vectors to the residual stream. At most 8 vectors, and 2097152 numbers in all (8 MiB as float32): layers × positions × the model's hidden size.