Base models
Every model that the API serves is a Base model, which treats what you send as the start of a text and continues it. So the shape of your prompt decides the shape of the answer.
What a Base model does
A Base model is a pretrained language model without instruction or chat tuning. It predicts the next token of a text, then the one after it, and so it continues the text that you send. It does not treat your text as a question to answer. A prompt such as "What is the capital of France?" can go on with more questions as well as with an answer. The prompt "The capital of France is" asks for the continuation that you want.
The model reads tokens, not characters. For a text prompt, the tokenizer of llama-8b adds the BOS token <|begin_of_text|> in front, so "The sky is" is 4 tokens. Inference adds no special tokens to a prompt of token ids, so the ids that you send are the whole prompt.
The API follows OpenAI's completions API, so the openai SDK works with it. It also returns the log probability of each token, the model's predictions over the prompt, and the Activations at the Layers that you choose.
No chat template
A chat model expects its conversation inside a template, with special tokens that mark each turn and who speaks. A Base model was not trained on such a template, so the API applies none. The model continues the text that you send, with only the BOS token in front.
For the same reason, the API serves no chat route. POST /v1/chat/completions always answers 400 invalid_request_error, with this message:
This API serves Base models, which have no chat format: send POST /v1/completions with a prompt instead of POST /v1/chat/completions with messages.In the openai SDK, that means client.completions.create(...) and not client.chat.completions.create(...). A conversation can still go in as plain text, as a transcript that the model continues.
Why the API is strict
The API checks the name and the type of every field before a request reaches Inference, and it refuses what it does not know. A field that it does not know gets 400 with the code invalid_request. A value of the wrong type gets 400, with param set to the field. The API does not replace a bad value with a default.
Research depends on knowing which settings made a result. If the API ignored a misspelled field such as temprature, the request would run at the default temperature, and nothing in the answer would show it. A refusal turns that silent mistake into an error that you see at once.
The same rule covers fields that other servers accept and ignore. vLLM accepts best_of and does nothing with it, so the API refuses best_of as an unknown field. Types are strict too: "2" is a string and not a number, and 1 is not true. Strict validation lists every check.
Default sampling
A field that you leave out gets OpenAI's default, or the value that turns a vLLM field off: temperature 1, top_p 1, top_k 0, no penalties. At these defaults, Inference samples each token from the model's own distribution, unchanged.
A model's repository can set its own sampling defaults in generation_config.json. The API sends Inference every field, defaults included, so that file never decides a value. A request then means the same on every model. When you send one request to two models, both sample the same way, and a difference in the text comes from the models.
max_tokens has rules of its own, which Request fields lists.
Why repeats differ
A Base model samples each token, so the same request at the default temperature writes other text on each run. The outputs in these docs are one sample each, recorded from meta-llama/Llama-3.1-8B, and your outputs differ.
With temperature 0, Inference takes the likeliest token each time, and with the same seed, the same request on the same model samples the same tokens again. vLLM's batching can still change a result a little. Inference runs your request in a batch with other requests, and the numbers that the GPU computes depend a little on that batch. A log probability can then move by a small amount, and now and then a token changes where two tokens were almost equally likely. Read a small difference between two runs as no difference, and send a request more than once when the difference matters.
Next steps
Sample continuations sets the sampling fields and sends several prompts. Completions lists every field and its default. Read token probabilities measures what the model predicts at each position.