Read token probabilities

Read how likely a Base model found each token: the tokens that it generates, the tokens of your prompt, or every token of its vocabulary. Use them to measure surprisal, to score a text, or to see what the model predicted.

Each section sets one field of POST /v1/completions. What to return lists these fields and their defaults.

Get log probabilities for generated tokensCopy Anchor Link

Set logprobs to a number k from 0 to 100. Each generated token then gets its log probability, and each generated position gets the k likeliest tokens. Leave logprobs out to get none.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": 5}'

The curl command and the Python program each send their own request, so their outputs show different samples. The curl command prints the Completion:

{
  "id": "cmpl-b3558603c61e9c6e",
  "object": "text_completion",
  "created": 1791780960,
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " growing clear",
      "logprobs": {
        "text_offset": [0, 8],
        "token_logprobs": [-6.239488124847412, -6.665775299072266],
        "tokens": [" growing", " clear"],
        "top_logprobs": [
          {
            " growing": -6.239488124847412,
            " the": -1.4269880056381226,
            " falling": -2.114488124847412,
            " blue": -2.489488124847412,
            " not": -2.926988124847412,
            " a": -3.176988124847412
          },
          {
            " clear": -6.665775299072266,
            " dark": -1.228275179862976,
            " darker": -1.353275179862976,
            " lighter": -3.0407752990722656,
            " brighter": -3.1657752990722656,
            " ever": -3.4157752990722656
          }
        ]
      },
      "finish_reason": "length",
      "prompt_logprobs": null
    }
  ],
  "usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}

The Python program prints one line for each generated token, with its text, its log probability, and its map of likeliest tokens:

' light' -5.864488124847412 {' light': -5.864488124847412, ' the': -1.4269880056381226, ' falling': -2.114488124847412, ' blue': -2.489488124847412, ' not': -2.926988124847412, ' a': -3.176988124847412}
'ening' -0.8648753762245178 {'ening': -0.8648753762245178, ' blue': -1.364875316619873, ',': -2.864875316619873, ' and': -3.114875316619873, ' but': -4.052375316619873}

choices[i].logprobs has four lists, with one entry for each generated token:

A map in top_logprobs also holds the generated token when it is not one of the k likeliest. So a map has k or k + 1 entries. In the Completion, " growing" is not among the 5 likeliest tokens, so its map has 6 entries. In the Python output, 'ening' is the likeliest token at its position, so its map has 5 entries.

logprobs: 0 still returns the log probability of each generated token. Each map then holds only the generated token.

The API refuses true and -1 with 400, because logprobs takes a number from 0 to 100. To get every token of the vocabulary, follow Get the whole vocabulary.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "logprobs": true}'

{
  "error": {
    "message": "Invalid type for 'logprobs': expected a number or null.",
    "type": "invalid_request_error",
    "param": "logprobs",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

For -1, the message is Invalid value for 'logprobs': expected a value greater than or equal to 0.

Get log probabilities for the promptCopy Anchor Link

Set prompt_logprobs to a number k from 0 to 100. Each prompt position then gets the log probability of the prompt's token and of the k likeliest tokens. prompt_logprobs is a field of vLLM that the openai SDK does not know, so the Python program sends it in extra_body. curl sends it with the other fields.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "prompt_logprobs": 5}'

The Python program prints one line for each token at each position. A line has the position, the token id, the token's text, its rank, and its log probability. The curl command prints the same fields in the Completion's JSON.

0 None
1 2 '#' 3 -2.6954822540283203
1 755 'def' 2 -2.1954822540283203
1 791 'The' 4 -3.6329822540283203
1 14924 'Question' 1 -1.1954821348190308
1 16309 'Tags' 5 -4.13298225402832
2 220 ' ' 1 -3.7540297508239746
2 502 ' new' 4 -5.129029750823975
2 1176 ' first' 3 -4.660279750823975
2 2768 ' following' 2 -4.285279750823975
2 7252 ' Best' 5 -5.222779750823975
2 13180 ' sky' 1144 -8.832155227661133
3 374 ' is' 1 -0.9844952821731567
3 574 ' was' 2 -2.109495162963867
3 596 "'s" 4 -3.359495162963867
3 753 '’s' 3 -3.109495162963867
3 3485 ' above' 5 -3.484495162963867

choices[i].prompt_logprobs has one entry for each prompt token, usage.prompt_tokens in all:

• Position 0 is null, because no token comes before it. On meta-llama/Llama-3.1-8B, position 0 is the BOS token <|begin_of_text|>, which Inference adds in front of a text prompt. The first token of your text is at position 1.
• Each later position maps a token id, as a string, to the token's logprob, its rank among all tokens, and its text in decoded_token. Rank 1 is the likeliest token.
• The prompt's own token is always in the map. When it is not one of the k likeliest, vLLM adds it with its rank.

At position 2, the prompt's token " sky" has rank 1144, so the model found 1,143 other tokens likelier after "The". At position 3, " is" has rank 1, so it was the likeliest token after "The sky".

prompt_logprobs does not work with stream: true. The API refuses the request with 400 and param prompt_logprobs before it holds any tokens.

To add up the log probabilities of a text that you already have, follow Score a text.

Get the whole vocabularyCopy Anchor Link

Set prompt_logprobs to -1 to get the log probability of every token in the vocabulary at each prompt position. On meta-llama/Llama-3.1-8B, that is 128,256 entries for each position. Use it for entropy, for the divergence between two models, or for the tail of the distribution. To rank candidates or to score a text, a number k is enough.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 1, "prompt_logprobs": -1}'

Inference holds every entry in memory, so the API takes -1 for at most 256 prompt tokens in all × n, over every prompt of the request. With output_residual_stream, the limit is 16. The API counts a text's tokens with the model's tokenizer, as the Hold does, and one token for each token id.

A larger request gets 400 with param prompt_logprobs before it holds any tokens. Size caps lists this cap with the others.

Get the whole vocabulary for generated tokensCopy Anchor Link

logprobs returns at most the 100 likeliest tokens at a generated position. To get every token, send the generated text back as part of a prompt:

1. Send the prompt, and keep the generated text and the usage.
2. Send the prompt and the generated text together as one new prompt, with prompt_logprobs: -1 and max_tokens: 1. The new prompt must fit the limit above.
3. Check that the new usage.prompt_tokens is the first request's prompt_tokens plus its completion_tokens. Inference tokenizes the text again, and a text can split into other tokens than the ones generated.
4. Read the new prompt_logprobs from position prompt_tokens of the first request on. Each of these positions holds a generated token, with every token of the vocabulary.

This program does the four steps for two generated tokens, and prints each generated position with the number of tokens in its entry:

import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
prompt = "The sky is"
first = client.completions.create(
    model="llama-8b", prompt=prompt, max_tokens=2, temperature=0
)
text = prompt + first.choices[0].text
again = client.completions.create(
    model="llama-8b",
    prompt=text,
    max_tokens=1,
    extra_body={"prompt_logprobs": -1},
)
start = first.usage.prompt_tokens
end = start + first.usage.completion_tokens
assert again.usage.prompt_tokens == end, "the text split into other tokens"
for position in range(start, end):
    print(position, len(again.choices[0].prompt_logprobs[position]))

Return the prompt with the CompletionCopy Anchor Link

Set echo: true to get your prompt back in choices[i].text, in front of the generated text. No separator comes between them.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "echo": true}'

The Python program prints the text, then the prompt tokens and the generated tokens of the usage:

'The sky is often beautiful'
4 2

The text starts with your prompt, without the BOS token. usage.prompt_tokens counts the BOS token that meta-llama/Llama-3.1-8B adds, and usage.completion_tokens counts only the generated tokens. For a text prompt, the generated text alone is text[len(prompt):].

With echo and logprobs, each list in choices[i].logprobs starts with one entry for each prompt token, then has one for each generated token. The first prompt token, BOS on meta-llama/Llama-3.1-8B, has null in token_logprobs and in top_logprobs, because no token comes before it. These prompt entries count toward the cap on the prompt's log probabilities in Size caps.

Next stepsCopy Anchor Link

Score a text adds up the log probabilities of a text to get its log-likelihood. Read Activations returns the residual stream at the Layers you choose.