Score a text

Get the log-likelihood of a text under a Base model: the sum of the log probabilities of its tokens. You need Python and the openai SDK, set up as in the Quickstart.

Score the textCopy Anchor Link

1. Find the token ids of your text. Send the text with prompt_logprobs: 0 and max_tokens: 1. The openai SDK does not know prompt_logprobs, so send it in extra_body. With 0, each position has one entry, the prompt's own token, keyed by its id. Ignore the one generated token.

   Copy Text

   import os
   
   from openai import OpenAI
   
   client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
   completion = client.completions.create(
       model="llama-8b",
       prompt="The sky is",
       max_tokens=1,
       extra_body={"prompt_logprobs": 0},
   )
   for position, tokens in enumerate(completion.choices[0].prompt_logprobs):
       print(position, tokens)

   The program prints the entry of each position:

   Copy Text

   0 None
   1 {'791': {'logprob': -3.6329822540283203, 'rank': 4, 'decoded_token': 'The'}}
   2 {'13180': {'logprob': -8.832155227661133, 'rank': 1144, 'decoded_token': ' sky'}}
   3 {'374': {'logprob': -0.9844952821731567, 'rank': 1, 'decoded_token': ' is'}}

   • Position 0 is None, because no token comes before it. meta-llama/Llama-3.1-8B puts the BOS token <|begin_of_text|> in front of a text, and its id is 128000. The response does not show it.
   • From position 1 on, the key is the id of your token, as a string. Each token of your text has an id and a log probability, because BOS takes position 0.
   • rank is 1 for the likeliest token at a position. Your token is not always the likeliest one. Here " sky" has rank 1144, and the model found 1,143 other tokens likelier after "The".

   Match tokens by id, not by text, because two ids can decode to the same text.

2. Send the ids as the prompt, with the BOS id 128000 first. Inference then scores these tokens and no others. It adds no special token, such as BOS, to a prompt of token ids, so put BOS first, as the model does for a text. Position 0 has no entry, and this program reads each of your ids, in scored, from position 1 on:

   Copy Text

   import os
   
   from openai import OpenAI
   
   client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
   completion = client.completions.create(
       model="llama-8b",
       prompt=[128000, 791, 13180, 374],
       max_tokens=1,
       extra_body={"prompt_logprobs": 0},
   )
   scored = [791, 13180, 374]
   positions = completion.choices[0].prompt_logprobs[1:]
   for token_id, tokens in zip(scored, positions):
       print(token_id, tokens[str(token_id)]["logprob"])

   Copy Text

   791 -3.6329822540283203
   13180 -8.832155227661133
   374 -0.9844952821731567

   tokens[str(token_id)] finds each token by its id. The program reads the log probability of your token, not of another token with the same text.

3. Add up the log probabilities from position 1 on:

   Copy Text

   import os
   
   from openai import OpenAI
   
   client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
   completion = client.completions.create(
       model="llama-8b",
       prompt=[128000, 791, 13180, 374],
       max_tokens=1,
       extra_body={"prompt_logprobs": 0},
   )
   scored = [791, 13180, 374]
   positions = completion.choices[0].prompt_logprobs[1:]
   print(sum(tokens[str(token_id)]["logprob"] for token_id, tokens in zip(scored, positions)))

   Copy Text

   -13.44963276386261

   The sum is the log-likelihood of your text after the BOS token. BOS has no log probability, and it is not part of your text.

Next stepsCopy Anchor Link

Read token probabilities returns the k likeliest tokens at each position of a prompt. Get the whole vocabulary returns every token's log probability at each position. Read Activations for text you already have reads the residual stream of the same text.