Evaluate with Inspect

Run an eval on a Base model with Inspect, the UK AI Security Institute's framework for evals. Its provider openai-api-completions sends each sample's input as a raw prompt to POST /v1/completions, with no chat template. You need the two shell variables of the Quickstart. The commands on this page work with inspect-ai 0.3.286 and the openai SDK 3.41.0.

Point Inspect at the APICopy Anchor Link

1. Install Inspect and the openai SDK:

   Copy Text

   pip install inspect-ai openai

2. Give Inspect the API's address. For a service named acs, Inspect reads the key from ACS_API_KEY, which the Quickstart sets, and the address from ACS_BASE_URL:

   Copy Text

   export ACS_BASE_URL="$ACS_API_BASE"

3. Name the model as openai-api-completions/acs/ and the model's full id, such as openai-api-completions/acs/llama-8b. Inspect takes acs/ off and sends the rest as model. List the models gives each id.

Run a taskCopy Anchor Link

1. Download capitals.py, or save this task as capitals.py. Each sample is a prompt with two examples, which a Base model continues. includes() scores a sample 1 when the model's text contains its target:

   Copy Text

   from inspect_ai import Task, task
   from inspect_ai.dataset import Sample
   from inspect_ai.scorer import includes
   from inspect_ai.solver import generate
   
   EXAMPLES = "The capital of Italy is Rome.\nThe capital of Japan is Tokyo.\n"
   
   
   @task
   def capitals():
       return Task(
           dataset=[
               Sample(input=EXAMPLES + "The capital of France is", target="Paris"),
               Sample(input=EXAMPLES + "The capital of Argentina is", target="Buenos Aires"),
               Sample(input=EXAMPLES + "The capital of Canada is", target="Ottawa"),
           ],
           solver=generate(),
           scorer=includes(),
       )

2. Run the eval on Llama 3.1 8B at temperature 0, with up to 4 tokens for each answer:

   Copy Text

   inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0 --max-tokens 4

   Inspect prints the task's accuracy, and writes the log of the eval to ./logs. inspect view shows the logs, with each sample's input and the model's text.

Always set --max-tokens. Without it, the provider asks for 1 token, and an answer such as Buenos Aires takes two.

Write prompts for a Base modelCopy Anchor Link

A Base model continues text, and it does not answer a question as a chat model does (What a Base model does). Write each input as the start of a text whose next words are the answer, as the examples in capitals.py do.

The provider sends one text. A sample's input must be a string or one user message. An input of several messages, or --system-message, stops the eval with a TypeError.

Set the request fieldsCopy Anchor Link

The provider sends model, prompt, max_tokens, and temperature with each request. It sends max_tokens: 1 without --max-tokens, and temperature: 0 without --temperature. These options of inspect eval set the other fields:

• The scorers read the first choice only, so --num-choices adds texts to the log and not to the score.
• --top-k, --best-of, and --logit-bias send nothing, and neither does --top-logprobs without --logprobs. Send top_k in --extra-body.
• The provider sends no request with a stream. So max_tokens must stay within the limit of a request without a stream (Tokens).
• The API refuses a field that it does not know, such as vLLM's return_token_ids, with 400 invalid_request (Strict validation).

Run a large evalCopy Anchor Link

1. Set --max-connections 16, the number of Completions that run at once for each API key (Traffic). Without it, Inspect opens 20 connections at first and up to 100. Past the key's line, the API answers 429 rate_limited.
2. Set --max-retries, such as --max-retries 3. Inspect tries each 429 again with no end. That includes budget_exceeded at the end of your tokens and spend_cap, which the API sends with x-should-retry: false.
3. Run the eval on a Warm model. The first request to a cold model waits for its start. A start of Llama 3.1 405B or Trinity Large takes longer than the API waits for Inference (Warm and cold models). Check whether a model is Warm shows the status route.

inspect eval capitals.py --model openai-api-completions/acs/llama-8b --temperature 0 --max-tokens 4 --max-connections 16 --max-retries 3

If the eval failsCopy Anchor Link

Next stepsCopy Anchor Link

Score a text gets the log-likelihood of a text, which an eval with several choices can compare. Run a Job sends many requests in the background. Limits lists each limit that an eval can meet.