Read Activations

Read the Activations of a Completion, which are the residual stream of the Base model at the Layers you choose. The residual stream explains what the numbers are.

Ask for them with the field output_residual_stream on POST /v1/completions. The field has the name that vllm-lens uses. The openai SDK does not know it, so the Python programs send it in extra_body. A request with Activations holds and spends tokens as any other Completion does, and the Activations cost no tokens (Budgets and Holds).

The Python programs need Python 3.14 (for compression.zstd), NumPy, and the openai SDK.

Read the residual streamCopy Anchor Link

List the Layers in output_residual_stream. The API sorts them and reads a repeated Layer once. Layer 0 is the first Layer. llama-8b has 32 Layers, from 0 to 31, as capabilities.n_layers in GET /v1/models says. To read every Layer, send "output_residual_stream": true. Each Layer counts against the size cap of a capture, so a read of every Layer takes a short prompt.

This request reads Layers 0 and 2:

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "output_residual_stream": [0, 2]}'

The curl command prints the Completion, with an activations field after usage:

{
  "id": "cmpl-998dce4da6c3c3b2",
  "object": "text_completion",
  "created": 1791780538,
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " burning\n",
      "logprobs": null,
      "finish_reason": "length",
      "prompt_logprobs": null
    }
  ],
  "usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 },
  "activations": {
    "residual_stream": {
      "data": "KLUv/aAAQAEArf4HDgDU8z9FEBCrlaQDMmACBRI4cdIDCRRI...",
      "dtype": "int16",
      "original_dtype": "torch.bfloat16",
      "shape": [2, 5, 4096],
      "compression": "zstd"
    }
  }
}

The Python program sends its own request. It decodes the residual stream, and prints its shape and the first 4 numbers of the first token at each Layer:

(2, 5, 4096)
[[0.004058837890625, -0.0002899169921875, -1.33514404296875e-05, -0.002349853515625], [0.0087890625, 0.01513671875, -0.07421875, -0.0537109375]]

The shape is 2 Layers, then 5 tokens, then the hidden size of 4096. The 5 tokens are the 4 prompt tokens, with the BOS token first, and the first of the 2 generated tokens. A read never has a row for the last generated token, as What a read returns explains.

Decode the tensorCopy Anchor Link

activations.residual_stream is an array in vllm-lens's encoding:

The Layers come in ascending order, each one once. The tokens are the prompt tokens and the generated tokens, except the last generated token. So the array has usage.prompt_tokens + usage.completion_tokens - 1 tokens.

The API checks that shape before it answers. When Inference's answer has no activations, or another shape, the API answers 502 upstream_server_error with the message Inference's answer does not fit the request, and the request costs nothing. In a stream, the error comes as the last event, and you pay for the tokens that came before it.

Every Python program in the guides decodes the array with this decode helper. Copy it as it is:

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

llama-8b computes in bfloat16, and NumPy has no bfloat16 type. So its numbers come as "dtype": "int16" with "original_dtype": "torch.bfloat16", and each int16 value holds the bits of one bfloat16 number. A bfloat16 number is the upper 16 bits of a float32 number, so decode shifts the bits into a float32 array. With PyTorch, use torch.from_numpy(array).view(torch.bfloat16) on the int16 array instead. Do not use array.astype(np.float32) on the int16 array, because it converts the integers and not the bits.

A Completion without output_residual_stream has no activations field.

Check the tensor before you use itCopy Anchor Link

Your code assumes which Layers and tokens each row holds. Check the shape and the numbers before you use them, so that a wrong assumption fails at once. For example, a prompt can have one more token than you counted, such as the BOS token that the model adds.

1. Decode the array to float32 with decode.
2. Check that the first axis has one row for each Layer that you asked for.
3. Check that the second axis has usage.prompt_tokens + usage.completion_tokens - 1 tokens.
4. Check that every number is finite.

This program does the four steps for the request above:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    extra_body={"output_residual_stream": [0, 2]},
)
residual_stream = decode(completion.activations["residual_stream"])

layers, tokens, hidden_size = residual_stream.shape
usage = completion.usage
assert layers == 2, residual_stream.shape
assert tokens == usage.prompt_tokens + usage.completion_tokens - 1, residual_stream.shape
assert np.isfinite(residual_stream).all(), "a number is not finite"
print(layers, tokens, hidden_size)

2 5 4096

The last axis is the model's hidden size. Each row of a Steering vector needs that many numbers.

Read Activations for text you already haveCopy Anchor Link

To read the Activations of a text, send it as the prompt with max_tokens: 1. The one generated token gets no row, so the array has one row for each prompt token.

The text of a Completion can split into other tokens when you send it back as text. If you have its token ids, send them as the prompt instead. Inference adds special tokens, such as BOS, only to a text prompt. So start the list of ids with the BOS id, 128000 for meta-llama/Llama-3.1-8B, as the model does for a text.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": [128000, 791, 13180, 374], "max_tokens": 1, "output_residual_stream": [1]}'

This array has 4 tokens, one for each id.

Read Activations for many promptsCopy Anchor Link

A request with output_residual_stream takes one prompt and n: 1. To read the Activations of many prompts, choose one of two ways:

• If you want the Activations of each prompt's tokens as files, for example to train a probe, harvest them. A harvest reads thousands of prompts on GPUs of its own and writes safetensors files. It generates no text.
• If you want each prompt's Completion too, or a Steering vector, submit a Job for each prompt. The result of each Job is then a Completion with its own activations. Ask for Activations in a Job shows such a Job.

Stream with ActivationsCopy Anchor Link

With stream: true, the Activations come in one last chunk before [DONE]. That chunk has the same id, object, created, and model as the others, "choices": [], and activations.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "stream": true, "output_residual_stream": [1]}'

data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" the","logprobs":null,"finish_reason":null}]}

data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[{"index":0,"text":" limit","logprobs":null,"finish_reason":"length"}]}

data: {"id":"cmpl-ada9e6238bfc6248","object":"text_completion","created":1791783562,"model":"meta-llama/Llama-3.1-8B","choices":[],"activations":{"residual_stream":{"data":"KLUv/WAAn/X+Aw4ACvYfQhAQ6jWkA8YkXqo52Z6TdczcoPxB...","dtype":"int16","original_dtype":"torch.bfloat16","shape":[1,5,4096],"compression":"zstd"}}}

data: [DONE]

Requests that the API refusesCopy Anchor Link

The API answers 400 invalid_request_error for these requests before it holds any tokens, and they do not reach Inference. param is output_residual_stream.

• An empty list, or a negative Layer.
• A Layer that the model does not have. Layer 32 of meta-llama/Llama-3.1-8B gets meta-llama/Llama-3.1-8B has Layers 0 to 31: it has no Layer 32.
• n other than 1, or several prompts.
• A capture over its size cap: Layers × tokens × the model's hidden size. The tokens are the prompt's, as the Hold counts them, and max_tokens. On meta-llama/Llama-3.1-8B, Layers × tokens is at most 8,192, for example all 32 Layers for 256 tokens.
• A capture of llama-8b or llama-405b without max_tokens, or of more than one Layer of trinity-truebase without it. Left out, max_tokens is the room left in the context after the prompt. That is up to 15,600 without a stream and 32,000 with one, more tokens than the capture's cap allows. Set max_tokens with every capture.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "n": 2, "output_residual_stream": [0]}'

{
  "error": {
    "message": "Set n to 1: activations and steering vectors are for one choice only.",
    "type": "invalid_request_error",
    "param": "output_residual_stream",
    "code": "invalid_request"
  },
  "request_id": "req_9f2c4e1a7b3d4c5e8f6a0b1c2d3e4f5a"
}

The API checks the Layers against the facts that GET /v1/models lists. For a served model without those facts, Inference checks the Layers, and the API passes its 400 on.

Next stepsCopy Anchor Link

Steer a Completion adds a vector to the residual stream and reads the change. Steer with a contrast vector builds a Steering vector from the Activations of two sets of prompts.