The residual stream
The residual stream is the vector that each Layer of a Base model reads and adds to, one for each token. Knowing what a read of it returns and what a Steering vector changes lets you choose Layers, scales, and checks that you can trust.
What a read returns
A read with output_residual_stream returns the output of each Layer that you list. Layer 0 is the first decoder block, and Layer i is the output of block i, before the model's final norm. The Layers come in ascending order, each one once, in whatever order you list them.
Each Layer has one row for each token that went through the model. That is every prompt token, with the BOS token first for a text prompt, and every generated token except the last. The model samples the last token from the final forward pass, and that token never goes back through the model, so it has no row. A read therefore has usage.prompt_tokens + usage.completion_tokens - 1 rows. So a text that you already have, sent with max_tokens: 1, reads as one row for each of its tokens.
The numbers come in the model's own precision. llama-8b computes in bfloat16, and NumPy has no bfloat16 type. So the tensor comes as int16, with original_dtype set to torch.bfloat16, and each int16 value holds the bits of one bfloat16 number. A bfloat16 number is the upper 16 bits of a float32 number, so decoding shifts the bits into a float32. A conversion of the integers with astype gives wrong numbers, because it converts the values and not the bits. Decode the tensor has the helper that does it.
How a Steering vector changes it
A Steering vector adds numbers to the residual stream. Inference adds scale times your vector at each Layer in layer_indices, at every token: the prompt tokens and the generated tokens. The model then runs on from the changed stream, so the change carries through every later Layer and into the tokens that it samples.
A read of a steered Layer includes the steer. It shows the Layer's own output plus scale times your vector. A read of an earlier Layer does not change, because the steer comes after it. Vectors that steer the same Layer add up.
These rules make a steer checkable. When one Layer L is steered with norm_match: false, a comparison with a scale-0 baseline shows the steer. That comparison works at each token that both requests share, such as the prompt with max_tokens: 1 and temperature: 0. There, Layer L - 1 does not move, and Layer L moves by scale times your vector, up to the rounding of bfloat16. On generated tokens, the steered text goes another way, and the rows differ for that reason alone. Check a steer on your own vector walks through the check.
Inference adds a vector at every token, or at the positions that you name (Steer chosen positions). It adds it only where a forward pass runs: never at the last generated token.
Why compare with scale 0
A request with Steering vectors takes another path through Inference. Inference reads Activations and steers with vllm-lens, a vLLM plugin, which runs its steering code at each forward pass. The request also computes every prompt token again instead of reading vLLM's prefix cache. A plain request can differ from a steered one for these reasons alone, even when the vector adds almost nothing.
A request with scale: 0 takes the same path as the steered request, and adds nothing. So a difference between the scale-0 answer and the steered answer comes from your vector, or now and then from vLLM's batching. With temperature: 0 on both requests, sampling does not change the text between them either.
Equal texts do not show that a steer ran. A small steer can leave every token as it was. A read of the steered Layer shows whether the vector arrived.
Choose a scale
No one scale fits every Layer and every model, because the norm of the residual stream differs between Layers and between models. A vector of one length is a large change where the stream is short, and a small change where it is long. A scale that works at one Layer can be too small or too large at another.
A scale that is too large shows in the text. The text starts to repeat phrases, or it breaks into fragments. Each steered Layer's addition also stays in the stream through the later Layers. So when you steer several Layers, the changes add up, and each Layer needs a smaller scale.
That is why a search for a scale starts small. Each answer is compared with the scale-0 baseline, and the scale goes up in small steps. The search stops when the text changes as you want, before it degrades. Steer with a contrast vector steers with one vector at a few scales.
Match the norm
With norm_match: true, Inference first scales your vector to the norm of the residual stream at each token, and then multiplies it by scale. A scale of 1 then adds a vector as long as the residual stream at that token, and 0.1 adds a tenth of that length. So one scale is the same fraction of the stream at every Layer and every token, on any model. Inference does not scale the sum back to the stream's original norm.
With norm_match: false, the default, Inference adds scale times your vector as you sent it. Its effect then depends on the norm of the stream at that Layer.
Next steps
Read Activations reads and decodes the residual stream. Steer a Completion writes a Steering vector and compares it with a zero-scale baseline. Steer with a contrast vector builds a vector from two sets of prompts.