Your first steered Completion

In this tutorial, you steer a Base model for the first time. First, you get access and make an API key. Then you run a Completion, read the residual stream at Layer 1, and add a Steering vector at that Layer. At the end, you will see the residual stream move by your vector.

You need curl, Python 3.14 (for compression.zstd), NumPy, and the openai SDK. Each output is one sample that meta-llama/Llama-3.1-8B wrote.

Get accessCopy Anchor Link

An admin approves each Application to use ACS Infra.

1. Open the Apply form on the ACS Infra website.
2. Fill in the form. Email, Name, Organization, and Planned usage are required.
3. Read the platform rules, and select the box I agree to these rules.
4. Select Apply for access. The page shows "Application received".
5. Wait for the email. When an admin approves your Application, you become a User with a monthly allowance of 5,000,000 tokens. The email has the subject "Your ACS Infra application is approved".
6. In the email, select Choose your password. The link works once, for 7 days.
7. The App opens the page "Set your password". Enter a password of at least 8 characters, and select Set the password. The App shows "Your password is set."
8. Select Sign in, and sign in with your email and your password.
9. Open Usage in the App. Monthly allowance shows the tokens that you have left this month.

Get access has the details, such as signing in with GitHub, Google, or Hugging Face when the App shows their buttons.

Make an API keyCopy Anchor Link

1. In the App, open API Keys.

2. Select New key, enter a name, such as laptop, and select Create key.

3. The App shows the dialog "Copy your key". Select Copy. The App shows the key only once.

4. Put the key and the API's address in shell variables:

   Copy Text

   export ACS_API_KEY=sk-acs-...
   export ACS_API_BASE=https://api.v2.infra.acsresearch.org/v1

Now check the key. List the Base models with it:

curl "$ACS_API_BASE/models" \
  -H "Authorization: Bearer $ACS_API_KEY"

{
  "object": "list",
  "data": [
    {
      "id": "llama-8b",
      "object": "model",
      "owned_by": "acs",
      "served_model_name": "meta-llama/Llama-3.1-8B",
      "gpu_shape": "1xL40S",
      "status": "live",
      "default": true,
      "capabilities": {
        "max_model_len": 131072,
        "max_logprobs": 100,
        "max_output_work_tokens": 32000,
        "max_inflight_per_key": 16,
        "max_queued_per_key": 64,
        "rate_limit_per_minute": 600,
        "logprobs": true,
        "prompt_logprobs": true,
        "prompt_logprobs_full_vocab": true,
        "full_vocab_max_prompt_tokens": 256,
        "activations": true,
        "max_steering_vectors": 8,
        "max_activation_prompt_tokens": 256,
        "n_layers": 32,
        "hidden_size": 4096,
        "chat_template": null
      }
    }
  ]
}

Use the id of a Base model as model in the next steps. This tutorial uses llama-8b, the weights meta-llama/Llama-3.1-8B. Its capabilities give 32 Layers in n_layers and a hidden size of 4096 in hidden_size.

Run a CompletionCopy Anchor Link

Send the prompt "The sky is", and ask for 2 tokens. Set temperature to 0, so that the model writes its likeliest token each time.

curl "$ACS_API_BASE/completions" \
  -H "Authorization: Bearer $ACS_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "llama-8b", "prompt": "The sky is", "max_tokens": 2, "temperature": 0}'

The curl command prints the Completion:

{
  "id": "cmpl-824562de43d1b203",
  "object": "text_completion",
  "created": 1791780344,
  "model": "meta-llama/Llama-3.1-8B",
  "choices": [
    {
      "index": 0,
      "text": " the limit",
      "logprobs": null,
      "finish_reason": "length",
      "prompt_logprobs": null
    }
  ],
  "usage": { "prompt_tokens": 4, "completion_tokens": 2, "total_tokens": 6 }
}

The Python program prints the text that the Base model wrote:

' the limit'

The API took the tokens of usage from your monthly allowance.

Capture ActivationsCopy Anchor Link

Ask for the residual stream at Layer 1 with output_residual_stream. The openai SDK does not know the field, so send it in extra_body. The API sends the numbers compressed, and Llama's numbers are bfloat16. This program decodes them with base64, zstd, and NumPy, and prints their shape and the first 4 numbers of the first token:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    temperature=0,
    extra_body={"output_residual_stream": [1]},
)
residual_stream = decode(completion.activations["residual_stream"])
print(residual_stream.shape)
print(residual_stream[0, 0, :4].tolist())

(1, 5, 4096)
[0.0145263671875, 0.00982666015625, -0.0791015625, -0.05419921875]

The shape is 1 Layer, then 5 rows, then 4096 numbers in each row, the hidden size. The 5 rows are the 4 prompt tokens, with <|begin_of_text|> first, and the first of the 2 generated tokens. The last generated token never goes back through the model. Decode the tensor explains each field. Keep the first 4 numbers for the next step.

Steer Layer 1Copy Anchor Link

Add a Steering vector at Layer 1 with apply_steering_vectors, and read Layer 1 in the same request. The vector has one number for each of the 4096 dimensions. This one is [1, 2, 3] and then zeros, with a scale of 2:

import base64
import os
from compression import zstd

import numpy as np
from openai import OpenAI

def decode(tensor):
    raw = zstd.decompress(base64.b64decode(tensor["data"]))
    array = np.frombuffer(raw, dtype=tensor["dtype"]).reshape(tensor["shape"])
    if tensor["original_dtype"] == "torch.bfloat16":
        array = (array.view(np.uint16).astype(np.uint32) << 16).view(np.float32)
    return array

vector = np.zeros(4096)
vector[:3] = [1, 2, 3]

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
completion = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    temperature=0,
    extra_body={
        "output_residual_stream": [1],
        "apply_steering_vectors": [
            {"activations": [vector.tolist()], "layer_indices": [1], "scale": 2}
        ],
    },
)
residual_stream = decode(completion.activations["residual_stream"])
print(repr(completion.choices[0].text))
print(residual_stream[0, 0, :4].tolist())

' a '
[2.015625, 4.0, 5.9375, -0.05419921875]

The first 3 numbers moved by 2 * [1, 2, 3], that is by 2, 4, and 6, within bfloat16's precision. The fourth number did not move. You steered the residual stream at Layer 1.

Compare with scale 0Copy Anchor Link

To see what the steer changed in the text, compare it with the same request at scale: 0. Do not compare it with the Completion from Run a Completion. A request with a Steering vector takes another path through Inference, and its text can differ from a plain request's for that alone. A scale of 0 adds nothing, so the two requests differ only in scale. This program sends the baseline and prints its text:

import os

import numpy as np
from openai import OpenAI

vector = np.zeros(4096)
vector[:3] = [1, 2, 3]

client = OpenAI(base_url=os.environ["ACS_API_BASE"], api_key=os.environ["ACS_API_KEY"])
baseline = client.completions.create(
    model="llama-8b",
    prompt="The sky is",
    max_tokens=2,
    temperature=0,
    extra_body={
        "output_residual_stream": [1],
        "apply_steering_vectors": [
            {"activations": [vector.tolist()], "layer_indices": [1], "scale": 0}
        ],
    },
)
print(repr(baseline.choices[0].text))

Compare its text with the steered text. A difference comes from your vector, or now and then from vLLM's batching (Why repeats differ). If the two texts are equal, the numbers that moved by 2, 4, and 6 still show that the steer ran. Choose a scale helps you find a scale that changes the text in the way you want.

Next stepsCopy Anchor Link

Read Activations explains the encoding of the residual stream, and the checks to run before you use it. Steer a Completion gives every field of a Steering vector, and shows how to check a steer. Steer with a contrast vector builds a Steering vector from two sets of prompts. Sample many rollouts is the next lesson.