Harvest Activations

A harvest reads the residual stream of a Base model at the Layers that you choose, for each token of many prompts. Inference writes it to files that you download, for example to train a sparse autoencoder or a probe.

Set ACS_API_BASE and ACS_API_KEY first, as Set up shows. A harvest runs in the background on GPUs of its own, apart from Completions. Harvests in Limits gives every number on this page. Download a harvest from a shell script does each step from one script.

Submit a harvestCopy Anchor Link

1. Send the model, the prompts, and the Layers. This harvest reads Layers 8, 16, and 24 of two prompts:

   Copy Text

   curl "$ACS_API_BASE/harvest" \
     -H "Authorization: Bearer $ACS_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}'

   The API answers 202 with the harvest. The answer has its job_id, "status": "pending", the model, n_prompts, the Layers that it reads, and reserved_tokens. tokens, started_at, and completed_at are null until the harvest starts or ends.

2. Put the harvest's job_id in a shell variable for the next commands, for example export JOB_ID=<JOB_ID> with the job_id from the answer. The Python programs below read it too.

The request takes these fields. An unknown field gets 400.

The API checks the request and the caps when you submit it. If it refuses the harvest, it creates nothing and answers the error object:

• 400 for a field that is not valid. param names the field, such as prompts, layers, or project_onto. A Layer that the model does not have gets a message such as meta-llama/Llama-3.1-8B has layers 0 to 31: it has no layer 40.
• 400 harvest_unsupported for a model that the API serves but does not harvest.
• 400 model_not_found for a model that the API does not serve.
• 413 for a request body over the harvest's own size cap.
• 429 harvest_quota_exceeded, harvest_concurrency_exceeded, or harvest_capacity_exceeded when a cap is full. The message names the cap, such as You have 3 harvests on small models queued or running, your most. Submit another when one ends, or cancel one.
• 429 budget_exceeded when your tokens cannot hold the prompts.
• 429 spend_cap when the API has paused new harvests to limit GPU spending. Its Retry-After header says when to try again.

A prompt longer than the model's context keeps its first tokens, and the rest is cut. Harvests in Limits says how many.

Check on the harvestCopy Anchor Link

Read the harvest. With wait, the API answers when the status changes, or after that many seconds:

curl "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
  -H "Authorization: Bearer $ACS_API_KEY"

Read it again until its status is done, failed, or cancelled. A harvest is in one of these states:

The times are ISO 8601 in UTC. created_at is when you submitted the harvest, and started_at is when its GPU run started. completed_at is when it was done, failed, or was canceled.

reserved_tokens is what the harvest holds while it runs, and tokens is what it was charged, once it ended. Holds of background work explains both.

When too many reads wait at once (Harvests in Limits), the API answers at once without waiting. That answer has the headers X-Acs-Longpoll: declined and Retry-After: 3. Wait 3 seconds, then read again.

Download the filesCopy Anchor Link

1. Wait until the harvest's status is done.

2. Read the harvest:

   Copy Text

   curl "$ACS_API_BASE/harvest/$JOB_ID" \
     -H "Authorization: Bearer $ACS_API_KEY"

   Its result has a link to each file, and urls_expire_at says when the links stop working:

   Field of result	What it is
   shard_urls	One link for each shard file, in order.
   stats_url	The link to stats.safetensors.
   manifest_url	The link to manifest.json.
   n_shards, layer_indices, contents	How many shards there are, the Layers in them, and activations or projections.
   tokens, bytes_written	The token positions in the files, start tokens included, and their size in bytes.
   timings	The seconds of the GPU run: load_s to load the model, forward_s to read the stream, save_s to write the files, and upload_s to upload them.

3. Download the shards and read the first one. This program needs requests, safetensors, and torch:

   Copy Text

   import os
   
   import requests
   from safetensors.torch import load_file
   
   API_BASE = os.environ["ACS_API_BASE"]
   HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
   JOB_ID = os.environ["JOB_ID"]
   
   answer = requests.get(f"{API_BASE}/harvest/{JOB_ID}", headers=HEADERS, timeout=60)
   answer.raise_for_status()
   result = answer.json()["result"]
   
   for index, url in enumerate(result["shard_urls"]):
       with requests.get(url, stream=True, timeout=600) as shard:
           shard.raise_for_status()
           with open(f"shard_{index:05d}.safetensors", "wb") as file:
               for chunk in shard.iter_content(chunk_size=1 << 20):
                   file.write(chunk)
   
   tensors = load_file("shard_00000.safetensors")
   print(tensors["prompt_0"].dtype, tuple(tensors["prompt_0"].shape))
   print(tensors["tokens_0"].tolist())

   The program prints the type and shape of the first prompt's tensor, then its token ids.

A link needs no API key, and works until urls_expire_at. Each answer of GET /v1/harvest/{job_id} for a done harvest makes new links, and the API keeps the files. Kept data gives the times.

A harvest that is not done has no result and no urls_expire_at.

Read the shardsCopy Anchor Link

Each shard holds shard_size prompts, in the order of prompts. For the prompt at index i of your prompts, a shard has two tensors:

The Layer axis follows layer_indices, in ascending order. Layer k is the output of decoder block k, before the model's final norm, as for a Completion (The residual stream). The last Layer is before the final norm too. tokens_<i> starts with the start token when add_special_tokens is true. The token ids let you match each row to its token without the tokenizer.

Avoid a second start tokenCopy Anchor Link

The tokenizer adds the model's start token, such as <|begin_of_text|>, in front of each prompt. If your text already starts with that token, the prompt gets two, and the API does not warn you. Then tokens_<i> is one longer than your own tokenization and repeats its first id.

Send "add_special_tokens": false for text that already starts with the start token. The tokenizer then reads each prompt as it is.

Read the manifest and the statisticsCopy Anchor Link

manifest.json describes the whole harvest. Inference writes it last, so a harvest whose manifest exists has every file. It has the old platform's file layout, version 2, so programs that read those files read these too. Its main keys:

stats.safetensors holds mean and std (float32, [Layers, hidden]), and layer_indices. The statistics count every token position but the first of each prompt. With add_special_tokens: true, that first position is the start token, whose vectors are outliers.

This program reads both. It needs requests, safetensors, and torch:

import os

import requests
from safetensors.torch import load

API_BASE = os.environ["ACS_API_BASE"]
HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
JOB_ID = os.environ["JOB_ID"]

answer = requests.get(f"{API_BASE}/harvest/{JOB_ID}", headers=HEADERS, timeout=60)
answer.raise_for_status()
result = answer.json()["result"]

manifest = requests.get(result["manifest_url"], timeout=60).json()
print(manifest["layer_indices"], manifest["residual_stream_convention"])

stats = load(requests.get(result["stats_url"], timeout=60).content)
print(stats["mean"].shape, stats["std"].shape)

The manifest holds your prompts' text, so keep it as private as the prompts.

Project onto directionsCopy Anchor Link

With project_onto, Inference writes the residual stream projected onto your directions, in place of the stream itself. The files are then much smaller. Inference scales each direction to length 1 first. Each direction has the model's capabilities.hidden_size, from List the models.

1. Put your directions, such as a probe's weights, in a 2-D array of [directions, hidden_size].

2. Send its raw bytes, little-endian, in base64 in data, with dtype and shape. This program submits 8 random directions of llama-8b:

   Copy Text

   import base64
   import os
   
   import numpy as np
   import requests
   
   API_BASE = os.environ["ACS_API_BASE"]
   HEADERS = {"Authorization": f"Bearer {os.environ['ACS_API_KEY']}"}
   
   directions = np.random.default_rng(0).standard_normal((8, 4096)).astype("<f4")
   
   body = {
       "model": "llama-8b",
       "prompts": ["The sky is", "Grass is"],
       "layers": [8, 16, 24],
       "project_onto": {
           "data": base64.b64encode(directions.tobytes()).decode("ascii"),
           "dtype": "float32",
           "shape": list(directions.shape),
           "compression": "none",
       },
   }
   answer = requests.post(f"{API_BASE}/harvest", headers=HEADERS, json=body, timeout=60)
   answer.raise_for_status()
   print(answer.json()["job_id"])

3. Put the printed job_id in JOB_ID, and download the files as before.

dtype is float32, float16, or bfloat16. For float16, use astype("<f2"). compression takes only none, and you can leave it out. Each prompt_<i> is then float32, [Layers, tokens, directions]. The manifest has contents: "projections" and a projection key, and the statistics describe the projections.

The API refuses directions that do not fit before any GPU starts. Directions of another hidden size get 400 with param project_onto. These get 400 too, each with its message:

Cancel a harvestCopy Anchor Link

You can cancel a harvest that is pending or running, as it is while Inference loads the model.

1. Submit the harvest:

   Copy Text

   curl "$ACS_API_BASE/harvest" \
     -H "Authorization: Bearer $ACS_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model": "llama-8b", "prompts": ["The sky is"], "layers": "all"}'

2. Put the new job_id in JOB_ID, then cancel the harvest:

   Copy Text

   curl -X DELETE "$ACS_API_BASE/harvest/$JOB_ID" \
     -H "Authorization: Bearer $ACS_API_KEY"

   The API answers 200 with the harvest, "status": "cancelled".

The API stops the harvest's GPU run, also one that is still starting. A harvest that you cancel while it is running and its GPU run started is charged the tokens that it holds, as tokens shows. Any other cancel costs nothing:

• Before its GPU run starts.
• While it waits pending again after an API restart.
• After the API process that ran it stopped without warning.

A second cancel answers the canceled harvest again. A cancel of a done or failed harvest gets 409 harvest_not_cancellable, and a job_id that you do not have gets 404 harvest_job_not_found.

Handle a failed harvestCopy Anchor Link

A failed harvest is charged no tokens. Its error is a message that says why:

If the API restarts while your harvest runs, the harvest goes back to pending, and its GPU run goes on. The API then follows the same GPU run, and charges the tokens once. When the API restarts says how.

The API reference lists every field of each harvest route.