Download a harvest from a shell script

Read the Activations of many prompts from one shell script. It submits a harvest, waits until the harvest ends, and downloads its files into a new folder. You need curl, jq, and the two shell variables of the Quickstart. Harvest Activations explains each route that the script uses.

Run the scriptCopy Anchor Link

1. Save this script as harvest.sh. Put your own prompts and Layers in the body.

   Copy Text

   #!/bin/sh
   set -eu
   
   harvest=$(curl -sS "$ACS_API_BASE/harvest" \
     -H "Authorization: Bearer $ACS_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model": "llama-8b", "prompts": ["The sky is", "Grass is"], "layers": [8, 16, 24]}')
   JOB_ID=$(echo "$harvest" | jq -r .job_id)
   if [ "$JOB_ID" = null ]; then
     echo "$harvest" >&2
     exit 1
   fi
   echo "Submitted harvest job $JOB_ID" >&2
   
   status=pending
   while [ "$status" = pending ] || [ "$status" = running ]; do
     sleep 3
     read=$(curl -sS "$ACS_API_BASE/harvest/$JOB_ID?wait=60" \
       -H "Authorization: Bearer $ACS_API_KEY")
     status=$(printf '%s' "$read" | jq -r .status)
     [ "$status" != null ] || { printf '%s\n' "$read" >&2; exit 1; }
   done
   echo "Harvest job $JOB_ID is $status" >&2
   [ "$status" = done ] || exit 1
   
   mkdir "harvest-$JOB_ID"
   cd "harvest-$JOB_ID"
   curl -sS "$ACS_API_BASE/harvest/$JOB_ID" \
     -H "Authorization: Bearer $ACS_API_KEY" \
     | jq -r '.result | ([["manifest.json", .manifest_url], ["stats.safetensors", .stats_url]] + [.shard_urls | to_entries[] | ["shard_\(("0000" + (.key | tostring))[-5:]).safetensors", .value]])[] | "url = \"\(.[1])\"\noutput = \"\(.[0])\""' \
     | curl -sSf --parallel --config -
   echo "Downloaded the files into harvest-$JOB_ID" >&2

2. Run the script:

   Copy Text

   sh harvest.sh

   The script writes its messages, such as Harvest job ... is done, to standard error. It exits with 1 when the API refuses the harvest or a read of it, and prints the answer. It also exits with 1 when the harvest ends failed or cancelled, because such a harvest has no files.

3. Read the files in the new folder harvest-<id>: manifest.json, stats.safetensors, and one shard_<n>.safetensors for each shard. Read the shards says what each file holds.

The script asks the API to wait up to 60 seconds for each change of status, and waits 3 seconds between two reads. So it sends few requests while a long harvest runs. The last curl downloads the files in parallel from the links in the harvest's result. Those links need no API key.

If the script stopsCopy Anchor Link

The harvest runs in the API and on Inference, not in the script, so it keeps running when the script stops. set -eu stops the script when a command fails, for example when curl cannot reach the API. Put the harvest's job_id, from the line Submitted harvest job ..., in JOB_ID. When the harvest is done, read it again with curl "$ACS_API_BASE/harvest/$JOB_ID" -H "Authorization: Bearer $ACS_API_KEY" for its links. Each answer has new links, so a link that expired is no problem.