Models
Each request names a Base model in model, by the model's id: llama-8b, llama-405b, or trinity-truebase, the ids of the old platform. A request without model runs on llama-8b. Each answer's model names the weights that ran it, such as meta-llama/Llama-3.1-8B. GET /v1/models lists the models that the API serves, with each model's facts and the limits that the API enforces on a request to it.
List the models
curl "$ACS_API_BASE/models" \
-H "Authorization: Bearer $ACS_API_KEY"The curl command prints the list:
{
"object": "list",
"data": [
{
"id": "llama-8b",
"object": "model",
"owned_by": "acs",
"served_model_name": "meta-llama/Llama-3.1-8B",
"gpu_shape": "1xL40S",
"status": "live",
"default": true,
"capabilities": {
"max_model_len": 131072,
"max_logprobs": 100,
"max_output_work_tokens": 32000,
"max_inflight_per_key": 16,
"max_queued_per_key": 64,
"rate_limit_per_minute": 600,
"logprobs": true,
"prompt_logprobs": true,
"prompt_logprobs_full_vocab": true,
"full_vocab_max_prompt_tokens": 256,
"activations": true,
"max_steering_vectors": 8,
"max_activation_prompt_tokens": 256,
"n_layers": 32,
"hidden_size": 4096,
"chat_template": null
}
}
]
}The API answers the list from its own settings and asks no Inference, so a list never starts a GPU.
Every User gets the same list, and may run each model in it.
Model fields
| Field | Meaning |
|---|---|
id | The Model id to send as model. |
object | Always "model". |
owned_by | Always "acs". |
served_model_name | The weights that Inference serves, which each answer's model names. |
gpu_shape | The GPUs of one container and their type, such as 1xL40S or 8xH200. |
status | Always "live". It does not say whether the model is Warm. |
default | true on the model that a request without model runs on. Set model on each request, because the default can change. |
capabilities | The facts of the model and the limits of a request to it, with the keys below. |
Keys of capabilities
| Key | Meaning |
|---|---|
max_model_len | The most tokens of prompt and generated text together. |
max_logprobs | The most top tokens of logprobs, and of a positive prompt_logprobs. |
max_output_work_tokens | The most generated tokens of one request, counted as prompts × n × max_tokens. |
max_inflight_per_key | Completions that run at once, for each API key. |
max_queued_per_key | Completions that wait for a place, for each API key. |
rate_limit_per_minute | Requests a minute that start work, for each API key, and as many other requests. |
logprobs, prompt_logprobs, prompt_logprobs_full_vocab | Always true: the model takes logprobs, prompt_logprobs, and prompt_logprobs: -1. |
full_vocab_max_prompt_tokens | The most prompt tokens in all × n with prompt_logprobs: -1. With output_residual_stream, the most is 16. |
activations | Always true: the model takes output_residual_stream and apply_steering_vectors. |
max_steering_vectors | The most vectors in apply_steering_vectors. |
max_activation_prompt_tokens | The most tokens, of the prompt and max_tokens, of a capture of every Layer. A capture of fewer Layers can take more tokens. |
n_layers | The number of the model's Layers, numbered from 0. Send Layers in output_residual_stream and layer_indices. |
hidden_size | The numbers in one vector of the residual stream. A Steering vector's rows have this length. |
chat_template | Always null, because the API applies no chat template. |
Size caps explains the caps of one request, and Limits lists every limit with the error past it.
Read one model
GET /v1/models/{model} answers one model of the list, with the same fields, as OpenAI's retrieve route does:
curl "$ACS_API_BASE/models/llama-8b" \
-H "Authorization: Bearer $ACS_API_KEY"The path takes the served_model_name too, with or without its slash encoded: /v1/models/meta-llama/Llama-3.1-8B answers the same model. A model that the list does not have gets 400 with the code model_not_found.
Check whether a model is Warm
GET /v1/models/{model}/status says whether the model's next request starts at once, or which stage its start has reached:
curl "$ACS_API_BASE/models/llama-405b/status" \
-H "Authorization: Bearer $ACS_API_KEY"{
"id": "llama-405b",
"object": "model.status",
"warm": false,
"boot": {
"stage": "weights_loading",
"stage_label": "Loading model weights",
"stage_age_s": 402
},
"activation_boot": {
"stage": "weights_loading",
"stage_label": "Loading model weights",
"stage_age_s": 402
}
}| Field | Meaning |
|---|---|
id | The Model id. |
object | Always "model.status". |
warm | true when a container of the model serves, so its next request starts at once. false when the next request waits for a start. |
boot | The stage of the start that runs, or null when none runs. Its fields are in the next table. |
activation_boot | The same as boot. |
boot field | Meaning |
|---|---|
stage | waiting_for_gpu: a request waits, and no container of the model runs yet. container_started: a container sets up the model's server. weights_loading: it loads the weights onto its GPUs, or restores its memory snapshot. weights_loaded: it prepares its memory and kernels. |
stage_label | The stage in words, such as Loading model weights. |
stage_age_s | How long the start has run, in seconds. For waiting_for_gpu, it counts from the waiting request. For the other stages, it counts from the container's start. |
A model can be Warm and start a second container at the same time. Inference reports each container's start, and the API reads that report, so this route never starts a GPU. If the report is missing, warm is true for 5 minutes after Inference answered a Completion of the model, and boot is null unless a request waits. A model that the API does not serve gets 400 with the code model_not_found. Warm and cold models explains why a cold model's first request waits.
Served models
The API serves three Base models:
model | What it is |
|---|---|
llama-8b | meta-llama/Llama-3.1-8B, Meta's Llama 3.1 8B Base model, on one L40S GPU. Its context length is 131,072 tokens, the whole context of Llama 3.1. It has 32 Layers, 0 to 31, and a hidden size of 4,096. Its tokenizer adds the BOS token <|begin_of_text|> in front of a text prompt. |
llama-405b | meta-llama/Llama-3.1-405B, Meta's Llama 3.1 405B Base model, on eight H200 GPUs. Its context length is 131,072 tokens, the whole context of Llama 3.1. It has 126 Layers, 0 to 125, and a hidden size of 16,384. Its tokenizer adds the BOS token <|begin_of_text|> in front of a text prompt. |
trinity-truebase | arcee-ai/Trinity-Large-TrueBase, Arcee's Trinity Large TrueBase, a Base model with a mixture of experts, on eight H200 GPUs. Its context length is 8,192 tokens, the model's own. It has 60 Layers, 0 to 59, and a hidden size of 3,072. |
Llama 3.1 is licensed under Meta's Llama 3.1 Community License, and each use of it must follow Meta's Acceptable Use Policy. Built with Llama.
A model that the API does not serve gets 400 with the code model_not_found, as Unknown model shows. A prompt and max_tokens that together are longer than max_model_len get 400 with the code context_length_exceeded, as Prompt too long shows.
Harvests on each model
Each served model harvests. A harvest runs on GPUs of its own, apart from the model's Completions:
model | GPUs of one harvest | Harvest caps |
|---|---|---|
llama-8b | One L40S GPU | The caps on Llama 3.1 8B |
llama-405b | Eight H200 GPUs | The caps on Llama 3.1 405B and Trinity Large, which the two models share |
trinity-truebase | Eight H200 GPUs | The caps on Llama 3.1 405B and Trinity Large, which the two models share |
Harvests in Limits gives each cap. A GPU that ran a harvest stays up 20 minutes after it. A harvest of the same model in that time starts without loading the model, and its timings.load_s is 0. A harvest that needs a new GPU is running while Inference loads the model, and timings.load_s counts that time.