Set up model masters

After deploying CosmicAC, add a model master for each supported model so its serving parameters prefill new Managed Inference Jobs.

After you deploy CosmicAC, add a model master for each supported model. A model master holds a model's default serving parameters. CosmicAC prefills the Serving configuration from these parameters when you create a Managed Inference Job for the model. For more detail, see Model masters.

You add each model master by sending requests to the app-node API from any machine that can reach your CosmicAC base URL. This guide uses curl to send the requests from a terminal. To manage model masters, see Manage model masters.

The base URL is the address where your CosmicAC UI is reachable, such as localhost, a server IP, or a domain name. It depends on how CosmicAC was deployed and where you connect from. If you do not know it, ask your admin. CosmicAC serves the app-node API under /api, so the requests below use <base-url>/api.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
Your CosmicAC base URL.
A terminal with curl.

Steps

Create a model master for each supported model

For each model below, send a POST request to <base-url>/api/v1/model-masters. Each request uses the model's recommended configuration. For the source values, see Recommended model parameters.

Qwen3-VL-235B-A22B-Thinking-FP8

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 500,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "Qwen/Qwen3-VL-235B-A22B-Thinking-FP8",
    "runtime_image": "vLLM 0.11.2 + CUDA 12.9",
    "data_type": "Auto",
    "quantisation": "None",
    "tensor_parallel": 8,
    "gpu_memory_utilisation": 0.9,
    "max_model_length": 27000,
    "max_concurrent_sequences": 256,
    "reasoning_parser": "None",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 500,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENABLE_EXPERT_PARALLEL", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" },
        { "name": "SWAP_SPACE", "value": "0" }
      ]
    }
  }'

MiniMax M2.5

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 500,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "MiniMaxAI/MiniMax-M2.5",
    "runtime_image": "vLLM 0.11.2 + CUDA 12.9",
    "data_type": "Auto",
    "quantisation": "None",
    "tensor_parallel": 4,
    "gpu_memory_utilisation": 0.85,
    "max_model_length": 131072,
    "max_concurrent_sequences": 64,
    "reasoning_parser": "None",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 500,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENABLE_EXPERT_PARALLEL", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" },
        { "name": "SWAP_SPACE", "value": "0" },
        { "name": "SAFETENSORS_FAST_GPU", "value": "true" }
      ]
    }
  }'

Qwen2-VL-2B-Instruct

curl -X POST <base-url>/api/v1/model-masters \
  -H "Content-Type: application/json" \
  -d '{
    "job_type": "INFERENCE_VLLM",
    "base_os_image": "Ubuntu 22.04 + CUDA 12.9",
    "disk_gb": 100,
    "cuda_driver_version": "CUDA 12.9",
    "model_name": "Qwen/Qwen2-VL-2B-Instruct",
    "runtime_image": "vLLM 0.11.2 + CUDA 12.9",
    "data_type": "Auto",
    "quantisation": "None",
    "tensor_parallel": 1,
    "gpu_memory_utilisation": 0.9,
    "max_model_length": 27000,
    "max_concurrent_sequences": 128,
    "reasoning_parser": "None",
    "multimodal": true,
    "replica": 1,
    "require_auth_header": true,
    "inference_param_overrides": {
      "root_disk_size_gb": 100,
      "env": [
        { "name": "TRUST_REMOTE_CODE", "value": "true" },
        { "name": "ENFORCE_EAGER", "value": "true" },
        { "name": "SWAP_SPACE", "value": "0" }
      ]
    }
  }'

Confirm the model masters

Confirm the model masters are active:

curl "<base-url>/api/v1/model-masters?status=active"

The list shows each model master you created, with its ID. When you next create a Managed Inference Job for one of these models, CosmicAC prefills the Serving configuration with the master's values.

Set up model masters

Prerequisites

Steps

Create a model master for each supported model

Confirm the model masters

Next steps

On this page