Managed Inference Job

A Managed Inference Job runs an open-source model inside a KubeVirt virtual machine instance (VMI) on your cluster and serves it behind an OpenAI-compatible API. It runs one of two runtimes, vLLM for language models or Parakeet for speech-to-text. You send requests the same way you would to any OpenAI-compatible endpoint.

When to use one

A Managed Inference Job fits when you want to call a model over an API and let CosmicAC run it for you. You pick an open-source model, and CosmicAC serves it.

If you instead want direct control of a GPU to run your code, a GPU Container Job is the better fit. It gives you a machine and a shell, and you set up the environment yourself.

What you get

An OpenAI-compatible API — your existing clients and SDKs work without changes.
Open-source models — served on your cluster with vLLM or Parakeet.
Managed serving — CosmicAC provisions and runs the model server, so you don't set up the runtime or GPU environment yourself.

Runtimes

A Managed Inference Job runs one of two runtimes.

vLLM — serves open-source language models behind an OpenAI-compatible chat endpoint.
Parakeet — serves the NVIDIA Parakeet speech-to-text model nvidia/parakeet-tdt-0.6b-v3 behind an OpenAI-compatible transcription endpoint.

Supported models

A vLLM job serves one of these supported models, each with a model master of recommended serving parameters and hardware.

Qwen3-VL-235B-A22B-Thinking-FP8 — Qwen/Qwen3-VL-235B-A22B-Thinking-FP8
MiniMax M2.5 — MiniMaxAI/MiniMax-M2.5
Qwen2-VL-2B-Instruct — Qwen/Qwen2-VL-2B-Instruct

For each model's serving parameters and hardware, see Recommended model parameters.

A Parakeet job serves one speech-to-text model, nvidia/parakeet-tdt-0.6b-v3.

Model masters

A model master stores what CosmicAC knows about a model, such as its runtime image and default serving parameters. It saves you from configuring every field each time you serve the model.

When you select a model while creating a Managed Inference Job, its model master data prefills the Serving configuration. You can adjust the prefilled values before you submit the job.

You manage model masters through the app-node API. After you deploy CosmicAC on your host machine, add a model master for each supported model. See Set up model masters. To update, add, or archive them, see Manage model masters. For the values to use, see Recommended model parameters.

How it works

A Managed Inference Job moves through a short lifecycle.

You create the job from the CLI or the web interface.
CosmicAC schedules it on a GPU node in your cluster.
CosmicAC provisions a VMI that serves the model with the chosen runtime.
When the model is serving, you call it through the OpenAI-compatible endpoint. The endpoint authenticates your requests and routes them to the running model.

Restarting the job replaces its VMI but keeps its storage and resources. Deleting the job removes its VMI, resources, and storage.

For how a request flows through the CosmicAC components, see Architecture.

How you connect

You call the model in two ways.

An OpenAI-compatible client — send requests over HTTP.
cosmicac-cli — run inference directly from the terminal.

Both authenticate with an API key. See Create an API key.

To connect a client to a vLLM endpoint, see Connect to a Managed Inference endpoint (vLLM). For a Parakeet endpoint, see Transcribe audio with a Parakeet endpoint.