Create a GPU Container Job with the CLI

Create a GPU Container Job from the CLI. Answer the prompts or pass flags, and CosmicAC creates the job.

Prerequisites

You need the following before you start:

A running CosmicAC deployment. See Installation.
The CosmicAC CLI installed and configured. See Install the CLI.

Steps

Create the job

Create the job interactively by answering prompts, or non-interactively by passing flags.

Start the interactive job setup:

cosmicac jobs create

Select GPU Container as the job type.

Set these fields:

Job name: a name to identify the job.
Tags: comma-separated labels for the job.
Location: the region where the job runs.
GPU type: the GPU to use. The CLI lists the GPUs available in your location.
GPU count: the number of GPUs for the job. One of 1, 2, 4, or 8.
Root disk size: the disk size in GB, for example 250. The minimum is 100.
Notifications: the job lifecycle events this job reports. All four are on by default, and interactive mode prompts for them with a checkbox.

An event reaches your webhook only if it's also turned on in Settings → Notifications. See What controls delivery.

GPU Container Job configuration describes each field and its CLI flag.

Confirm the job

List your jobs to confirm CosmicAC created the job:

cosmicac jobs list

The job appears in the table with its ID, name, tags, and status. Wait for it to provision. You can connect to the job once its status is running. See Connect to a GPU Container Job.

Help and troubleshooting

Job stuck in Creating or Starting

If a job stays in Creating or Starting, check the status of its KubeVirt virtual machine instance (VMI).

Find the job's container ID:
```
cosmicac jobs detail <jobId>
```
The output lists the Container ID for each container.
Find the VMI for the container.

CosmicAC creates one VMI for each container and names it <container-id>-n0. A multi-node job has one VMI per node.

From a machine with kubectl access to your Kubernetes cluster, run:
```
kubectl get vmi -n <namespace>
```
Replace <namespace> with the namespace configured in K8S_NAMESPACE.
Check the VMI status:
- If the VMI is not Running, inspect its events:
```
kubectl describe vmi <container-id>-n0 -n <namespace>
```
- If the VMI is Running but the job stays in Creating or Starting, cosmicac-wrk-agent-instance cannot reach cosmicac-wrk-server-k8s-nvidia. These two components connect directly, and some cluster network configurations can block the connection.
  
  To route the connection through a relay, see Set up a relay for CosmicAC.