Jobs

A job is one submission: a source, a command, and the hardware it needs. Submitting it starts a machine and runs your command immediately, no separate step to trigger it. The job has a status: queued, starting, running, succeeded, failed, or cancelled. When it reaches succeeded or failed, that job is done. Want to run it again? Re-run makes a new job with a fresh id, a fresh status, and its own logs.

Under the hood, one submission can still move across machines: if a provider takes the box away mid-job, we retry on a different one, and if the job checkpoints the new machine picks up where the last one left off. That is what keeps a job's status honest even after a lost machine, not a second job. See Checkpoints and failover for what happens when a machine is lost.

Jobs live under Jobs in the console sidebar, and run your command to completion. Looking for a workload that stays up and answers calls instead? See Endpoints.

Where a job's work comes from

Either a saved pod version, restored onto the machine before your command runs, or a public Docker image.

Image jobs on container providers need two things in the image

On our container-based (docker pool) providers we cannot run a container inside your container, so we start yours and fetch our agent into it. That needs sh and either curl or wget present in your image. Without them the job ends as image_bootstrap_failed, and the retry avoids those providers. It does not count against the one attempt your own code is given. Hotaisle refuses an image-source job outright rather than attempting this boot at all, so an image job never lands there.

Data your command needs comes from the image, a volume, or a URL the command fetches itself. A job has no separate input form: it is a literal command, the same as on every other platform that runs jobs.

Multiple GPUs

A job can use 1, 2, 4 or 8 GPUs on one machine. The console's new-job page has a GPU count picker next to the GPU model picker. --gpus is for an --image job; a pod-version job runs on the pod's own saved hardware and does not take --gpus:

aq job run --image <ref> --gpu-model "H100" --gpus 2 -- torchrun --nproc_per_node=2 train.py

The price shown is for the whole machine, not per GPU, so an 8-GPU node's rate already covers all 8 cards. Your command has to use the GPUs itself, which is what --nproc_per_node does above.

This is one machine with several GPUs, not several machines: a job does not span multiple nodes.

What it costs

The console shows one price: the rate of the machine your job will actually be placed on, per minute, for the GPU you picked. You are billed by the second the machine is up, which is a little longer than your own code runs: the window opens when the machine comes up and closes when we release it, so it covers the tail of the machine's setup and its teardown as well as your run itself.

That price can move. If the machine dies we place the retry on another one, which may be cheaper or dearer than the first, and a checkpointed job is also moved to a cheaper machine of the same type when one appears. The rate you see is the one we book at, not a rate we hold for the life of the job.

There is deliberately no worst-case total. One used to be shown, computed as time limit × max attempts × machines × the dearest machine you allowed, and every factor in it was misleading: the dearest matching machine was a whole multi-GPU node your job would never rent at that GPU count, and the attempt count governed re-runs of your code rather than the machine losses that actually spend wall-clock time. A total we cannot honour is worse than no total.

Your time limit still bounds a single attempt of your code.

When a job can't start, or doesn't finish clean

A job that never got a machine ends as unservable, never as plain failed. The two mean different things and we keep them apart:

Reason	What actually happened
`unservable`	We could not get the job a machine. Your code never ran.
`failed`	Your code ran and exited non-zero, or every attempt lost its machine.
`provider_unreachable`	We lost the machine and could not confirm the provider stopped it. See below.
`attempts_exhausted`	The job moved across machines as many times as it's allowed to and still lost.

Checkpoints and failover: what survives a lost machine
The CLI: aq job
Endpoints: deploy a workload that scales to zero and answers calls

Where a job's work comes from

Multiple GPUs

What it costs

When a job can't start, or doesn't finish clean

Next

On this page