Jobs

Run GPU work that takes minutes or hours, and keep it alive when a provider takes the machine away.

A job is a piece of GPU work with a source, a command, the hardware it needs, and a bound on what it may cost. Each time it runs you get a run, and each run is served by one or more attempts, one per machine.

That last part is the point, and it is what the rest of a job is built toward: when a provider takes the machine away mid-run, we detect the loss and queue the run to retry on a different provider, excluding the one that just lost it, and if the job checkpoints the next attempt picks up from the last one. Automatically starting that next attempt is still being finished, so today the run does not yet resume on its own once the machine is gone. See Checkpoints and failover for the exact current behavior.

Jobs live under Jobs in the console sidebar.

Two shapes, one object

The entrypoint decides which shape a job has. There is no setting to pick.

Entrypoint	Shape	What it does
`command`	Batch	Runs your command to completion. One run per machine. This is the default.
`http`, `comfyui`	Service	Serves a process behind a token. Several runs share a machine.

Service jobs behave exactly as endpoints did before. If you had an endpoint, you have a service job now, with the same address and the same tokens.

Where a job's work comes from

Either a saved setup version, restored onto the machine before your command runs, or a public Docker image.

Image jobs on container providers need two things in the image

On our container-based (docker pool) providers we cannot run a container inside your container, so we start yours and fetch our agent into it. That needs sh and either curl or wget present in your image. Without them the attempt ends with image_bootstrap_failed and the retry avoids those providers rather than burning your remaining attempts. Hotaisle refuses an image-source job outright rather than attempting this boot at all, so an image job never lands there.

What it costs, and what bounds it

Every job has a bound whether you set one or not, and it is expressed in the unit we actually bill: wall-clock time.

worst case for one run  =  time limit  ×  max attempts  ×  machines at once  ×  the
                           dearest machine you allowed

The console shows that figure next to the estimate, and it is a ceiling, not a forecast: it assumes every attempt runs the whole time limit on the most expensive machine you permitted.

You can add an optional monthly budget on top. When the month's measured spend reaches it, new runs stop with budget_exhausted instead of starting.

Why there is no per-run dollar cap

Endpoints used to ask for one and it was removed on purpose: the same amount of money bought 13 cold calls or 200 warm ones, so it was a limit you could not translate into your own units. A wall-clock bound translates. A budget over a metered unit translates. A per-run dollar ceiling still would not, and it is not coming back.

When we cannot start a run

Runs that never got a machine end as unservable, never as failed. The two mean different things and we keep them apart:

Reason	What actually happened
`unservable`	We could not get you a machine. Your code never ran.
`failed`	Your code ran and exited non-zero, or every attempt lost its machine.
`budget_exhausted`	The job is over its monthly budget.
`cost_unreadable`	We could not read what this job has spent, so we did not start a machine.
`provider_unreachable`	We lost the machine and could not confirm the provider stopped it. See below.

cost_unreadable is deliberately not folded into budget_exhausted. Telling you that you overspent when in fact our billing read failed would be worse than saying nothing.

Checkpoints and failover: what survives a lost machine
Triggers: schedules, tokens and webhooks
The CLI: aq job

Two shapes, one object

Where a job's work comes from

What it costs, and what bounds it

When we cannot start a run

Next

On this page