Checkpoints and failover

Cheap GPU capacity is cheap partly because it can be taken back. Jobs are built so that this costs you time rather than the whole run.

What we promise, exactly

The boundary

You stop a run, or pause a setup: we capture the state on the way out and you can restore it anywhere. A provider takes the machine away: we recover to the last scheduled checkpoint, not to the instant it vanished.

Everything below is about the second case.

What happens when a machine goes

A provider's own instance list stops carrying the machine. That, not a quiet agent, is what makes it gone.
The attempt is closed as provider_killed.
If the job has attempts left, the run goes back to placing, excluding the provider that just lost it from the next attempt.

Automatic re-placement is not live yet

Steps 3 and 4 describe the mechanism as designed and as it is being finished. Today, a run that goes back to placing after a lost machine does not yet pick up a new one on its own: getting from "excluded that provider" to "running again elsewhere" is still being wired up. Do not rely on a job finishing itself through a lost machine right now; a run stuck in that state needs your own intervention, not ours.

Once a new attempt does start (by hand today, automatically once the piece above ships), it restores the last checkpoint before running your command if the job checkpoints, or starts from the beginning if it does not.
When there are two attempts to compare, the run page shows the handoff as its own event, with what was lost and what was resumed from. It is never folded away.

Why a quiet machine is not a dead one

If our agent stops answering, we do not immediately start a second attempt. We wait, and then we ask the provider to stop the machine, and only a confirmed stop counts as gone.

That caution is deliberate: two machines running one run would bill you twice and race on the outputs, where the loser silently overwrites the winner. If we cannot confirm the stop, the run ends as provider_unreachable rather than quietly starting a second machine beside one that may still be working.

Setting up checkpoints

Give the job the paths to save and how often:

Field	What it is
Paths	Directories your code writes recoverable state into
Every	How often we snapshot them, in seconds

Your code does not have to know about any of this to benefit. But if it should behave differently on a resume, we tell it:

Variable	Value
`AQ_ATTEMPT`	`1`, `2`, … which attempt this is
`AQ_RESUMED_FROM_CHECKPOINT`	`1` if a checkpoint was actually restored, else `0`
`AQ_RUN_ID`	The run this is

AQ_RESUMED_FROM_CHECKPOINT is 0 on a retry that had nothing to resume from, because to your script that is the same situation as a fresh start.

Pick an interval you would be willing to lose. A five-minute interval on a four-hour job means a lost machine costs you at most five minutes of compute plus the time to find another machine.

Datacenter-only, and why the default flips

Community machines are cheaper and likelier to be taken back. So the default follows your checkpoint policy rather than being fixed:

Your job	Default	Why
Has checkpoints	Datacenter off	A lost machine costs you one interval, so the cheaper pool is worth it
No checkpoints	Datacenter on	A lost machine costs you the entire run

You can override it either way. The console states which default applied and why, because a default you cannot see is one you cannot disagree with.

What failover cannot do

A single-provider allowlist has nowhere to fail over to. If you restrict a job to one provider, losing that machine ends the run. The console warns when you do this.
A pinned job cannot fail over. It runs on your machine, and we will not rent another on your behalf.
Attempts are bounded. When they run out the run ends as attempts_exhausted rather than retrying forever at your expense.