Checkpoints and failover

Cheap GPU capacity is cheap partly because it can be taken back. Jobs are built so that this costs you time rather than the whole run.

What we promise, exactly

The boundary

You stop a run, or stop a pod: we capture the state on the way out and you can restore it anywhere. A provider takes the machine away: we recover to the last scheduled checkpoint, not to the instant it vanished.

Everything below is about the second case.

What happens when a machine goes

A provider's own instance list stops carrying the machine. That, not a quiet agent, is what makes it gone.
The attempt is closed as provider_killed.
The run goes back to placing, excluding the provider that just lost it from the next attempt. A machine we lost is not charged against the attempt your own code is given, so losing boxes does not use up your retry.
A new machine is rented automatically, without you doing anything. The new attempt restores the last checkpoint before running your command if the job checkpoints, or starts from the beginning if it does not.
When there are two attempts to compare, the job page shows the handoff as its own event, with what was lost and what was resumed from. It is never folded away.

Why a quiet machine is not a dead one

If our agent stops answering, we do not immediately start a second attempt. We wait, and then we ask the provider to stop the machine, and only a confirmed stop counts as gone.

That caution is deliberate: two machines running one run would bill you twice and race on the outputs, where the loser silently overwrites the winner. If we cannot confirm the stop, the run ends as provider_unreachable rather than quietly starting a second machine beside one that may still be working.

Setting up checkpoints

Give the job the paths to save and how often:

Field	What it is
Paths	Directories your code writes recoverable state into
Every	How often we snapshot them, in seconds

Your code does not have to know about any of this to benefit. But if it should behave differently on a resume, we tell it:

Variable	Value
`AQ_ATTEMPT`	`1`, `2`, … which attempt this is
`AQ_RESUMED_FROM_CHECKPOINT`	`1` if a checkpoint was actually restored, else `0`
`AQ_RUN_ID`	The run this is

AQ_RESUMED_FROM_CHECKPOINT is 0 on a retry that had nothing to resume from, because to your script that is the same situation as a fresh start.

Pick an interval you would be willing to lose. A five-minute interval on a four-hour job means a lost machine costs you at most five minutes of compute plus the time to find another machine.

On-demand-only, and why the default flips

Preemptible machines are cheaper and likelier to be taken back. So the default follows your checkpoint policy rather than being fixed:

Your job	Default	Why
Has checkpoints	On-demand off	A lost machine costs you one interval, so the cheaper pool is worth it
No checkpoints	On-demand on	A lost machine costs you the entire run

You can override it either way. The console states which default applied and why, because a default you cannot see is one you cannot disagree with.

What failover cannot do

A single-provider allowlist has nowhere to fail over to. If you restrict a job to one provider, losing that machine ends the run. The console warns when you do this.
A pinned job cannot fail over. It runs on your machine, and we will not rent another on your behalf.
Attempts are bounded, on two separate counters. Your own code is run once: if it exits on its own terms we do not re-run it, because code that fails on one machine almost always fails the same way on the next. Machine losses are counted separately and generously, so a run can move across many machines and still have its single code attempt intact. When either bound is reached the run ends as attempts_exhausted rather than retrying forever at your expense.