> ## Documentation Index
> Fetch the complete documentation index at: https://docs.freesolo.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Spot training

> Opt into Spot for SFT and GRPO, with automatic recovery checkpoints.

Flash uses Modal for ordinary training. With Flash 1.4.22, SFT and GRPO can
explicitly select Spot when the control plane has that provider enabled.
Spot capacity can be interrupted; choose it when you can tolerate queueing and
repeated work after recovery. Custom training remains on one Modal GPU.

## Enable Spot

Add this to an otherwise valid SFT or GRPO config:

```toml theme={null}
[gpu]
spot = true
```

`provider = "spot"` or `providers = ["spot"]` also selects Spot. Use one form;
do not combine `provider` and `providers`, mix Spot with Modal, or set
`spot = false` alongside a Spot provider. An unavailable Spot provider does not
silently fall back to Modal. Managed users need no local provider credentials;
[self-hosted operators](https://docs.freesolo.co/guides/self-hosting.md#spot-training) configure them
on the control plane.

Preview before submission:

```bash theme={null}
flash train config.toml --dry-run
flash train config.toml --cost
flash train config.toml --background
flash runs status <run-id> --follow --json
```

## GPU shapes and queueing

Spot supports **H100, H200, and B200**, in one- or eight-card shapes. Omit
`type` and `count` for automatic selection within the provider. A GPU type
without a count is a single-card constraint. To permit an eight-card allocation:

```toml theme={null}
[gpu]
spot = true
type = ["H100", "H200"]
count = 8
```

`count` is a ceiling, so this can still select one card when it fits. A ceiling
of two or four does not create a two- or four-card Spot shape. Model memory,
training batch geometry, provider quota, and live capacity still determine
whether the run is admissible.

Flash checks a fresh provider price before provisioning. The configured entry
thresholds are strictly below $1.00/GPU-hour for H100/H200 and $4.00/GPU-hour for
B200. Running-instance price caps are $1.50, $2.00, and \$5.00 respectively.
These are allocation thresholds, **not customer prices or capacity guarantees**.
Use your authenticated quote for the amount you accept. A run can remain
`queued` while price or capacity is unsuitable; no training GPU is provisioned
just to wait for a qualifying price. Queueing is bounded by the managed run
deadline and can be cancelled with `flash runs cancel`.

## Checkpoint recovery

Omit `train.save_every` for Spot: recovery checkpointing is automatic and
uses a roughly ten-minute target based on measured step duration. Flash saves
after the first completed step, and saves before another estimated step would
exceed the target. Steps that themselves take ten minutes or longer are saved
after every completed step. A long in-flight step can still be lost.

`train.save_at_steps` remains available for required checkpoint boundaries and
requires a positive `max_steps`. The final step is saved too. List published
adapters with:

```bash theme={null}
flash runs checkpoint <run-id>
```

After a detected Spot preemption, Flash can reallocate and resume from the
latest recovery checkpoint within its managed retry budget. Unsaved work may
repeat; recovery is not unlimited and does not promise uninterrupted execution.
Use the current attempt in `flash runs log <run-id>` to distinguish recovered
progress from superseded logs. Ordinary Modal preemption does not use this
Spot-specific retry policy.

Recovery checkpoints and exported adapters serve different purposes: an
adapter alone does not contain all optimizer state needed for automatic resume.
For a deliberate new training run from an adapter, use
[warm-start](https://docs.freesolo.co/guides/training.md#warm-start-safely).

## Billing and cancellation

Review `--cost` before submission. Provider threshold values are not a live
invoice; the preview and run status report the applicable Flash estimate and
charge. Recovery attempts can repeat compute. Monitor accepted cost and run
state, and cancel through Flash so it can clean up the owned resources.
See [Cost and billing](https://docs.freesolo.co/reference/cost-model.md).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.