Enable Spot
Add this to an otherwise valid SFT or GRPO config:provider = "spot" or providers = ["spot"] also selects Spot. Use one form;
do not combine provider and providers, mix Spot with Modal, or set
spot = false alongside a Spot provider. An unavailable Spot provider does not
silently fall back to Modal. Managed users need no local provider credentials;
self-hosted operators configure them
on the control plane.
Preview before submission:
GPU shapes and queueing
Spot supports H100, H200, and B200, in one- or eight-card shapes. Omittype and count for automatic selection within the provider. A GPU type
without a count is a single-card constraint. To permit an eight-card allocation:
count is a ceiling, so this can still select one card when it fits. A ceiling
of two or four does not create a two- or four-card Spot shape. Model memory,
training batch geometry, provider quota, and live capacity still determine
whether the run is admissible.
Flash checks a fresh provider price before provisioning. The configured entry
thresholds are strictly below $1.00/GPU-hour for H100/H200 and $4.00/GPU-hour for
B200. Running-instance price caps are $1.50, $2.00, and $5.00 respectively.
These are allocation thresholds, not customer prices or capacity guarantees.
Use your authenticated quote for the amount you accept. A run can remain
queued while price or capacity is unsuitable; no training GPU is provisioned
just to wait for a qualifying price. Queueing is bounded by the managed run
deadline and can be cancelled with flash runs cancel.
Checkpoint recovery
Omittrain.save_every for Spot: recovery checkpointing is automatic and
uses a roughly ten-minute target based on measured step duration. Flash saves
after the first completed step, and saves before another estimated step would
exceed the target. Steps that themselves take ten minutes or longer are saved
after every completed step. A long in-flight step can still be lost.
train.save_at_steps remains available for required checkpoint boundaries and
requires a positive max_steps. The final step is saved too. List published
adapters with:
flash runs log <run-id> to distinguish recovered
progress from superseded logs. Ordinary Modal preemption does not use this
Spot-specific retry policy.
Recovery checkpoints and exported adapters serve different purposes: an
adapter alone does not contain all optimizer state needed for automatic resume.
For a deliberate new training run from an adapter, use
warm-start.
Billing and cancellation
Review--cost before submission. Provider threshold values are not a live
invoice; the preview and run status report the applicable Flash estimate and
charge. Recovery attempts can repeat compute. Monitor accepted cost and run
state, and cancel through Flash so it can clean up the owned resources.
See Cost and billing.