Skip to main content
Flash uses Modal for ordinary training. With Flash 1.4.22, SFT and GRPO can explicitly select Spot when the control plane has that provider enabled. Spot capacity can be interrupted; choose it when you can tolerate queueing and repeated work after recovery. Custom training remains on one Modal GPU.

Enable Spot

Add this to an otherwise valid SFT or GRPO config:
provider = "spot" or providers = ["spot"] also selects Spot. Use one form; do not combine provider and providers, mix Spot with Modal, or set spot = false alongside a Spot provider. An unavailable Spot provider does not silently fall back to Modal. Managed users need no local provider credentials; self-hosted operators configure them on the control plane. Preview before submission:

GPU shapes and queueing

Spot supports H100, H200, and B200, in one- or eight-card shapes. Omit type and count for automatic selection within the provider. A GPU type without a count is a single-card constraint. To permit an eight-card allocation:
count is a ceiling, so this can still select one card when it fits. A ceiling of two or four does not create a two- or four-card Spot shape. Model memory, training batch geometry, provider quota, and live capacity still determine whether the run is admissible. Flash checks a fresh provider price before provisioning. The configured entry thresholds are strictly below $1.00/GPU-hour for H100/H200 and $4.00/GPU-hour for B200. Running-instance price caps are $1.50, $2.00, and $5.00 respectively. These are allocation thresholds, not customer prices or capacity guarantees. Use your authenticated quote for the amount you accept. A run can remain queued while price or capacity is unsuitable; no training GPU is provisioned just to wait for a qualifying price. Queueing is bounded by the managed run deadline and can be cancelled with flash runs cancel.

Checkpoint recovery

Omit train.save_every for Spot: recovery checkpointing is automatic and uses a roughly ten-minute target based on measured step duration. Flash saves after the first completed step, and saves before another estimated step would exceed the target. Steps that themselves take ten minutes or longer are saved after every completed step. A long in-flight step can still be lost. train.save_at_steps remains available for required checkpoint boundaries and requires a positive max_steps. The final step is saved too. List published adapters with:
After a detected Spot preemption, Flash can reallocate and resume from the latest recovery checkpoint within its managed retry budget. Unsaved work may repeat; recovery is not unlimited and does not promise uninterrupted execution. Use the current attempt in flash runs log <run-id> to distinguish recovered progress from superseded logs. Ordinary Modal preemption does not use this Spot-specific retry policy. Recovery checkpoints and exported adapters serve different purposes: an adapter alone does not contain all optimizer state needed for automatic resume. For a deliberate new training run from an adapter, use warm-start.

Billing and cancellation

Review --cost before submission. Provider threshold values are not a live invoice; the preview and run status report the applicable Flash estimate and charge. Recovery attempts can repeat compute. Monitor accepted cost and run state, and cancel through Flash so it can clean up the owned resources. See Cost and billing.