Skip to main content
Evaluate on data excluded from training. Flash’s env test validates an actual environment locally without model inference; env eval sends requests to a deployed checkpoint and grades its responses. Evaluation inference is billable even when you choose not to upload the results.

Publish a held-out suite

Add evaluations.py next to environment.py before publishing the environment:
evaluations.py
The loader also accepts a module-level EVALUATIONS list. A scorer may return an EvalResult, a finite float, or a boolean. Keep grading deterministic when possible and test it against clearly correct and incorrect responses. The environment factory argument lets a suite reuse the environment’s task logic.
Train with the resulting environment ID. Flash evaluates using the immutable published environment associated with the target run, not your current working copy. A run without a published managed environment cannot use this command; custom controllers own their evaluation data and grading.

Deploy and evaluate

Use an exact deployed checkpoint identity, including /final or /step-N. The command does not take a local environment path or a bare base-model ID. Results normally upload to the training run’s project. An upload failure is reported separately from model scoring; preserve the local output when investigating it. See CLI reference for all flags.

Multi-turn evaluation

For an environment with episode hooks, an episode-grading suite can declare grades_episodes = True and implement score(case, response, state). The finished transcript is available through the evaluation state. Flash drives start_episode, step_episode, and the turn limit to produce a complete interaction. Keep the same tool behavior, image observations, and transcript budget used in training. See Multi-turn environments.

Compare runs fairly

Use the same environment revision, suite, sampling settings, and output budget for each checkpoint. Report errors and incomplete cases alongside the score; a mean over successful requests alone can hide failures. Keep prompt and completion token counts when comparing inference cost or speed. For custom training, keep held-out records in the controller and evaluate saved adapters through the serving API. Do not include evaluation examples in gradient updates.