env test validates an
actual environment locally without model inference; env eval sends requests
to a deployed checkpoint and grades its responses. Evaluation inference is
billable even when you choose not to upload the results.
Publish a held-out suite
Addevaluations.py next to environment.py before publishing the environment:
evaluations.py
EVALUATIONS list. A scorer may return
an EvalResult, a finite float, or a boolean. Keep grading deterministic when
possible and test it against clearly correct and incorrect responses. The
environment factory argument lets a suite reuse the environment’s task logic.
Deploy and evaluate
/final or /step-N.
The command does not take a local environment path or a bare base-model ID.
Results normally upload to the training run’s project. An upload failure is
reported separately from model scoring; preserve the local output when
investigating it. See CLI reference for all flags.
Multi-turn evaluation
For an environment with episode hooks, an episode-grading suite can declaregrades_episodes = True and implement score(case, response, state). The
finished transcript is available through the evaluation state. Flash drives
start_episode, step_episode, and the turn limit to produce a complete
interaction. Keep the same tool behavior, image observations, and transcript
budget used in training. See Multi-turn environments.