> ## Documentation Index
> Fetch the complete documentation index at: https://docs.freesolo.co/llms.txt
> Use this file to discover all available pages before exploring further.

# Evaluate checkpoints

> Publish held-out suites and compare deployed checkpoints against the same task.

Evaluate on data excluded from training. Flash's `env test` validates an
actual environment locally without model inference; `env eval` sends requests
to a deployed checkpoint and grades its responses. Evaluation inference is
billable even when you choose not to upload the results.

## Publish a held-out suite

Add `evaluations.py` next to `environment.py` before publishing the environment:

```python evaluations.py theme={null}
from flash.envs.meta.evaluations import BaseEvalSuite, EvalCase


class HeldOut(BaseEvalSuite):
    name = "held-out"

    def cases(self):
        return [
            EvalCase(id="addition", input="What is 17 + 25?", expected="42"),
            EvalCase(id="subtraction", input="What is 60 - 19?", expected="41"),
        ]

    def score(self, case, response):
        return float(response.strip() == case.expected)


def load_evaluations(environment=None):
    return [HeldOut()]
```

The loader also accepts a module-level `EVALUATIONS` list. A scorer may return
an `EvalResult`, a finite float, or a boolean. Keep grading deterministic when
possible and test it against clearly correct and incorrect responses. The
`environment` factory argument lets a suite reuse the environment's task logic.

```bash theme={null}
flash env test . --algorithm sft
flash env push . --project <project-uuid> --name my-environment
```

Train with the resulting environment ID. Flash evaluates using the immutable
published environment associated with the target run, not your current working
copy. A run without a published managed environment cannot use this command;
custom controllers own their evaluation data and grading.

## Deploy and evaluate

```bash theme={null}
flash models deploy <run-id>/final --wait
flash env eval <run-id>/final --suite held-out --concurrency 4
```

Use an exact deployed checkpoint identity, including `/final` or `/step-N`.
The command does not take a local environment path or a bare base-model ID.

| Option | Behavior |
| - | - |
| `--suite NAME` | Select a named suite; omit to run the available suites. |
| `--max-cases N` | Run at most N cases per selected suite. |
| `--temperature T` | Nonnegative sampling temperature; default 0. |
| `--max-tokens N` | Output budget; default 512. |
| `--concurrency N` | 1–32 concurrent requests; default 1. Start small. |
| `--split SPLIT`, `--param KEY=VALUE` | Configure the published environment loader. |
| `--no-upload` | Print results locally without saving them to the dashboard. |
| `--project UUID` | Override the project receiving the evaluation record; cannot be combined with `--no-upload`. |

Results normally upload to the training run's project. An upload failure is
reported separately from model scoring; preserve the local output when
investigating it. See [CLI reference](https://docs.freesolo.co/reference/cli.md#environments) for all flags.

## Multi-turn evaluation

For an environment with episode hooks, an episode-grading suite can declare
`grades_episodes = True` and implement `score(case, response, state)`. The
finished transcript is available through the evaluation state. Flash drives
`start_episode`, `step_episode`, and the turn limit to produce a complete
interaction. Keep the same tool behavior, image observations, and transcript
budget used in training. See [Multi-turn environments](https://docs.freesolo.co/guides/environments/multi-turn.md).

## Compare runs fairly

Use the same environment revision, suite, sampling settings, and output budget
for each checkpoint. Report errors and incomplete cases alongside the score;
a mean over successful requests alone can hide failures. Keep prompt and
completion token counts when comparing inference cost or speed.

For custom training, keep held-out records in the controller and evaluate saved
adapters through the [serving API](https://docs.freesolo.co/guides/deploy-and-chat.md#use-it-from-your-own-code).
Do not include evaluation examples in gradient updates.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.