Datasets

Datasets live inside the environment. A Flash config points at one published environment id. The environment’s load_environment() function returns the dataset, prompt builder, and reward logic that Flash uses for SFT, GRPO, and OPD, and local validation.

[environment]
id = "your-org/your-env"

Task records

Author dataset rows with input and output:

Dataset key	Description
`input`	Prompt text for the model.
`output`	Target answer or gold completion. For SFT this can be a scalar answer, `{ "messages": [...] }`, or a bare list of chat messages.
`metadata`	Optional dict preserved on `example.metadata` and available to scoring.

The algorithm decides which column the model learns from. SFT trains directly on output, the gold answer. GRPO and OPD use only input: the model generates its own answers from the prompt, and learns from your reward (GRPO) or a teacher’s token-level grading (OPD). output is optional for both. GRPO reads it only if your score_response uses it as a reference; OPD ignores it entirely, since the teacher, not your reward, is the training signal. load_task_examples(...) accepts a local file path or an iterable of records.

File formats: .jsonl, .json, .csv, .txt, or .bson.
Field mapping: input -> example.input, output -> example.output, metadata -> example.metadata.
Original row: the untouched record stays available as example.record.

dataset/train.jsonl

{"input":"What is 2 + 2?","output":"4"}
{"input":"What is 3 + 5?","output":"8"}

Each row must be input plus an optional output; alternate prompt or target key names are not accepted. Records are canonicalized to exactly input/output/metadata.

You do not have to author these rows by hand. If your app already calls an LLM, record its traffic as traces and run flash traces export: every recorded call becomes one input/output row, written to dataset/train.jsonl by default. flash env setup can also pull a project’s traces in while it scaffolds, so a new environment starts with your own data instead of the starter rows.

When Flash builds your training records it keeps only input/output/metadata and silently drops every other top-level key before the row reaches training. Anything your scorer needs beyond the gold output string (a puzzle’s initial_board, the oracle_ids a retrieval must return, unit tests to check code against, a grading rubric) has to live under metadata, or it is gone with no runtime warning. The one exception is the top-level image/images fields, which the multimodal loader reads from the original row (see Image inputs).

Message-shaped SFT targets

For SFT, output is the gold completion appended after the environment’s prompt messages. A scalar output becomes one assistant message. To teach a multi-turn trajectory or native tool calling, set output to {"messages": [...]} or a bare list of chat messages. Flash preserves those assistant, tool-call, tool-result, and reply messages when it builds the SFT example.

dataset/train.jsonl

{"input":"Refund my last order.","output":{"messages":[{"role":"assistant","content":null,"tool_calls":[{"id":"call_refund","type":"function","function":{"name":"refund_order","arguments":"{\"order\":\"last\"}"}}]},{"role":"tool","tool_call_id":"call_refund","content":"{\"ok\":true}"},{"role":"assistant","content":"Your last order has been refunded."}]}}

A message-shaped output is validated rather than coerced, so a malformed trajectory fails instead of collapsing into one bare turn:

{"messages": ...} must hold a list, and every entry in it must be an object.
{"messages": [...]} may not carry sibling keys - that shape is exactly one key.
A list mixing message objects with non-objects is rejected.

A scalar output is still rendered into a single assistant message, so the strictness applies to targets that look like chat messages, not to plain answers. Use Environment.sft_completion(example) when your environment must synthesize or transform the gold completion before SFT.

Validate thinking-model SFT targets

SFT on a thinking model (thinking = true) expects each gold completion to literally contain a <think>...</think> block. Catch missing blocks locally before submitting a run:

from freesolo.datasets import load_dataset, warn_missing_think_tags

dataset = load_dataset("dataset/train.jsonl")
missing = warn_missing_think_tags(dataset.examples)  # UserWarning + list of offending ids

It returns the ids of offending examples and emits a UserWarning naming the first few. Unlabeled records (no output) are skipped.

Image inputs

Every catalog model is multimodal, so a prompt can include images and Flash trains and serves on them. Images belong in the prompt, not in the gold completion: an image inside an SFT output message is rejected. Build a multimodal prompt the OpenAI way: a user message whose content is a list of parts, mixing text with one or more image parts. Flash accepts image, image_url, and input_image parts, and each image source can be a data: URI, raw bytes, a PIL image, or a package-root-relative path under dataset/ (e.g. dataset/photo.jpg). Remote http(s) URLs are not accepted; see below:

environment.py

import base64
from pathlib import Path

from freesolo.environments import EnvironmentSingleTurn

ROOT = Path(__file__).parent


class VisionEnv(EnvironmentSingleTurn):
    def build_prompt_messages(self, example, prompt_text):
        raw = (ROOT / "dataset" / example.metadata["image"]).read_bytes()
        uri = "data:image/jpeg;base64," + base64.b64encode(raw).decode()
        return [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": example.input},
                    {"type": "image_url", "image_url": {"url": uri}},
                ],
            }
        ]

A task record can also carry a top-level image (single) or images (a list). These are the one exception to the field whitelist above: the multimodal loader reads them from the original row (example.record) and appends them to the first user message, so they survive even though they are not part of the canonical input/output/metadata mapping. Either way, one example may hold up to 8 images.

Remote URLs

Remote http(s) image URLs are not supported, and there is no flag to enable them: Flash never fetches a user-supplied URL at training time, so the training input stays fixed at submit. Include the image in the environment package as a relative path, or embed it as a data URI. A relative path is resolved from the package root and must stay inside dataset/ (e.g. dataset/photo.jpg). Algorithm notes: SFT trains on image prompts with a text completion, and GRPO carries the prompt’s images through the episode. OPD needs a teacher that can see the image - select a vision-capable teacher_model, or the run is rejected rather than distilled from a teacher that never saw it. See image-bearing OPD. Package image files under dataset/ so they upload with the environment (see What gets uploaded).

Load sidecars

Read packaged files relative to __file__ so paths work locally and in a managed run.

environment.py

from pathlib import Path
from freesolo.datasets.records import load_task_examples
from freesolo.environments import EnvironmentSingleTurn

ROOT = Path(__file__).parent


class MathEnv(EnvironmentSingleTurn):
    def __init__(self, *, split: str = "train") -> None:
        # read a packaged dataset file relative to environment.py
        self.dataset = load_task_examples(ROOT / "dataset" / f"{split}.jsonl")

The rest of the env class (build_prompt_messages, score_response) is covered in Environments. Then select the split from your Flash config:

[environment]
id = "your-org/math"

[environment.params]
split = "eval"

[environment.params] values are passed to your load_environment(**kwargs). split is also honored by Flash itself: for an environment packaged with dataset files, split = "eval" selects dataset/eval.jsonl (or .json) as the dataset Flash trains on: SFT targets and GRPO problem selection alike. If the environment packages a default train split but the requested split file does not exist, the run fails at load time instead of silently falling back to train.jsonl. An explicit dataset_path param takes precedence over split.

If your load_environment() builds self.dataset in code, those rows are what Flash trains on - the packaged file is only the fallback for an environment that exposes no dataset of its own. Filtering, subsampling, augmentation, and deduplication done in Python are honored rather than overwritten by a re-read of the file. That holds even when your filter matches nothing: an empty in-code dataset fails rather than quietly falling back to the unfiltered file and training on the rows you meant to exclude. An explicitly selected dataset_path, records, or side split still wins over an in-code dataset.

What gets uploaded

For a local environment directory, flash env push includes:

environment.py, always at the artifact root.
Sibling Python helper files.
Sidecar directory named dataset.
Common sibling data files such as .jsonl, .json, .csv, .txt, .md, .parquet, .tsv, .yaml, and .yml.

Workspace metadata, cache directories, virtualenvs, and version-control files are skipped, and secrets are never uploaded: .env files and files matching *.key, *.pem, *.pfx, *.p12, credentials*, or SSH key names (id_rsa*, id_ed25519*, and similar) are excluded. Keep the artifact small: environment uploads are capped at 64 MB compressed, 256 MB uncompressed, and 5000 files. For large corpora, keep the data in an external store and pass the identifier or URL through [environment.params]. When you push a single .py file (or a folder with just one top-level .py), only that entrypoint, a sibling README.md/TRAINING.md, and any dataset/ tree are packaged, and sibling helper modules are not. An evaluations.py is not packaged in single-file mode either; push the folder if you want flash env eval suites to travel with the environment.

Get started

Flash

Reference

Task records

Message-shaped SFT targets

Validate thinking-model SFT targets

Image inputs

Remote URLs

Load sidecars

What gets uploaded

Next steps

Single-turn environments

Training

​Task records

​Message-shaped SFT targets

​Validate thinking-model SFT targets

​Image inputs

​Remote URLs

​Load sidecars

​What gets uploaded

​Next steps

Single-turn environments

Training

Task records

Message-shaped SFT targets

Validate thinking-model SFT targets

Image inputs

Remote URLs

Load sidecars

What gets uploaded

Next steps