Deploy & chat

Deploy a trained adapter and chat with it from the CLI or any OpenAI-compatible client, once its training run reaches done. flash models deploy registers your adapter with Freesolo’s managed serving service. Send requests to it and pay per token (see Billing).

Deploy

flash models deploy <run-id>

Preview what a deploy would do without creating it:

flash models deploy <run-id> --dry-run

Every real deploy resolves the adapter to an immutable Hugging Face commit, runs a bounded serving smoke against that revision, and atomically activates the stable run-id alias only if the smoke passes; otherwise the existing alias is left unchanged. flash models deploy returns while the revision is still queued, so the model is not servable the moment the command exits. To gate a script on the deployment, add --wait:

flash models deploy <run-id> --wait

It blocks until the revision is servable and exits 0, or exits 1 if the deployment fails, rolls back, or the wait times out. The default timeout is 2400 seconds; pass --wait 600 for your own budget, or --wait 0 for a single state read that does not block. Interrupting with Ctrl-C stops the waiting, not the deployment. You can also deploy and undeploy from the model’s detail page in the dashboard. Both paths run the same verification and produce the same alias, so a deployment started in the web app is visible to flash models deployments and can be torn down with flash models undeploy.

Deploy a specific checkpoint

Every training step that saved an adapter is independently deployable. List a run’s deployable checkpoints:

flash runs checkpoint <run-id>

Then serve a specific step instead of the final adapter:

flash models deploy <run-id>/step-<N>

Deploy a checkpoint while a run is still training, or after a GRPO run stops with useful intermediate steps. Checkpoint deployments attach serving metadata to the run without altering its training state.

A run that never finalized (cancelled or preempted mid-training) has no final adapter, so a plain flash models deploy <run-id> fails with an error that lists the saved checkpoint steps and the exact flash models deploy <run-id>/step-N command to deploy one of them instead.

Stable aliases and immutable revisions

Final adapters use RUN_ID@final.<40-char-sha> and saved checkpoints use RUN_ID@step-N.<40-char-sha>. Each run has exactly one shared mutable bare RUN_ID alias. Deploying another checkpoint does not create a second bare alias; after the new revision passes its bounded smoke, it moves the shared alias for every caller using RUN_ID. The CLI warns before moving it away from a different checkpoint. Older immutable revisions can remain directly callable. RUN_ID/step-N is a deploy and export selector. Once you have deployed that checkpoint, flash models chat RUN_ID/step-N works from the CLI too, resolving to that checkpoint’s verified revision without depending on where the bare alias currently points. From an OpenAI client, set model to the bare run alias or the full immutable revision from flash models deployments; the RUN_ID/step-N shorthand is a Flash-CLI selector, not an OpenAI model name.

Billing

Serving is billed per token. Prompt and completion tokens have per-model rates, and cached prompt tokens use the model’s cached-input rate. See prices in Supported models. Cached prompt tokens are cheaper than uncached prompt tokens. Prefix caching is always on: when a request’s prompt shares a leading prefix with a recent one — a shared system prompt, or the growing context of a multi-turn chat — the serving reuses that prefix from cache instead of recomputing it. Only the input prefix can be cached, so completion tokens always bill at the full per-token rate. When a request hits the prefix cache, the response reports how many prompt tokens were served from cache in usage.prompt_tokens_details.cached_tokens. Sub-cent usage is carried forward, so small requests are accounted for without rounding each one up to a cent.

Chat from the CLI

flash models chat <run-id> -m "Summarize the plot of Hamlet in two sentences."

flash models chat accepts the stable run alias, a full immutable revision, or RUN_ID/step-N for a checkpoint you have already deployed. A RUN_ID/step-N target resolves to that checkpoint’s verified revision, so deploy it first with flash models deploy RUN_ID/step-N. -m/--message is required; --system, --max-tokens (default 512), and --temperature (default 0.0) are optional. See the CLI reference for the full flag list.

flash models chat <run-id> -m "Write a haiku about mountain weather" --temperature 0.8 --max-tokens 128

--system is transient — it applies to that one request and is not stored with the deployment. Use it to probe the adapter with the same system prompt it was trained with:

flash models chat <run-id> -m "What is 6*7?" --system "Answer with just the number."

Manage deployments

flash models deployments         # human-readable deployment status
flash models deployments --json  # complete machine-readable records
flash models undeploy <run-id>   # disable the alias and immutable revisions

Human output shows run id, step, revision, state, verification time, OpenAI model, and detail. JSON output includes openai_base_url for clients.

Use it from your own code

Run-scoped chat route

For a deployed run you own, post directly to the Flash control plane. The route checks run ownership, resolves the active verified deployment, and preserves the run’s serving options such as trained thinking mode:

curl -X POST "https://flash.freesolo.co/v1/runs/<run-id>/chat" \
  -H "Authorization: Bearer <your-freesolo-api-key>" \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Hello!"}],"max_tokens":256}'

The response uses the OpenAI chat-completions shape. Omit selectors to use the shared bare run alias, or pass exactly one of "step": N and "adapter_revision": "<full-immutable-revision>" to pin a checkpoint that has already passed deployment verification for that run. If the run has no ready deployment, the route returns a conflict with a deploy hint.

Direct OpenAI-compatible endpoint

The direct serving endpoint remains available. Use flash models deployments --json and pass its openai_base_url as the SDK base URL. Set model to the shared bare run alias or a full immutable revision. A valid Freesolo API key is required, and the key’s org must own the adapter named in model:

from openai import OpenAI

client = OpenAI(
    base_url="https://serve.freesolo.co/v1",  # openai_base_url from flash models deployments --json
    api_key="<your-freesolo-api-key>",       # use the same freesolo key as flash login
)

resp = client.chat.completions.create(
    model="<run-id>",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.content)

For deployed adapters, serving preserves the run’s trained thinking mode. A caller cannot override enable_thinking per request for an adapter.

Reasoning is returned separately

When a generation runs in thinking mode, the reasoning comes back in reasoning_content and only the answer in content. The </think> separator is not included in either field:

resp = client.chat.completions.create(
    model="<run-id>",
    messages=[{"role": "user", "content": "Hello!"}],
)
print(resp.choices[0].message.reasoning_content)  # how it got there
print(resp.choices[0].message.content)  # the answer

Streaming splits the same way: reasoning tokens arrive as choices[0].delta.reasoning_content and answer tokens as choices[0].delta.content, so accumulate both. If a generation stops before the reasoning closes, everything generated is returned as reasoning_content and content is empty. Non-thinking completions are unaffected and use content alone. flash models chat folds both fields into one balanced <think>...</think> block; only direct endpoint calls expose the split shape.

Send images

Every catalog model accepts image inputs, so a served adapter or base model can take multimodal prompts over the same OpenAI-compatible API. Pass images as OpenAI image_url content parts. Serving accepts base64 data: URIs only; remote (http(s)) and file: URLs are rejected. A single request accepts up to four images:

import base64

with open("photo.jpg", "rb") as f:
    data_uri = "data:image/jpeg;base64," + base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="<run-id>",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "What is in this image?"},
                {"type": "image_url", "image_url": {"url": data_uri}},
            ],
        }
    ],
)

To train on image data, see Image inputs.

Structured outputs

Constrain a response to valid JSON with the OpenAI-standard response_format. A training structured_outputs constraint becomes the deployed adapter’s default. A request-level response_format can override it with text, json_object, or json_schema. This works on adapters and base models:

resp = client.chat.completions.create(
    model="<run-id>",
    messages=[{"role": "user", "content": "Give me a person as JSON."}],
    response_format={
        "type": "json_schema",
        "json_schema": {
            "schema": {
                "type": "object",
                "properties": {"name": {"type": "string"}, "age": {"type": "integer"}},
                "required": ["name", "age"],
            }
        },
    },
)

{"type": "json_object"} forces any valid JSON with no fixed schema, and {"type": "text"} leaves output unconstrained. With thinking enabled, reasoning is free-form and the grammar begins after </think>. Real deployments fail before alias activation if structured output is invalid, unclosed, or truncated. See the Structured outputs guide to also constrain training rollouts.

Chat with a base model (no adapter)

Using the same OpenAI-compatible client, set model to any id from flash models list, such as Qwen/Qwen3.5-4B, to generate against the base weights with no LoRA and no flash models deploy:

resp = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",   # a base-model id, not a run id — no deploy needed
    messages=[{"role": "user", "content": "Hello!"}],
)

Any valid Freesolo API key reaches a base model — there is no adapter to own — and the tokens are billed to your own org at that model’s serving prices. Unlike a deployed adapter, which serves the reasoning behavior it was trained with, a base model honors the enable_thinking you pass in chat_template_kwargs.

resp = client.chat.completions.create(
    model="Qwen/Qwen3.5-4B",
    messages=[{"role": "user", "content": "Answer directly: 17 * 19"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Serve in your own account

Everything above runs on Freesolo’s managed serving, billed per token. You can instead run the same serving stack in your own Modal or RunPod account with flash serve deploy. The GPU runs and bills in your account, and you call the provider’s HTTPS endpoint directly - no Freesolo gateway in the request path.

This needs the server extra: pip install 'freesolo-flash[server]'.

One command provisions one deployment: one base model, one run’s adapter.

export HF_TOKEN=hf_...                     # lets the container fetch your adapter
export FLASH_SERVING_KEY=$(python -c 'import secrets; print(secrets.token_urlsafe(32))')
export MODAL_TOKEN_ID=... MODAL_TOKEN_SECRET=...   # `modal token new` writes these

flash serve deploy \
  --provider modal \
  --model Qwen/Qwen3.5-4B \
  --run <run-id> \
  --deployment-id my-4b-serving \
  --image ghcr.io/freesolo-co/freesolo-flash-serve@sha256:<digest> \
  --artifact-repo <hub-repo> \
  --artifact-subfolder <path-within-repo> \
  --lora-rank 32 \
  --modal-workspace <your-workspace> \
  --modal-environment main \
  --modal-region us-east

For RunPod, export RUNPOD_API_KEY and swap the placement flags rather than adding to them - --provider runpod takes --runpod-account and --runpod-data-center, and rejects the --modal-* flags instead of ignoring them:

flash serve deploy \
  --provider runpod \
  --model Qwen/Qwen3.5-4B \
  --run <run-id> \
  --deployment-id my-4b-serving \
  --image ghcr.io/freesolo-co/freesolo-flash-serve@sha256:<digest> \
  --artifact-repo <hub-repo> \
  --artifact-subfolder <path-within-repo> \
  --lora-rank 32 \
  --runpod-account <your-account-id> \
  --runpod-data-center <data-center-id>

--image must be pinned to a digest (name@sha256:...), never a tag, so the deployment cannot drift to different content after it is recorded. Add --dry-run to resolve and validate every input - including the adapter’s provenance - without contacting the provider or spending anything.

Call your endpoint

deploy prints the endpoint URL. Authenticate with the FLASH_SERVING_KEY you generated, which is scoped to that one deployment:

curl https://<your-endpoint>/v1/chat/completions \
  -H "Authorization: Bearer $FLASH_SERVING_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "<run-id>", "messages": [{"role": "user", "content": "hello"}]}'

The deployment serves /healthz, /v1/models, and /v1/chat/completions. It receives its adapters in an immutable manifest at boot, so flash models deploy cannot drive it and never should: those commands carry FREESOLO_INTERNAL_KEY, and a customer-owned endpoint does not read that header at all.

Check and tear down

flash serve status proves the current state without changing anything, and flash serve undeploy removes one generation and proves its resources are gone. Both take the same identity flags as deploy, or the identity string that deploy printed via --deployment-identity. Pass the provider resource ids deploy printed to undeploy so each deletion binds to the exact generation. Provider credentials are read for one call and never stored, so export them again for each command.

If deploy ends in provisioning or outcome_unknown, resources may be live and billing in your account. Run flash serve status to inspect, then flash serve undeploy to stop them. Do not re-run deploy - that provisions and bills a second time.

Export to your own HuggingFace repo

Copy a trained adapter out of Freesolo’s managed storage into a repo you own:

flash models export --adapter-id <run-id> --repository <owner>/<name>

--adapter-id and --repository are required; --api-key (defaults to HF_TOKEN) and --public are optional. See the CLI reference for the full flag list.

Get started

Flash

Reference

Deploy

Deploy a specific checkpoint

Stable aliases and immutable revisions

Billing

Chat from the CLI

Manage deployments

Use it from your own code

Run-scoped chat route

Direct OpenAI-compatible endpoint

Reasoning is returned separately

Send images

Structured outputs

Chat with a base model (no adapter)

Serve in your own account

Call your endpoint

Check and tear down

Export to your own HuggingFace repo

Next steps

Serving prices

CLI reference

​Deploy

​Deploy a specific checkpoint

​Stable aliases and immutable revisions

​Billing

​Chat from the CLI

​Manage deployments

​Use it from your own code

​Run-scoped chat route

​Direct OpenAI-compatible endpoint

​Reasoning is returned separately

​Send images

​Structured outputs

​Chat with a base model (no adapter)

​Serve in your own account

​Call your endpoint

​Check and tear down

​Export to your own HuggingFace repo

​Next steps

Serving prices

CLI reference

Deploy

Deploy a specific checkpoint

Stable aliases and immutable revisions

Billing

Chat from the CLI

Manage deployments

Use it from your own code

Run-scoped chat route

Direct OpenAI-compatible endpoint

Reasoning is returned separately

Send images

Structured outputs

Chat with a base model (no adapter)

Serve in your own account

Call your endpoint

Check and tear down

Export to your own HuggingFace repo

Next steps