# Media (/docs/media)

Media is a CoreSpeed-hosted capability: no third-party account to connect, and
the same `/mcp` seam as everything else, so identity, organization holds,
API-key caps, activity recording, and capability visibility all apply. It does
three things: **generation** from a catalog of more than a thousand image, video,
audio and speech models; **understanding**, which answers a question about a
video, image, or audio file you point it at; and **transcription**, which turns
speech in a recording into text.

## Tools \[#tools]

| Tool                 | Use it for                                                                                      |
| -------------------- | ----------------------------------------------------------------------------------------------- |
| `media__list_models` | Search the generation catalog with credit rates. No arguments returns the featured picks.       |
| `media__get_model`   | One model's input schema, vendor, and current rate. Call it before generating.                  |
| `media__generate`    | Run a model. Waits inline up to `wait_seconds`, otherwise hands back a running job.             |
| `media__get_result`  | A job's outcome: fresh signed URLs for a finished job, or its current status.                   |
| `media__list_jobs`   | This organization's recent jobs, running and finished. Receipts are kept about 30 days.         |
| `media__cancel`      | Stop a running job. Canceled jobs are never charged.                                            |
| `media__understand`  | Ask a question about a video, image, or audio file and get a text answer.                       |
| `media__transcribe`  | Turn speech in an audio or video file into text, with speaker labels and optional word timings. |
| `media__list_voices` | Find a voice in Fish Audio's public library for the speech models.                              |

## Generate: pick a model, then run it \[#generate-pick-a-model-then-run-it]

1. `media__list_models` — search by name, `kind` (image, video, or audio), or
   exact `category`, and choose a model id.
2. `media__get_model` — read its `input_schema` and structured `price`.
3. `media__generate` — build `input` against that schema. Files come back
   inline, or as a `job_id` that finishes server-side.

The schema and the rate both come from the catalog, never from the client.

`input` is validated against the model's own `input_schema`. Unknown top-level
keys are rejected — the schema's properties are the allowlist — and safety
parameters are policy, not input: whatever the caller sends for a `safety*`,
`nsfw*`, or `moderation*` key is discarded and the safe value is enforced on the
server. When the upstream model rejects an input anyway, its own validation
detail is returned to you, unbilled.

File-typed inputs accept two sources:

* a **public `https` URL** fetchable without authentication;
* a **file in your Drive** — its path or file id: an upload
  (`drive__create_upload`, below), a connector download, a mail attachment,
  or an earlier generation's output — so image → video chains stay entirely
  inside CoreSpeed.

### A model needs a rate before it can run \[#a-model-needs-a-rate-before-it-can-run]

`price` in the catalog is structured, and it is a **rate, not a total**:

```json
{ "state": "priced", "credits_per_unit": 4, "unit": "image" }
```

`{ "state": "unavailable" }` carries no rate fields and means there is no rate
right now. Generation is refused (`pricing_unavailable`, retryable) rather than
billed blind — retry shortly.

## Long generations become jobs \[#long-generations-become-jobs]

`wait_seconds` (default 45, max 300) is the time budget for the whole call. The
server answers inside it, one of two ways:

```json title="finished inside the budget"
{
  "model": "fal-ai/flux/schnell",
  "kind": "image",
  "job_id": "job_…",
  "status": "succeeded",
  "assets": [
    {
      "id": "art_…",
      "mimeType": "image/png",
      "size": 1284021,
      "uri": "https://api.corespeed.io/artifacts/…",
      "expiresAt": "2026-09-10T14:02:11.000Z",
      "width": 1024,
      "height": 1024
    }
  ]
}
```

```json title="still running"
{ "job_id": "job_…", "status": "running", "model": "fal-ai/kling-video/v2" }
```

A running job keeps going on the server for up to **60 minutes**, then expires
unbilled. Collect it with `media__get_result` from any session — any agent in
the organization can pick up a job another one started — and pass
`wait_seconds` there to long-poll instead of spinning.

Three rules keep this from costing you twice:

* **Do not set a client-side tool timeout equal to or below `wait_seconds`.**
  That kills the very reply carrying your `job_id`. Disable the timeout for this
  call, or set a backstop of at least twice `wait_seconds`.
* **If the client timed out anyway, nothing is lost.** The newest entry in
  `media__list_jobs` is your job. Do not resubmit.
* **Signed URLs expire after 12 hours.** Call `media__get_result` again for a
  fresh set; the stored artifact does not go anywhere.

Job states are `running`, `succeeded`, `failed`, `canceled`, and `expired`. A
terminal failure comes back as a tool-level error with `generation_failed` (or
`empty_output` when the upstream finished without a file), carrying the
upstream's reason in the text when it gave one; `generation_canceled` or
`generation_expired`; an unknown id answers `job_not_found`. Every code is in
[Errors & status codes](/docs/reference/errors#memory-media-web-and-social).

## Speech and voices \[#speech-and-voices]

The Fish Audio models in the catalog turn text into speech:
`fish-audio/tts/drama-3` (directed performance), `fish-audio/tts/s2.1-pro` and
`fish-audio/tts/s2-pro`. Find them with `media__list_models` (`query: "fish"`,
or `kind: "audio"`). Their `input` takes the script as `text`, with any inline
directions the model supports — `[whispers]`, `[shaky breath]`,
`<emphasis>…</emphasis>`.

Choose the voice one of two ways:

1. **A library voice.** `media__list_voices` searches Fish Audio's public
   library by name and language; each row has a `sample_url` to listen to first.
   Pass its `voice_id`. Use `licensed_only: true` for commercial work: the rest
   of the library is community uploads, and some imitate real people. A
   licensed voice bills at a higher rate than the model alone.
2. **Your own voice.** Your own voice is an audio file in your Drive: a
   recording of one speaker, 10–30 seconds of clean speech (up to 10 MB).
   Pass it as `reference_audio_url`, with the exact words it says as
   `reference_text`. Fish Audio keeps nothing: the file is sent with each
   call, so keep it short. It bills at the model's rate.

Only use a voice you have the speaker's permission to use.

For dialogue, start each turn with `<|speaker:0|>`, `<|speaker:1|>`, … and pass
one voice per speaker, in order: an array of `voice_id`s, or an array of
`reference_audio_url` files with an array of `reference_text`s, one each.

`fish-audio/voice-design/v1` designs a new voice from a description and returns
up to four short samples in your Drive. To speak with one, pass its file as
`reference_audio_url`.

## Transcribe: speech to text \[#transcribe-speech-to-text]

`media__transcribe` takes one `media` source — a Drive path or file id, or a
public `https` URL — up to 60 minutes and 100 MB: WAV, MP3, AAC/M4A, FLAC, Ogg,
WebM or MOV. Send a whole conversation in one call: speaker labels are
consistent only within one answer.

The transcript is the text answer, with speakers marked inline
(`<|speaker:0|>`) and emotion and sound cues such as `[laughter]` unless
`tag_audio_events` is false. `timestamps: true` adds word-level `segments` and
timed `speaker_turns`. `language` is only a hint; the language is detected
either way. Long recordings take minutes to process.

## Understand: ask about a file \[#understand-ask-about-a-file]

`media__understand` takes a `prompt` and one `media` source — a file in your
Drive (its path or file id), a public `https` URL, or a YouTube watch URL.
`mime_type` is inferred from the URL extension or the origin's `Content-Type`,
or carried by the Drive file.

* **Video** can be windowed with `start_seconds` / `end_seconds`.
  `max_cost_credits` converts a credit budget into a window at the per-second
  rate; it bounds the media input only, and an explicit `end_seconds` overrides
  it. Video is sampled at one frame per second, so fast action between frames
  can be missed, and timestamps stated in the answer are not guaranteed exact.
* **Audio and images** reject all three window parameters, because clipping
  cannot be enforced for them. The whole file is analysed up to the
  capability's context limit.

The result carries `analyzed_seconds` (an estimate — a paging hint, not a
duration), `truncated` when the answer was cut at the output limit, and `usage`
with the metered quantities the call settled for.

## Upload a private file \[#upload-a-private-file]

For a file that only exists on local disk:

1. Call `drive__create_upload` with the exact `mime_type` and `size_bytes`.
   The result includes `file_id`, `upload_url`, the `PUT` method, the
   headers to send — including `Authorization` — `expires_at`, and
   `lands_at`: the path the file will have in your [Drive](/docs/drive).
2. `PUT` the raw bytes to `upload_url` with every returned header, within
   **10 minutes**. The ticket is one-use, and the declared type and size are
   bound to it.
3. Use the path (or `file_id`) with `media__understand` or any file-typed
   `media__generate` input. The file sits under `tmp/uploads/` for 7 days;
   `drive__move` it anywhere else to keep it.

Every output a generation produces is a Drive file too, under
`tmp/media/<job_id>/`, for 7 days unless you move it.

Files up to 2 GB are accepted. A deployment edge may still reject a very large
body with HTTP `413`; treat that as non-retryable for the deployment rather than
retrying the same upload.

## Pricing and holds \[#pricing-and-holds]

* **Generation** is billed once, after success, at the model's rate times the
  usage the upstream actually reports. For calls whose size is only known
  afterwards, the final amount can differ from the rate card. Failures,
  cancels, and expiries never charge. For expensive kinds — video especially —
  check `media__get_model` first.
* **Speech** bills per UTF-8 byte of `text`, tags included, at the model's
  rate per 1,000 bytes — so a character in Chinese or Japanese costs three
  bytes. A licensed library voice adds a surcharge per 1,000 bytes on the text
  it speaks. Voice
  design bills once per call, however many samples it returns.
* **Transcription** bills 9 credits per minute of audio, per second of the whole
  recording, silence included.
* **Understanding** bills 0.5 credits per image (an oversized image counts as
  several), 12 credits per minute of video and 4.2 per minute of audio, both
  metered per second, plus about 1.1 credits per 100 answer tokens. The question
  itself is free. Failures never charge.

The live figures are in the tool descriptions your client shows from
`tools/list` and in each model's `price`. Every charge lands itemized in the
organization ledger; see [Billing & credits](/docs/billing). An organization
billing hold or suspension stops a media call before it runs — as an `isError`
result with `payment_required` or `org_suspended` — and API-key spend caps apply
the same way.

## Inspect in the dashboard \[#inspect-in-the-dashboard]

[Dashboard → Media](https://app.corespeed.io/media) shows everything the
organization generated. Running jobs finish server-side and land there whichever
agent started them. Turning the capability off in
[Capability controls](/docs/capability-controls) hides the `media__*` tools from
`tools/list`; stored artifacts and job receipts are unaffected.