Media

Media is a CoreSpeed-hosted capability: no third-party account to connect, and the same /mcp seam as everything else, so identity, organization holds, API-key caps, activity recording, and capability visibility all apply. It does three things: generation from a catalog of more than a thousand image, video, audio and speech models; understanding, which answers a question about a video, image, or audio file you point it at; and transcription, which turns speech in a recording into text.

ToolsCopy Anchor Link

Generate: pick a model, then run itCopy Anchor Link

1. media__list_models — search by name, kind (image, video, or audio), or exact category, and choose a model id.
2. media__get_model — read its input_schema and structured price.
3. media__generate — build input against that schema. Files come back inline, or as a job_id that finishes server-side.

The schema and the rate both come from the catalog, never from the client.

input is validated against the model's own input_schema. Unknown top-level keys are rejected — the schema's properties are the allowlist — and safety parameters are policy, not input: whatever the caller sends for a safety*, nsfw*, or moderation* key is discarded and the safe value is enforced on the server. When the upstream model rejects an input anyway, its own validation detail is returned to you, unbilled.

File-typed inputs accept two sources:

• a public https URL fetchable without authentication;
• a file in your Drive — its path or file id: an upload (drive__create_upload, below), a connector download, a mail attachment, or an earlier generation's output — so image → video chains stay entirely inside CoreSpeed.

A model needs a rate before it can runCopy Anchor Link

price in the catalog is structured, and it is a rate, not a total:

{ "state": "priced", "credits_per_unit": 4, "unit": "image" }

{ "state": "unavailable" } carries no rate fields and means there is no rate right now. Generation is refused (pricing_unavailable, retryable) rather than billed blind — retry shortly.

Long generations become jobsCopy Anchor Link

wait_seconds (default 45, max 300) is the time budget for the whole call. The server answers inside it, one of two ways:

{
  "model": "fal-ai/flux/schnell",
  "kind": "image",
  "job_id": "job_…",
  "status": "succeeded",
  "assets": [
    {
      "id": "art_…",
      "mimeType": "image/png",
      "size": 1284021,
      "uri": "https://api.corespeed.io/artifacts/…",
      "expiresAt": "2026-09-10T14:02:11.000Z",
      "width": 1024,
      "height": 1024
    }
  ]
}

{ "job_id": "job_…", "status": "running", "model": "fal-ai/kling-video/v2" }

A running job keeps going on the server for up to 60 minutes, then expires unbilled. Collect it with media__get_result from any session — any agent in the organization can pick up a job another one started — and pass wait_seconds there to long-poll instead of spinning.

Three rules keep this from costing you twice:

• Do not set a client-side tool timeout equal to or below wait_seconds. That kills the very reply carrying your job_id. Disable the timeout for this call, or set a backstop of at least twice wait_seconds.
• If the client timed out anyway, nothing is lost. The newest entry in media__list_jobs is your job. Do not resubmit.
• Signed URLs expire after 12 hours. Call media__get_result again for a fresh set; the stored artifact does not go anywhere.

Job states are running, succeeded, failed, canceled, and expired. A terminal failure comes back as a tool-level error with generation_failed (or empty_output when the upstream finished without a file), carrying the upstream's reason in the text when it gave one; generation_canceled or generation_expired; an unknown id answers job_not_found. Every code is in Errors & status codes.

Speech and voicesCopy Anchor Link

The Fish Audio models in the catalog turn text into speech: fish-audio/tts/drama-3 (directed performance), fish-audio/tts/s2.1-pro and fish-audio/tts/s2-pro. Find them with media__list_models (query: "fish", or kind: "audio"). Their input takes the script as text, with any inline directions the model supports — [whispers], [shaky breath], <emphasis>…</emphasis>.

Choose the voice one of two ways:

1. A library voice. media__list_voices searches Fish Audio's public library by name and language; each row has a sample_url to listen to first. Pass its voice_id. Use licensed_only: true for commercial work: the rest of the library is community uploads, and some imitate real people. A licensed voice bills at a higher rate than the model alone.
2. Your own voice. Your own voice is an audio file in your Drive: a recording of one speaker, 10–30 seconds of clean speech (up to 10 MB). Pass it as reference_audio_url, with the exact words it says as reference_text. Fish Audio keeps nothing: the file is sent with each call, so keep it short. It bills at the model's rate.

Only use a voice you have the speaker's permission to use.

For dialogue, start each turn with <|speaker:0|>, <|speaker:1|>, … and pass one voice per speaker, in order: an array of voice_ids, or an array of reference_audio_url files with an array of reference_texts, one each.

fish-audio/voice-design/v1 designs a new voice from a description and returns up to four short samples in your Drive. To speak with one, pass its file as reference_audio_url.

Transcribe: speech to textCopy Anchor Link

media__transcribe takes one media source — a Drive path or file id, or a public https URL — up to 60 minutes and 100 MB: WAV, MP3, AAC/M4A, FLAC, Ogg, WebM or MOV. Send a whole conversation in one call: speaker labels are consistent only within one answer.

The transcript is the text answer, with speakers marked inline (<|speaker:0|>) and emotion and sound cues such as [laughter] unless tag_audio_events is false. timestamps: true adds word-level segments and timed speaker_turns. language is only a hint; the language is detected either way. Long recordings take minutes to process.

Understand: ask about a fileCopy Anchor Link

media__understand takes a prompt and one media source — a file in your Drive (its path or file id), a public https URL, or a YouTube watch URL. mime_type is inferred from the URL extension or the origin's Content-Type, or carried by the Drive file.

• Video can be windowed with start_seconds / end_seconds. max_cost_credits converts a credit budget into a window at the per-second rate; it bounds the media input only, and an explicit end_seconds overrides it. Video is sampled at one frame per second, so fast action between frames can be missed, and timestamps stated in the answer are not guaranteed exact.
• Audio and images reject all three window parameters, because clipping cannot be enforced for them. The whole file is analysed up to the capability's context limit.

The result carries analyzed_seconds (an estimate — a paging hint, not a duration), truncated when the answer was cut at the output limit, and usage with the metered quantities the call settled for.

Upload a private fileCopy Anchor Link

For a file that only exists on local disk:

1. Call drive__create_upload with the exact mime_type and size_bytes. The result includes file_id, upload_url, the PUT method, the headers to send — including Authorization — expires_at, and lands_at: the path the file will have in your Drive.
2. PUT the raw bytes to upload_url with every returned header, within 10 minutes. The ticket is one-use, and the declared type and size are bound to it.
3. Use the path (or file_id) with media__understand or any file-typed media__generate input. The file sits under tmp/uploads/ for 7 days; drive__move it anywhere else to keep it.

Every output a generation produces is a Drive file too, under tmp/media/<job_id>/, for 7 days unless you move it.

Files up to 2 GB are accepted. A deployment edge may still reject a very large body with HTTP 413; treat that as non-retryable for the deployment rather than retrying the same upload.

Pricing and holdsCopy Anchor Link

• Generation is billed once, after success, at the model's rate times the usage the upstream actually reports. For calls whose size is only known afterwards, the final amount can differ from the rate card. Failures, cancels, and expiries never charge. For expensive kinds — video especially — check media__get_model first.
• Speech bills per UTF-8 byte of text, tags included, at the model's rate per 1,000 bytes — so a character in Chinese or Japanese costs three bytes. A licensed library voice adds a surcharge per 1,000 bytes on the text it speaks. Voice design bills once per call, however many samples it returns.
• Transcription bills 9 credits per minute of audio, per second of the whole recording, silence included.
• Understanding bills 0.5 credits per image (an oversized image counts as several), 12 credits per minute of video and 4.2 per minute of audio, both metered per second, plus about 1.1 credits per 100 answer tokens. The question itself is free. Failures never charge.

The live figures are in the tool descriptions your client shows from tools/list and in each model's price. Every charge lands itemized in the organization ledger; see Billing & credits. An organization billing hold or suspension stops a media call before it runs — as an isError result with payment_required or org_suspended — and API-key spend caps apply the same way.

Inspect in the dashboardCopy Anchor Link

Dashboard → Media shows everything the organization generated. Running jobs finish server-side and land there whichever agent started them. Turning the capability off in Capability controls hides the media__* tools from tools/list; stored artifacts and job receipts are unaffected.