Back to rankings

Gemini API: Function Responses Can Now Return Audio, Video and Office Files

Gemini CLIView original changelog

Tools can now return Office and OpenDocument files, audio and video to the model as function responses, not only text and images. In the GenerateContent API, function response parts accept these types plus Files API fileData URIs. In the Interactions API, function_result steps accept document, audio and video blocks.

Key Takeaways

  • Function responses are no longer limited to text and images: audio, video and Office or OpenDocument files can be returned.
  • GenerateContent function response parts accept the new media types and Files API fileData URIs.
  • displayName now works on both inlineData and fileData parts.
  • Video parts accept mediaProcessing set to STATIC or AGENTIC to choose how the video is handled.
  • Interactions API function_result steps accept document, audio and video blocks, inline or by URI.
  • Video blocks in the Interactions API are available in v1beta and v1alpha and take a processing option.

What Is New

Google expanded what a tool can hand back to a Gemini model after a function call. Previously a function result was limited mostly to text and images. Tools can now return Office and OpenDocument files, audio and video, so an agent can call a tool that produces a recording, a clip or a spreadsheet and have the model reason over it directly.

GenerateContent API

Function response parts now accept audio, video, and Office and OpenDocument files, as well as fileData URIs that point to files uploaded through the Files API. The displayName field works on both inlineData and fileData parts. Video parts also accept a mediaProcessing setting of STATIC or AGENTIC, which controls how the video is processed.

Interactions API

In the Interactions API, function_result steps now accept document, audio and video blocks alongside the existing text and image blocks. Content can be supplied as inline data or as a URI. Video blocks are available in v1beta and v1alpha and take a processing option.

Why It Matters

Agent workflows often need a tool to fetch or generate rich media, for example a transcription tool that returns audio or a rendering tool that returns a clip. Returning that media in the function response avoids extra upload steps.