Development · Tip
Multimodal AI Agent with Gemini 2.0 and Phidata
Python recipe for an agent handling image, audio, video, and web search
A concise Python code snippet demonstrating how to build a multimodal agent using Google's Gemini 2.0 Flash model and the Phidata (phi) framework. It combines image, audio, and video inputs with live DuckDuckGo web search into a single unified query. It serves as a practical starter template for complex multimodal reasoning workflows.
How it works
Full code (multimodal_ai_agent.py):
from phi.agent import Agent
from phi.model.google import Gemini
from phi.tools.duckduckgo import DuckDuckGo
from google.generativeai import upload_file, get_file
import time
# 1. Initialize the Multimodal Agent
agent = Agent(model=Gemini(id="gemini-2.0-flash-exp"), tools=[DuckDuckGo()], markdown=True)
# 2. Image Input
image_url = "https://example.com/sample_image.jpg"
# 3. Audio Input
audio_file = "sample_audio.mp3"
# 4. Video Input
video_file = upload_file("sample_video.mp4")
while video_file.state.name == "PROCESSING":
time.sleep(2)
video_file = get_file(video_file.name)
# 5. Multimodal Query
query = """
Combine insights from the inputs:
1. **Image**: Describe the scene and its significance.
2. **Audio**: Extract key messages that relate to the visual.
3. **Video**: Provide insights that connect with the image and audio context.
4. **Web Search**: Find the latest updates or events linking all these topics.
Summarize the overall theme or story these inputs convey.
"""
# 6. Multimodal Agent generates unified response
agent.print_response(query, images=[image_url], audio=audio_file, videos=[video_file], stream=True)
What it's for
- Analyze multimedia files and ground the analysis with live web search results
- Synthesize narratives across mixed media such as images, spoken audio, and recorded video
- Bootstrap a Python agent using Phidata and the Google Gemini API
Key points
- Uses Google Gemini 2.0 Flash experimental model via Phidata
- Integrates DuckDuckGo search tool for real-time web retrieval
- Demonstrates video upload and processing status polling via Google's File API
- Executes a unified prompt accepting simultaneous image, audio, and video inputs
- Streams structured markdown responses directly to the terminal
Pricing
Open source. Code template is free; relies on Gemini API which has free and paid tiers
Details
- Type
- Tip
- Category
- Development
- Topic
- AI Agents
- Open source
- Yes
- Platforms
- Code library
- For
- Python developers and AI engineers building multimodal applications
- Alternative to
- LangChain, LlamaIndex, CrewAI
- Tags
- python, gemini, ai agents, phidata, multimodal, llm
Why it was saved
- @Saboo_Shubham_: A working, copy-ready pattern for multimodal agents, kept because the code is complete and readable in the screenshot.
Compare
Related
Open directory and CLI for discovering and installing AI agent skills
Enterprise platform to build autonomous AI agents and automated business workflows.
Starter repository for building an autonomous social AI agent with Eliza
AI agent and chatbot platform for automated customer support and workflows.
Always-on cloud agent box with preinstalled browser and Telegram interface
Open-source graphical interface for running autonomous AI browser agents locally.
Updated