fix(integrations): langchain add multimodal content transformation functions for images, audio, and files #5278
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
…eport-binary-data
Semver Impact of This PR🟢 Patch (bug fixes) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨
Bug Fixes 🐛
Documentation 📚
Internal Changes 🔧Release
Other
🤖 This preview updates automatically when you update the PR. |
| "file_id": content_block.get("file_id", ""), | ||
| } | ||
|
|
||
| # Handle legacy image_url format (OpenAI style) |
There was a problem hiding this comment.
langchain supports providing content in a provider-native format.
So, I'm wondering whether we are deliberately only handling langchain-standard blocks and OpenAI-formatted blocks?
See https://docs.langchain.com/oss/python/langchain/messages#message-content
| header, content = url.split(",", 1) | ||
| mime_type = header.split(":")[1].split(";")[0] if ":" in header else "" |
There was a problem hiding this comment.
I'm worried about inconsistencies in the URI decoding across different AI integrations. Can this be centralized?
…tive content formats
| "modality": "image", | ||
| "mime_type": file_data.get("mime_type", ""), | ||
| "uri": file_data.get("file_uri", ""), | ||
| } |
There was a problem hiding this comment.
Google-style formats ignore block type, hardcode "image" modality
Medium Severity
When a content block has a standard type like "audio", "video", or "file" but stores its data using Google's inline_data or file_data format (instead of base64, url, file_id, or source), the function falls through the standard type handler without returning. It then hits the Google-style handlers which hardcode modality to "image", discarding the correctly computed modality. For example, {"type": "audio", "inline_data": {"mime_type": "audio/mp3", "data": "..."}} would incorrectly get modality: "image" instead of "audio".
Description
Add more support on message types for
gen_ai.request.messagesIssues
Closes: https://linear.app/getsentry/issue/TET-1637/redact-images-langchain