-
Notifications
You must be signed in to change notification settings - Fork 2.7k
Description
Description
When ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS is set to false (or OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT is disabled), the ADK's telemetry/tracing.py sets span attributes to empty Python dicts {} instead of valid OpenTelemetry attribute types.
OpenTelemetry span attributes can only be primitive types (bool, str, bytes, int, float) or sequences of those types. Setting a dict as an attribute value causes:
-
Warnings in the logs:
Invalid type dict for attribute 'gcp.vertex.agent.llm_response' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types Invalid type dict for attribute 'gcp.vertex.agent.llm_request' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types Invalid type dict for attribute 'gcp.vertex.agent.tool_response' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types Invalid type dict for attribute 'gcp.vertex.agent.tool_call_args' value. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or a sequence of those types -
Crashes in the
opentelemetry-exporter-gcp-loggingwhen trying to serialize spans:File ".../opentelemetry/exporter/cloud_logging/__init__.py", line 190, in _set_payload_in_log_entry struct.update(sanitized) ... ValueError: Unexpected type -
No traces appearing in Vertex AI Agent Engine's trace viewer, as the telemetry export fails silently.
Location of Bug
google/adk/telemetry/tracing.py - multiple locations:
- Line 152:
span.set_attribute('gcp.vertex.agent.tool_call_args', {}) - Line 182:
span.set_attribute('gcp.vertex.agent.tool_response', {}) - Line 268:
span.set_attribute('gcp.vertex.agent.llm_request', {}) - Line 293:
span.set_attribute('gcp.vertex.agent.llm_response', {}) - Line 349:
span.set_attribute('gcp.vertex.agent.data', {})
Suggested Fix
Change all instances of {} to '{}' (a JSON string representation of an empty object):
# Before (broken)
span.set_attribute('gcp.vertex.agent.tool_call_args', {})
# After (fixed)
span.set_attribute('gcp.vertex.agent.tool_call_args', '{}')Environment
- ADK Version: 1.21.0
- OpenTelemetry SDK: 1.37.0
- opentelemetry-exporter-gcp-logging: 1.11.0a0
- Deployment: Vertex AI Agent Engine (Reasoning Engine)
Steps to Reproduce
- Deploy an ADK agent to Vertex AI Agent Engine with telemetry enabled (
GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY=true) - Ensure
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANSisfalseor not capturing content - Run a query against the agent
- Check Cloud Logging for the warnings and
ValueError: Unexpected typeerrors - Observe that traces do not appear in the Vertex AI trace viewer