-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
feat(node): Add Claude Code Agent SDK instrumentation #17844
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: develop
Are you sure you want to change the base?
Conversation
size-limit report 📦
|
node-overhead report 🧳Note: This is a synthetic benchmark with a minimal express app and does not necessarily reflect the real-world performance impact in an application.
|
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this! For the first pass, the biggest lift here is to try to auto patch the functions we need automatically instead of asking user to import patched method, then we can move to tackling the other TODOs you have
Thanks SO much for all these. I'll get started on them. I tried REALLY hard to figure out how to hook into the existing query, and I couldn't get it to work no matter what I tried. I'll chat with you in slack on it, but I'd love some advice / guidance. I tried a bunch of different angles - but each time I ran into effectively timing issues where we couldn't hook fast enough. Felt like a limitation on how Claude Code's SDK works - but could be a total skill issue on my side. |
Hello @codyde, are you still working on this? if not, let's close this, we're trying to clean up the stale PRs |
I definitely am! I pushed up a few more commits today that included fixes for some of the other items you mentioned - but im struggling to get through this proxy one. I might need some pairing time to take a look at it together since im less familiar with the functionality. |
4df75cc to
6e267e0
Compare
Adds Sentry tracing instrumentation for the @anthropic-ai/claude-agent-sdk
following OpenTelemetry Semantic Conventions for Generative AI.
Key features:
- Captures agent invocation, LLM chat, and tool execution spans
- Records token usage, model info, and session tracking
- Supports input/output recording based on sendDefaultPii setting
- Provides createInstrumentedClaudeQuery() helper for clean DX
Due to ESM-only module constraints, this integration uses a helper function
pattern instead of automatic OpenTelemetry instrumentation hooks.
Usage:
```typescript
import { createInstrumentedClaudeQuery } from '@sentry/node';
const query = createInstrumentedClaudeQuery();
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
…g in Claude Code integration - Add SEMANTIC_ATTRIBUTE_SENTRY_OP to all span creation calls (invoke_agent, chat, execute_tool) - Capture exceptions to Sentry in catch block with proper mechanism metadata - Ensure child spans (currentLLMSpan, previousLLMSpan) are always closed in finally block - Prevents incomplete traces if generator exits early
…umentation
- Add OpenTelemetry-based automatic instrumentation via SentryClaudeCodeAgentSdkInstrumentation
- Extract ClaudeCodeOptions to dedicated types.ts file
- Remove backwards compatibility exports (patchClaudeCodeQuery, createInstrumentedClaudeQuery)
- Rename integration to claudeCodeAgentSdkIntegration
- Register instrumentation in OTEL preload for automatic patching
- Update NextJS re-exports to match simplified API
Users now only need:
```typescript
Sentry.init({ integrations: [Sentry.claudeCodeAgentSdkIntegration()] });
import { query } from '@anthropic-ai/claude-agent-sdk'; // Auto-instrumented
```
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude <[email protected]>
…ec compliance - Fix GEN_AI_SYSTEM_ATTRIBUTE to use 'anthropic' per OpenTelemetry semantic conventions - Add GEN_AI_REQUEST_AVAILABLE_TOOLS_ATTRIBUTE for capturing available tools from system init - Add GEN_AI_RESPONSE_FINISH_REASONS_ATTRIBUTE for tracking stop_reason - Use getTruncatedJsonString for proper payload truncation in span attributes - Expand tool categorization with new tools (KillBash, EnterPlanMode, AskUserQuestion, Skill, MCP tools) - Add better error metadata with function name in mechanism data - Export patchClaudeCodeQuery for manual instrumentation use cases - Add comprehensive integration tests for Claude Code Agent SDK instrumentation
6e267e0 to
3f69bfd
Compare
dev-packages/node-integration-tests/suites/tracing/claude-code/mock-server.mjs
Fixed
Show fixed
Hide fixed
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
dev-packages/node-integration-tests/suites/tracing/claude-code/scenario-tools.mjs
Show resolved
Hide resolved
RulaKhaled
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks a lot better! Thank you :)
I’ve left some comments, and I’ll continue the review next week—mainly to make sure we’re not missing any attributes and that the tests are passing. We’re so close to getting this in.
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/otel-instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/tracing/claude-code/instrumentation.ts
Outdated
Show resolved
Hide resolved
dev-packages/node-integration-tests/suites/tracing/claude-code/scenario.mjs
Outdated
Show resolved
Hide resolved
Follow best practice pattern used by other AI integrations (OpenAI, Anthropic) where options are passed directly to instrumentClaudeCodeAgentSdk() rather than exposed on the integration object.
- Remove patchClaudeCodeQuery from public exports to match pattern of other AI integrations (OpenAI, Anthropic, etc.) which only expose the integration - Change SENTRY_ORIGIN from 'auto.ai.claude-code' to 'auto.ai.claude_code' to follow Sentry naming conventions (underscores instead of hyphens) - Update integration tests to match new origin naming
- Remove claudeCodeAgentSdkIntegration from nextjs index.types.ts - Rename otel-instrumentation.ts to instrumentation.ts - Rename instrumentation.ts to helpers.ts (matches other AI integrations) - Fix prompt capture: use 'prompt' instead of 'inputMessages' per SDK API - Add cache token attribute support for tracking cached/cache_write tokens - Export new cache attributes from @sentry/core
- Fix startSpanManual usage to follow OpenAI/Anthropic pattern: - Use regular callback that returns the generator - Separate generator function handles span lifecycle in finally block - Add accumulative token usage on invoke_agent span - Clean up console.logs in test scenario
…helper - Add patchClaudeCodeQuery to scenario-tools.mjs and scenario-errors.mjs - Export patchClaudeCodeQuery from index.ts for test usage - Replace Math.random() with deterministic session ID in mock-server.mjs
…us propagation - Each assistant message now creates its own chat span instead of merging multiple turns - Token usage is recorded on each individual chat span from the assistant message - Child spans (currentLLMSpan, previousLLMSpan) now inherit error status when parent fails - Moved token accumulation from result message to assistant message handling
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Accidental comment
RulaKhaled
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok can we fix lint issues/tests and resolve cursor comments? Meanwhile I will be testing this locally with both esm/cjs
- Export patchClaudeCodeQuery from packages/node/src/index.ts for public API access - Add handling for 'error' type messages from Claude Code SDK in the generator loop - Error messages now properly set encounteredError flag, capture exception, and set span status
Don't overwrite span error status with success at end of try block.
The unconditional span.setStatus({ code: 1 }) was overwriting any error
status set when processing 'error' type messages from the SDK.
| previousLLMSpan.end(); | ||
| previousLLMSpan = null; | ||
| previousTurnTools = []; | ||
| } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LLM spans get ended twice with multiple assistant messages
Low Severity
When handling multiple assistant messages, previousLLMSpan is assigned a span that was already ended on line 283-286. Then when the result message handler or next assistant message handler runs, it calls setStatus() and end() on previousLLMSpan without checking if the span is still recording. The finally block correctly uses isRecording() before ending spans, but the main flow at lines 245-250 and 385-391 lacks this check. This causes setStatus() and end() to be called on already-ended spans. While OpenTelemetry likely treats this as a no-op, it's inconsistent with the pattern used in the finally block and indicates a state management issue.
Additional Locations (1)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Fixed in commit db79f79 - Added isRecording() checks before calling setStatus() and end() on previousLLMSpan in both the assistant message handler (line 245) and result message handler (line 385). This matches the pattern used in the finally block and prevents operations on already-ended spans.
Prevent calling setStatus() and end() on already-ended spans when handling multiple assistant messages. This matches the pattern used in the finally block for consistency.
The invoke_agent span now captures the prompt parameter directly at span creation time when recordInputs is true. Previously, gen_ai.request.messages was only set from conversation_history in system messages, which doesn't exist in the real Claude Agent SDK (only prompt and options are accepted). This ensures the user's input is properly recorded on the invoke_agent span in production, matching how other AI SDK integrations (like LangGraph) handle input capture.
… Code SDK - Use original error object when handling SDK error messages to preserve stack trace instead of creating new Error with just the message - Add scenario-with-options.mjs that explicitly passes recordInputs/recordOutputs options to patchClaudeCodeQuery for proper test coverage
Summary
Adds Sentry tracing instrumentation for the
@anthropic-ai/claude-agent-sdk(Claude Code Agent SDK) following OpenTelemetry Semantic Conventions for Generative AI.This integration enables AI monitoring for Claude Code agents with comprehensive telemetry:
gen_ai.invoke_agent)gen_ai.chat)gen_ai.execute_tool)sendDefaultPii)Implementation
Uses automatic OpenTelemetry instrumentation via
import-in-the-middlehooks - the same pattern as other AI integrations (Anthropic, OpenAI, Vercel AI, etc.). When the integration is added, it automatically patches thequeryfunction from@anthropic-ai/claude-agent-sdk.Important: Sentry must be initialized before importing
@anthropic-ai/claude-agent-sdkfor auto-instrumentation to work.Usage
Options
recordInputsbooleansendDefaultPiirecordOutputsbooleansendDefaultPiiagentNamestring'claude-code'Captured Telemetry
Span Hierarchy
Attributes (OpenTelemetry GenAI Semantic Conventions)
gen_ai.system:anthropicgen_ai.operation.name:invoke_agent|chat|execute_toolgen_ai.agent.name: Custom orclaude-codegen_ai.request.model: Model identifiergen_ai.request.available_tools: Available tools from system initgen_ai.response.id: Response/session IDgen_ai.response.model: Actual model usedgen_ai.response.finish_reasons: Stop reasongen_ai.response.text: Response text (whenrecordOutputs: true)gen_ai.response.tool_calls: Tool calls made (whenrecordOutputs: true)gen_ai.tool.name: Tool name (e.g.,Read,Bash,WebSearch)gen_ai.tool.type:function|extension|datastoregen_ai.tool.input: Tool input (whenrecordInputs: true)gen_ai.tool.output: Tool output (whenrecordOutputs: true)gen_ai.usage.input_tokens: Input token countgen_ai.usage.output_tokens: Output token countgen_ai.usage.cache_creation_input_tokens: Cache creation tokensgen_ai.usage.cache_read_input_tokens: Cache read tokensTool Type Classification
Bash,Read,Write,Edit,Glob,Grep,Task,TodoWrite,NotebookEdit,SlashCommand,AskUserQuestion,Skill, etc.WebSearch,WebFetch,ListMcpResources,ReadMcpResourceManual Instrumentation
For advanced use cases where auto-instrumentation is not suitable,
patchClaudeCodeQueryis exported for manual patching:Testing
Includes comprehensive integration tests covering:
sendDefaultPii: truerecordInputs/recordOutputsoptions