feat(anthropic): Record finish reasons in AI monitoring spans#5678
Open
ericapisani wants to merge 2 commits intomasterfrom
Open
feat(anthropic): Record finish reasons in AI monitoring spans#5678ericapisani wants to merge 2 commits intomasterfrom
ericapisani wants to merge 2 commits intomasterfrom
Conversation
Capture the stop_reason from Anthropic API responses and set it as GEN_AI_RESPONSE_FINISH_REASONS span data. Works for both streaming (via MessageDeltaEvent) and non-streaming responses. Co-Authored-By: Claude <noreply@anthropic.com>
Contributor
Semver Impact of This PR🟡 Minor (new features) 📋 Changelog PreviewThis is how your changes will appear in the changelog. New Features ✨Anthropic
Pydantic Ai
Other
Bug Fixes 🐛
Documentation 📚
Internal Changes 🔧Anthropic
Docs
Openai Agents
Other
🤖 This preview updates automatically when you update the PR. |
Contributor
Codecov Results 📊✅ 13 passed | Total: 13 | Pass Rate: 100% | Execution Time: 8.34s All tests are passing successfully. ❌ Patch coverage is 0.00%. Project has 14283 uncovered lines. Files with missing lines (1)
Generated by Codecov Action |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Unrelated
default_integrations=Falseadded to one test only- Removed the accidentally committed default_integrations=False parameter from test_streaming_create_message_async to match all other streaming tests.
Or push these changes by commenting:
@cursor push f7619664d2
Preview (f7619664d2)
diff --git a/tests/integrations/anthropic/test_anthropic.py b/tests/integrations/anthropic/test_anthropic.py
--- a/tests/integrations/anthropic/test_anthropic.py
+++ b/tests/integrations/anthropic/test_anthropic.py
@@ -508,7 +508,6 @@
sentry_init(
integrations=[AnthropicIntegration(include_prompts=include_prompts)],
traces_sample_rate=1.0,
- default_integrations=False,
send_default_pii=send_default_pii,
)
events = capture_events()This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.
| if response_id is not None: | ||
| span.set_data(SPANDATA.GEN_AI_RESPONSE_ID, response_id) | ||
| if finish_reasons is not None: | ||
| span.set_data(SPANDATA.GEN_AI_RESPONSE_FINISH_REASONS, finish_reasons) |
Contributor
There was a problem hiding this comment.
Since anthropic always generates a single output (i.e., no candidates) we can work with the single finish reason in _collect_ai_data (i.e., keep track of a string rather than a list).
If we only convert the finish reason to list form in _set_output_data the preceding code is easier to reason about.
Suggested change
| span.set_data(SPANDATA.GEN_AI_RESPONSE_FINISH_REASONS, finish_reasons) | |
| finish_reason: "str | None" = None, | |
| ) -> None: | |
| """ | |
| Set output data for the span based on the AI response.""" | |
| span.set_data(SPANDATA.GEN_AI_RESPONSE_MODEL, model) | |
| if response_id is not None: | |
| span.set_data(SPANDATA.GEN_AI_RESPONSE_ID, response_id) | |
| if finish_reason is not None: | |
| span.set_data(SPANDATA.GEN_AI_RESPONSE_FINISH_REASONS, [finish_reason]) |
Comment on lines
+226
to
+228
| stop_reason = getattr(event.delta, "stop_reason", None) | ||
| if stop_reason is not None: | ||
| finish_reasons = [stop_reason] |
Contributor
There was a problem hiding this comment.
non-blocking nitpick: We don't need the getattr(), since event.type == "message_delta" forces event to be an instance of MessageDeltaEvent.
Suggested change
| stop_reason = getattr(event.delta, "stop_reason", None) | |
| if stop_reason is not None: | |
| finish_reasons = [stop_reason] | |
| if event.delta.stop_reason is not None: | |
| finish_reason = event.delta.stop_reason |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add
GEN_AI_RESPONSE_FINISH_REASONSspan data to the Anthropic integration by capturingstop_reasonfrom API responses.For non-streaming responses, the
stop_reasonis read directly from theMessageresult. For streaming responses, it's extracted from theMessageDeltaEventdelta and passed through the_collect_ai_datahelper.This brings the Anthropic integration in line with the OpenAI integration's finish reason tracking.