Top 5 Agent Observability Platforms for Tracing LLM Tool Calls

Quick picks

Langfuse — A candidate for nested tool observations, configurable metric or score alerts, and a choice of Cloud or self-hosting. Confirm instrumentation and whether your alert needs require custom filtering or metrics.

LangSmith — Consider it when production-trace evaluation is important: evaluators can be filtered to runs invoking a specific tool, with configured feedback routed through project webhooks. Its Cloud regions are US and EU, and deployments cannot be migrated between regions.

Arize Phoenix — Its Agent and Tool span types and parent-child traces make it worth considering for teams seeking explicit tool attribution and deployment control. Self-hosting brings operational responsibilities, and the cited documentation does not establish built-in failed-tool-call alerts.

Braintrust — A fit to investigate for nested agent spans with per-step latency, token, and cost details, plus threshold alerts for error-rate changes, latency, and scores. Custom tool execution may need instrumentation; no dedicated preconfigured failed-tool-call alert is established by the cited sources.

Helicone — A possible starting point for gateway-based model request logging, with a self-hosting option. The cited quickstart does not establish nested tool-call spans, so verify your integration if you need detailed agent attribution.

Product shortlist

1. Langfuse

Langfuse

Langfuse is a flexible option for teams that want agent traces with explicit relationships between model decisions and tool activity. Its trace model supports nested tool observations under the agent or span that orchestrated them, which can make multi-step runs easier to inspect.

I like that teams can choose Cloud or self-hosting, and configure alerts around observation metrics or scores rather than relying only on manual trace review.

What I like

• Nested tool observations can preserve which agent step orchestrated a call and which generation requested it, helping teams investigate a run’s sequence.
• Alerts support configurable metrics, filters, thresholds, and evaluation windows, with notifications through Slack, webhooks, or GitHub Actions.
• Cloud regions are documented for the EU, US, and Japan; self-hosting gives teams control over their infrastructure.

Limitations

• Trace detail depends on suitable instrumentation and nesting; integrations may not capture every tool call identically.
• The alert documentation does not promise a ready-made failed-tool-call alert, so teams may need to filter observations and define the relevant metric. Cloud alert limits vary by plan.

My take

Langfuse stands out when trace attribution and deployment choice matter. I’d consider it for teams that want to inspect tool calls in the context of the agent step that triggered them, while keeping the option to manage their own infrastructure.

Best for

Teams that want nested agent and tool-call traces, configurable alerts, and a choice between regional Cloud deployment and self-hosting. Before adopting it, verify that your instrumentation captures the spans you need and that the data-handling arrangements fit your requirements.

Sources: What does a good trace look like?; Alerts; Data Regions & Availability.

2. LangSmith

LangSmith

LangSmith is worth considering if you want production traces tied to configurable evaluation, especially when you need to inspect runs that invoke particular tools. I like that its online evaluators can filter traces by tool use and send results through webhook automation, which gives teams a route from observed behavior to their own notification workflows.

Its deployment choices are another plus, though the residency details deserve careful review before you settle on Cloud.

What I like

• Online evaluators can focus on production runs that invoke a specific tool, helping teams assess tool-related behavior without treating every trace alike.
• Configured project webhooks can route evaluator feedback into an existing workflow.
• Cloud, Hybrid, and Self-Hosted deployment paths give teams options for how tracing is handled.

Limitations

• Webhook notifications require configuration, and the documented workflow is not a turnkey alert for every failed tool call.
• LangSmith Cloud regions are documented as US and EU, and a deployment and its underlying database cannot be migrated between regions.

My take

I would shortlist LangSmith when evaluation is part of the observability workflow, not just trace review. Filtering evaluators to runs that use a particular tool can help teams focus on relevant behavior, while webhook automation can route the resulting feedback for follow-up.

Best for

Teams that want configurable evaluation of production traces, including runs involving specific tools, and can verify that LangSmith’s deployment and regional constraints suit their data requirements.

Sources: LangSmith data plane; Set up LLM-as-a-judge online evaluators.

3. Arize Phoenix

Arize Phoenix

Arize Phoenix is a good fit if you want a trace view that makes the relationship between an agent and its tool calls explicit. Its trace model uses nested spans, including Agent and Tool types, so you can follow tool activity within a broader request; you can run Phoenix yourself or use Phoenix Cloud.

What I like

• The Agent and Tool span types help distinguish agent work from calls to external tools.
• Parent-child spans show how tool activity fits into a multi-step request, which is useful when investigating where a run went wrong.
• Self-hosting lets your team choose the infrastructure and storage boundary, with SQLite available by default and PostgreSQL supported for production deployments.

Limitations

• Your application still needs suitable instrumentation to emit tool spans and errors; the trace model alone does not capture them.
• The cited documentation does not establish built-in production alerting for failed tool calls. Self-hosting also leaves deployment, scaling, and authentication responsibilities with your team.

My take

I like Phoenix when trace structure and deployment control matter more than having every operational detail handled for you. Separate instances and databases can support environment isolation, but they add infrastructure work.

Best for

Consider Phoenix if you need detailed, nested tool-call traces and want to control where a self-hosted deployment runs. Verify cloud data-residency options and failed-tool alerting requirements separately before choosing Phoenix Cloud or relying on it for production incident response.

Sources: What are Traces; Architecture.

4. Braintrust

Braintrust

Braintrust is worth considering if you need more than a record of the final model response. Its traces can show tool calls, retrieval, and other agent steps as nested spans, with per-step latency, token counts, and cost.

I like that this structure makes it easier to follow where a multi-step run spent time or encountered an error. Hosted, customer-cloud, and self-hosted data-plane options also give teams different ways to handle deployment and data-residency requirements.

What I like

• Nested spans can represent tool calls, retrieval, and other steps in an agent run, making the execution path easier to inspect.
• Trace details can include latency, token counts, and cost for individual steps, which helps teams investigate where a run used resources.
• Documented alerts can track thresholds for error-rate changes, latency, and scores.

Limitations

• Automatic capture depends on the chosen SDK or framework integration; custom tool execution may need additional instrumentation.
• The documentation describes general error-rate alerts, but not a dedicated, preconfigured alert specifically for failed tool calls. Verify how tool errors are recorded and filtered in your setup.
• A customer-cloud data plane brings infrastructure and operational requirements. Teams assessing residency should also confirm what metadata and telemetry reach the separate UI or control plane.

My take

I’d shortlist Braintrust when span-level attribution and deployment flexibility matter more than plug-and-play capture. Its nested trace view is a useful fit for investigating complex runs, while threshold alerts can help surface broader error-rate or latency changes.

The main caveat is that useful tool-call traces depend on instrumentation: teams with custom execution paths should confirm what their integration captures. For sensitive workloads, customer-cloud deployment may help keep logged AI data in your environment, but it is not a reason to skip a full review of control-plane data flows and operating needs.

Best for

Teams that want detailed, nested agent traces and metric-threshold alerts, especially those considering a customer-cloud data plane for residency needs. It may be less suitable if you require automatic capture for custom tool execution or a ready-made alert specifically for failed tool calls.

Sources: AI tracing | AI tracing tool | LLM logging; Observe; Architecture.

5. Helicone

Helicone

Helicone is worth considering if you want a relatively direct path to logging model requests, especially when your application can send supported-provider traffic through its AI Gateway. Its quickstart uses an OpenAI-compatible API and shows requests in a central dashboard.

I like the choice between cloud hosting and self-hosting, but there’s an important distinction for agent teams: request logs alone do not establish detailed traces of nested tool calls.

What I like

• The documented gateway integration and automatic request logging offer a straightforward starting point for seeing supported model requests.
• Self-hosting gives teams the option to run the observability service within their own infrastructure and choose where it is deployed.
• The documentation identifies a separate EU API endpoint for some operations, which may be relevant when assessing regional requirements.

Limitations

• The cited quickstart documents request-level visibility, not detailed nested agent spans or tool-call input/output attribution.
• The available sources do not establish a first-party alert for failed tool calls, so verify alerting needs against the current product documentation.
• Self-hosting puts deployment and ongoing maintenance on your team; verify current requirements and feature parity before committing.

My take

I’d treat Helicone as a promising fit when gateway-based request logging is the main need, rather than assume it provides the full agent trace you need. Before choosing it for multi-step debugging, check that your specific SDK or instrumentation captures each tool call as a distinct span with the inputs, outputs, and errors you need.

Best for

Best for teams seeking gateway-based model request logging, with a self-hosting option for infrastructure control. If detailed nested tool-call attribution or failed-call alerting is essential, verify those capabilities for your integration first.

Sources: Quickstart; Introducing Helicone Self-Hosting.

How to choose

How to Choose the Right Agent Observability Platform

Start with the trace you need: can you connect each tool call to the agent step that triggered it, and inspect the inputs, outputs, and errors? Confirm your framework or custom instrumentation emits that detail.

Langfuse recommends nesting tool observations under the agent or span that orchestrated them; Phoenix represents tool and agent activity with nested spans, but applications need suitable instrumentation to emit them. What does a good trace look like? · What are Traces

Then check deployment and alerting against your requirements. Langfuse documents regional cloud options and self-hosting; Braintrust supports threshold alerts for error-rate changes, but its documentation does not establish a dedicated, preconfigured failed-tool-call alert.

Cloud regions, self-hosting, and customer-cloud options differ, so verify the deployment and data-handling terms that apply to your use case. Data Regions & Availability · Observe

Common Mistakes to Avoid

Don’t assume a model request log is a nested agent trace, or that a platform captures custom tool execution automatically. Helicone’s cited quickstart documents request-level logging, while Braintrust notes that custom tool execution may require instrumentation.

Validate both with representative runs. Quickstart · AI tracing | AI tracing tool | LLM logging

Frequently Asked Questions

Will alerts catch every failed tool call? Not automatically; confirm how tool errors are recorded and whether you can filter and alert on them.

Is self-hosting always simpler for data residency? It offers infrastructure control, but your team takes on deployment and operational responsibilities.

Architecture

Final Verdict — Which One Would I Choose?

I’d choose based on verified trace detail first, then alerting and data-handling fit. Run a small integration check before committing, especially if failed-call alerts or residency controls are hard requirements.

Sources: What does a good trace look like?; Alerts; Data Regions & Availability; LangSmith data plane; Set up LLM-as-a-judge online evaluators; What are Traces; Architecture; AI tracing | AI tracing tool | LLM logging; Observe; Architecture; Quickstart; Introducing Helicone Self-Hosting.

Which option fits?

I’d start by deciding what your team must see in a trace, then shortlist platforms whose documented span model matches that need. For nested tool attribution, compare the trace structure in Langfuse, Phoenix, or Braintrust; if you also want production evaluation filtered by tool use, LangSmith is another option to assess.

Helicone may suit request-level gateway logging, but its cited quickstart is not evidence of detailed nested tool spans.

Before committing, send representative successful and failed runs through the intended integration. Check that tool inputs, outputs, errors, and parent-child relationships are actually captured, then test the alert workflow you need.

For sensitive workloads, review region or customer-managed deployment options alongside the data flows and operational work they entail.