Notebooks
A
Arize AI
Langgraph Agent Tracing Tutorial

Langgraph Agent Tracing Tutorial

agentsllmsLlamaIndexarize-phoenixopenaitracingtutorialsevalsllmopsai-monitoringaiengineeringprompt-engineeringdatasetsllm-evalai-observabilityllm-evaluationsmolagentsanthropiclangchain

Open In Colab

phoenix logo
Docs | GitHub | Community

Tracing a LangGraph Agent

LangGraph provides tools to easily define a structured AI Agent. However, it can be challenging to understand what is going on under the hood and to pinpoint the cause of issues. Phoenix makes your LLM applications observable by visualizing the underlying structure of each call to your query engine and surfacing problematic "spans" of execution based on latency, token count, or other evaluation metrics.

In this tutorial, you will:

  • Build a simple SQL database agent using LangGraph,
  • Record trace data in OpenInference format,
  • Inspect the traces and spans of your application to identify sources of latency and cost

ℹ️ This notebook requires an OpenAI API key.

Setup

First let's install our required packages and set our API keys

[ ]
[ ]

Configure the database

We will be creating a SQLite database for this tutorial. SQLite is a lightweight database that is easy to set up and use. We will be loading the chinook database, which is a sample database that represents a digital media store. Find more information about the database here.

For convenience, we have hosted the database (Chinook.db) on a public GCS bucket.

[ ]

We will use a handy SQL database wrapper available in the langchain_community package to interact with the database. The wrapper provides a simple interface to execute SQL queries and fetch results. We will also use the langchain_openai package to interact with the OpenAI API for language models later in the tutorial.

[ ]

Utility functions

We will define a few utility functions to help us with the agent implementation. Specifically, we will wrap a ToolNode with a fallback to handle errors and surface them to the agent.

[ ]

Define tools for the agent

We will define a few tools that the agent will use to interact with the database.

  1. list_tables_tool: Fetch the available tables from the database
  2. get_schema_tool: Fetch the DDL for a table
  3. db_query_tool: Execute the query and fetch the results OR return an error message if the query fails

For the first two tools, we will grab them from the SQLDatabaseToolkit, also available in the langchain_community package.

[ ]

The third will be defined manually. For the db_query_tool, we will execute the query against the database and return the results.

[ ]

While not strictly a tool, we will prompt an LLM to check for common mistakes in the query and later add this as a node in the workflow.

[ ]

Define the workflow

We will then define the workflow for the agent. The agent will first force-call the list_tables_tool to fetch the available tables from the database, then follow the steps mentioned at the beginning of the tutorial.

Using Pydantic with LangChain

This notebook uses Pydantic v2 BaseModel, which requires langchain-core >= 0.3. Using langchain-core < 0.3 will result in errors due to mixing of Pydantic v1 and v2 BaseModels.

[ ]

Visualize the graph

[ ]

Auto-Instrument Agent

This example uses a hosted Phoenix instance. If you don't have one already, create one for free here to get your API key.

If you'd rather self-host Phoenix, you can do so by following the instructions here.

[ ]
[ ]

Run the agent

[ ]

Next Steps

You should now see traces in the Phoenix UI!

These traces contain information about the execution of the agent, including the tool calls, the LLM calls, and the overall execution path.

From here you can use the traces you've captured to: