Langgraph Agent Tracing Tutorial
Tracing a LangGraph Agent
LangGraph provides tools to easily define a structured AI Agent. However, it can be challenging to understand what is going on under the hood and to pinpoint the cause of issues. Phoenix makes your LLM applications observable by visualizing the underlying structure of each call to your query engine and surfacing problematic "spans" of execution based on latency, token count, or other evaluation metrics.
In this tutorial, you will:
- Build a simple SQL database agent using LangGraph,
- Record trace data in OpenInference format,
- Inspect the traces and spans of your application to identify sources of latency and cost
ℹ️ This notebook requires an OpenAI API key.
Setup
First let's install our required packages and set our API keys
Configure the database
We will be creating a SQLite database for this tutorial. SQLite is a lightweight database that is easy to set up and use. We will be loading the chinook database, which is a sample database that represents a digital media store.
Find more information about the database here.
For convenience, we have hosted the database (Chinook.db) on a public GCS bucket.
We will use a handy SQL database wrapper available in the langchain_community package to interact with the database. The wrapper provides a simple interface to execute SQL queries and fetch results. We will also use the langchain_openai package to interact with the OpenAI API for language models later in the tutorial.
Utility functions
We will define a few utility functions to help us with the agent implementation. Specifically, we will wrap a ToolNode with a fallback to handle errors and surface them to the agent.
Define tools for the agent
We will define a few tools that the agent will use to interact with the database.
list_tables_tool: Fetch the available tables from the databaseget_schema_tool: Fetch the DDL for a tabledb_query_tool: Execute the query and fetch the results OR return an error message if the query fails
For the first two tools, we will grab them from the SQLDatabaseToolkit, also available in the langchain_community package.
The third will be defined manually. For the db_query_tool, we will execute the query against the database and return the results.
While not strictly a tool, we will prompt an LLM to check for common mistakes in the query and later add this as a node in the workflow.
Define the workflow
We will then define the workflow for the agent. The agent will first force-call the list_tables_tool to fetch the available tables from the database, then follow the steps mentioned at the beginning of the tutorial.
Using Pydantic with LangChain
This notebook uses Pydantic v2 BaseModel, which requires langchain-core >= 0.3. Using langchain-core < 0.3 will result in errors due to mixing of Pydantic v1 and v2 BaseModels.
Visualize the graph
Run the agent
Next Steps
You should now see traces in the Phoenix UI!
These traces contain information about the execution of the agent, including the tool calls, the LLM calls, and the overall execution path.
From here you can use the traces you've captured to:
- Visualize the execution path of your agent, track token usage, latency, and errors
- Evaluate various metrics of your agent's execution, such as the quality of its tool outputs and overall path
- Add annotations to traces to capture feedback or additional context on each execution
- Use your annotations and evaluations to create datasets to power experiments, or export for fine-tuning.