Weights & Biases Prompts is a suite of LLMOps tools built for the development of LLM-powered applications.

Use W&B Prompts to visualize and inspect the execution flow of your LLMs, analyze the inputs and outputs of your LLMs, view the intermediate results and securely store and manage your prompts and LLM chain configurations.

🪄 View Prompts In Action

In this notebook we will demostrate W&B Prompts:

• Using our 1-line LangChain integration
• Using our Trace class when building your own LLM Pipelines

See here for the full W&B Prompts documentation

Installation

Setup

This demo requires that you have an OpenAI key

W&B Prompts

W&B Prompts consists of three main components:

Trace table: Overview of the inputs and outputs of a chain.

Trace timeline: Displays the execution flow of the chain and is color-coded according to component types.

Model architecture: View details about the structure of the chain and the parameters used to initialize each component of the chain.

After running this section, you will see a new panel automatically created in your workspace, showing each execution, the trace, and the model architecture

Maths with LangChain

Set the LANGCHAIN_WANDB_TRACING environment variable as well as any other relevant W&B environment variables. This could includes a W&B project name, team name, and more. See wandb.init for a full list of arguments.

Create a standard math Agent using LangChain

Use LangChain as normal by calling your Agent.

You will see a Weights & Biases run start and you will be prompted to create a new API key at wandb.ai/settings if you haven't already. Store your API key securely. It can only be viewed once when created. Once you enter your API key, the inputs and outputs of your Agent calls will start to be streamed to the Weights & Biases App.

Once each Agent execution completes, all calls in your LangChain object will be logged to Weights & Biases

LangChain Context Manager

Depending on your use case, you might instead prefer to use a context manager to manage your logging to W&B.

✨ New: Custom columns can be logged directly to W&B to display in the same Trace Table with this snippet:

import wandb
wandb.log(custom_metrics_dict, commit=False})

Use commit=False to make sure that metadata is logged to the same row of the Trace Table as the LangChain output.

Non-Lang Chain Implementation

A W&B Trace is created by logging 1 or more "spans". A root span is expected, which can accept nested child spans, which can in turn accept their own child spans. A Span represents a unit of work, Spans can have type AGENT, TOOL, LLM or CHAIN

When logging with Trace, a single W&B run can have multiple calls to a LLM, Tool, Chain or Agent logged to it, there is no need to start a new W&B run after each generation from your model or pipeline, instead each call will be appended to the Trace Table.

In this quickstart, we will how to log a single call to an OpenAI model to W&B Trace as a single span. Then we will show how to log a more complex series of nested spans.

Logging with W&B Trace

Call wandb.init to start a W&B run. Here you can pass a W&B project name as well as an entity name (if logging to a W&B Team), as well as a config and more. See wandb.init for the full list of arguments.

You will see a Weights & Biases run start and be prompted to create a new API key at wandb.ai/settings if you haven't already. Store your API key securely. It can only be viewed once when created. Once you enter your API key, the inputs and outputs of your Agent calls will start to be streamed to the Weights & Biases App.

Note: A W&B run supports logging as many traces you needed to a single run, i.e. you can make multiple calls of run.log without the need to create a new run each time

You can also set the entity argument in wandb.init if logging to a W&B Team.

Logging a single Span

Now we will query OpenAI times and log the results to a W&B Trace. We will log the inputs and outputs, start and end times, whether the OpenAI call was successful, the token usage, and additional metadata.

You can see the full description of the arguments to the Trace class here.

Logging a LLM pipeline using nested Spans

In this example we will simulate an Agent being called, which then calls a LLM Chain, which calls an OpenAI LLM and then the Agent "calls" a Calculator tool.

The inputs, outputs and metadata for each step in the execution of our "Agent" is logged in its own span. Spans can have child

Once each Agent execution completes, all calls in your LangChain object will be logged to Weights & Biases