Example Intent Classification Pipeline
title: Guide - Building an intent classification pipeline description: Build an intent classification pipeline to understand how users are using your LLM application and how performance differs by intent. category: Classification
LLM Application Intent Classification
This guide demonstrates how to build an intent classification pipeline using Langfuse trace data. With both supervised and unsupervised approaches, you can automate the labeling and analysis of traces from your Langfuse projects.
Why is this useful?
- LLM applications often contain a number of mixed intents
- Breaking down evaluations by intent helps to identify which groups of traces perform badly in production
- Measuring the volume/intent is necessary to make sure that datasets used for offline/development evaluation are representative of production usage
You can approach intent classification in two ways:
- Supervised approach: You provide a model with labeled training data, and the model will output one of the pre-defined labels when making predictions.
- Unsupervised approach: The model attempts to find clusters within the data, which you can then label appropriately.
By the end of this notebook, you'll have two basic pipelines that will:
- Extract trace data from one of your Langfuse projects.
- Train an intent classification model.
- Predict the intent of traces using both supervised and unsupervised approaches.
- Upload predicted intent results back to Langfuse as tags.
Thank you @thompsgj for the contribution (pr) of this notebook to the Langfuse docs!
Setup
Begin by setting up your environment. First, disable unwanted warnings:
Install the necessary packages:
Configure your Langfuse project credentials (retrieve these from your Langfuse Project Settings):
Select an embedding model from the Sentence Transformers library:
Supervised intent classification pipeline (dummy data)
This section outlines the process of creating a supervised intent classification model with Langfuse trace data. The steps include retrieving trace data, using scikit-learn and sentence transformers to build and train the model, predicting intents, and tagging traces with labels in Langfuse. This method requires labeled data but ensures consistent predictions for predefined intents, ideal for clearly defined intent identification.
1. Retrieve Langfuse traces
Initialize the Langfuse client:
Optional: Create dummy trace data
If your project is empty, you can run the next two cells to create some simple dummy trace data to use for this notebook. The remainder of this section expects a trace with a "message" key in the input. You may need to adjust the notebook to your trace data's structure if you use data with another structure.
Fetch data from your project
{'id': '7e687860-55eb-47f0-b632-e568d5dfb57b',
, 'timestamp': datetime.datetime(2024, 10, 8, 7, 7, 51, 549000, tzinfo=datetime.timezone.utc),
, 'input': {'message': 'Please revert to the beginning'},
, 'tags': [],
, 'public': False,
, 'htmlPath': '/project/cm200q5d4003v6upt2pnmihyj/traces/7e687860-55eb-47f0-b632-e568d5dfb57b',
, 'latency': 0.0,
, 'totalCost': 0.0,
, 'observations': [],
, 'scores': [],
, 'bookmarked': False,
, 'projectId': 'cm200q5d4003v6upt2pnmihyj',
, 'createdAt': '2024-10-08T07:07:52.917Z',
, 'updatedAt': '2024-10-08T07:07:52.917Z',
, 'name': None,
, 'output': None,
, 'sessionId': None,
, 'release': None,
, 'version': None,
, 'userId': None,
, 'metadata': None,
, 'externalId': None} Construct a DataFrame for analysis:
2. Build and train an intent classification model
Prepare a small labeled dataset:
Split the data and define an embedding transformer:
array(['greeting', 'menu', 'menu', 'greeting', 'restart', 'greeting', , 'restart', 'menu', 'greeting', 'greeting', 'restart', 'greeting', , 'menu', 'menu', 'menu'], dtype=object)
array(['menu'], dtype=object)
array([[0.30275492, 0.39219684, 0.30504823]])
0.3921968431842116
Classification Report:
precision recall f1-score support
greeting 0.83 1.00 0.91 5
menu 0.67 1.00 0.80 4
restart 1.00 0.50 0.67 6
accuracy 0.80 15
macro avg 0.83 0.83 0.79 15
weighted avg 0.86 0.80 0.78 15
3. Run predictions on traces
4. Tag traces with labels
Tags in Langfuse
Unsupervised intent classification pipeline (production data)
The unsupervised intent classification pipeline demonstrates how to cluster and label Langfuse trace data without predefined categories. It uses embedding techniques, clustering algorithms, and LLM-generated labels to automatically identify and tag intents, offering flexibility for unlabeled data but potentially less consistency than supervised methods.
We will use sample data from the public demo project (RAG on Langfuse Documentation) to understand what people are most interested in when interacting with the demo application.
1. Fetch traces from Langfuse
We will fetch 15,000 messages sent to the demo application to create meaningful clusters.
2. Embed messages
Encoding batches: 100%|██████████| 30/30 [00:32<00:00, 1.09s/it]
3. Create clusters based on embeddings
{-1: 9005,
, 0: 544,
, 83: 438,
, 92: 396,
, 3: 298,
, 86: 215,
, 1: 155,
, 94: 147,
, 58: 146,
, 77: 133} 4. Derive cluster labels
5. Inspect the clusters
{'greeting': 2199,
, 'number_identifier': 544,
, 'end_conversation': 489,
, 'what_is_langfuse': 358,
, 'test': 299,
, 'unknown': 177,
, 'who_are_you': 101,
, 'what_can_you_do': 86,
, 'define_langfuse': 71,
, 'langfuse_usage': 66,
, 'ask_name': 51,
, 'how_it_works': 50,
, 'weather_inquiry': 46,
, 'summarize': 44,
, 'greetings': 43,
, 'open_source_query': 42,
, 'affirmation': 41,
, 'compare_langfuse_langsmith': 37,
, 'affirmative': 35,
, 'trace_in_langfuse': 34} {21: 'how can i use the langfuse.trace function ?',
, 828: 'What exactly is a trace in langfuse?',
, 1455: 'how does a trace look like in langfuse?',
, 1563: 'What langfuse uses to represent the traces/',
, 1744: 'i want to know about trace in langfuse',
, 1953: 'How does langfuse tracing work?',
, 2349: 'What is a trace in langfuse?',
, 2439: 'Hello! How exactly are traces created in langfuse',
, 3001: 'What is tracing in the context of Langfuse?',
, 3761: 'How does tracing work in Langfuse?',
, 4759: 'what is traces in simple language in langfuse',
, 4877: 'Hello, can you explain how does the tracing work in Langfuse?',
, 5547: 'What is a trace in Langfuse?',
, 5751: 'what is called traces in langfuse. explain clearly with eaxmple',
, 6508: 'how to trace using langfuse?',
, 6914: "what dose 'trace' means in langfuse",
, 6919: 'how do i look at traces in langfuse',
, 7275: 'what is tracing in Langfuse? what purpose does it serve?',
, 7585: 'what is a trace in langfuse and how to create it?',
, 7774: 'what is a trace in langfuse?'} 6. Add clusters as tags back to Langfuse
Conclusion
Each approach has its pros and cons.
The supervised approach requires a lot of effort upfront to prepare a labelled dataset of an appropriate size. During inference, it will only be able to assign labels that it was trained on, so it will not handle new cases well. However, the inference will be consistent.
The unsupervised approach offers more flexibility in working with unlabeled data. It can output a variety of new labels you didn't define beforehand. However, the labels may not be consistent between runs (ex., 'hello', 'greeting', or 'start_conversation'). Additionally, the clusters may be more or less permissive than if you had labelled the data.
Combining both approaches may be ideal. Unsupervised intent classification can help you quickly get an overview of a large volume of data, helping you with initial exploratory analysis. As you understand your trace data better and get more samples, you may benefit from running the supervised model on your data using the intent labels you most care about. Or, you may want to use the embedded data stored in the vector database to run similarity searches and reuse the labels from previous runs on new instances!