🎨 NeMo Data Designer: Generate Diverse RAG Evaluations

📚 What you'll learn

This tutorial demonstrates how to generate comprehensive evaluation datasets for Retrieval-Augmented Generation (RAG) systems, customized to your content and use cases.

👋 IMPORTANT – Environment Setup

• If you haven't already, follow the instructions in the README to install the necessary dependencies.

• You may need to restart your notebook's kernel after setting up the environment.

• In this notebook, we assume you have a self-hosted instance of Data Designer up and running.

• For deployment instructions, see the Installation Options section of the NeMo Data Designer documentation.

📦 Import the essentials

• The data_designer module of nemo_microservices exposes Data Designer's high-level SDK.

• The essentials module provides quick access to the most commonly used objects.

⚙️ Initialize the NeMo Data Designer Client

• NeMoDataDesignerClient is responsible for submitting generation requests to the microservice.

🎛️ Define model configurations

• Each ModelConfig defines a model that can be used during the generation process.

• The "model alias" is used to reference the model in the Data Designer config (as we will see below).

• The "model provider" is the external service that hosts the model (see the model config docs for more details).

• By default, the microservice uses build.nvidia.com as the model provider.

🏗️ Initialize the Data Designer Config Builder

• The Data Designer config defines the dataset schema and generation process.

• The config builder provides an intuitive interface for building this configuration.

• The list of model configs is provided to the builder at initialization.

🌱 Loading Seed Data

• We'll use the symptom-to-diagnosis dataset as our seed data.

• This dataset contains patient symptoms and corresponding diagnoses which will help generate realistic medical scenarios.

🌱 Why use a seed dataset?

• Seed datasets let you steer the generation process by providing context that is specific to your use case.

• Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.

• During generation, prompt templates can reference any of the seed dataset fields.

💡 About datastores

• You can use seed datasets from either the Hugging Face Hub or a locally deployed datastore.

• By default, we use the local datastore deployed with the Data Designer microservice.

• The datastore endpoint is specified in the deployment configuration.

👋 Note: At this time, we only support using a single file as the seed. If you have multiple files you would like to use as
seeds, it is recommended you consolidated these into a single file.

⚙️ Document Processing

Now we'll create a Document Processor class that handles loading and chunking the source documents.

This class uses langchain's RecursiveCharacterTextSplitter and unstructured.io for robust document parsing.

🏗️ Data Models

• Let's define Pydantic models for structured output generation.

• These schemas will ensure our generated data has consistent structure and validation.

🎲 Adding Categorical Columns for Controlled Diversity

Now we'll add categorical columns to control the diversity of our RAG evaluation pairs. We'll define:

1. Difficulty levels: easy, medium, hard

2. Reasoning types: factual recall, inferential reasoning, etc.

3. Question types: answerable vs. unanswerable (with weighting)

🦜 Adding LLM-Structured Column for Q&A Pair Generation

Now let's set up the core of our data generation: the Q&A pair column that will produce structured question-answer
pairs based on our document context and control parameters.

🔍 Quality Assessment: LLM-as-a-Judge

When generating our synthetic dataset, we need to determine the quality of the generated data
We use the LLM-as-a-Judge strategy to do this.

To do so, we need to define the rubric that the LLM should use to assess generation quality along with a prompt that provides relavant instructions.

🔁 Iteration is key – preview the dataset!

1. Use the preview method to generate a sample of records quickly.

2. Inspect the results for quality and format issues.

3. Adjust column configurations, prompts, or parameters as needed.

4. Re-run the preview until satisfied.

📊 Analyze the generated data

• Data Designer automatically generates a basic statistical analysis of the generated data.

• This analysis is available via the analysis property of generation result objects.

🆙 Scale up!

• Happy with your preview data?

• Use the create method to submit larger Data Designer generation jobs.