🎨 NeMo Data Designer 101: Structured Outputs and Jinja Expressions

📚 What you'll learn

In this notebook, we will continue our exploration of Data Designer, demonstrating more advanced data generation using structured outputs and Jinja expressions.

If this is your first time using Data Designer, we recommend starting with the first notebook in this 101 series.

👋 IMPORTANT – Environment Setup

• If you haven't already, follow the instructions in the README to install the necessary dependencies.

• You may need to restart your notebook's kernel after setting up the environment.

• In this notebook, we assume you have a self-hosted instance of Data Designer up and running.

• For deployment instructions, see the Installation Options section of the NeMo Data Designer documentation.

📦 Import the essentials

• The data_designer module of nemo_microservices exposes Data Designer's high-level SDK.

• The essentials module provides quick access to the most commonly used objects.

⚙️ Initialize the NeMo Data Designer Client

• NeMoDataDesignerClient is responsible for submitting generation requests to the microservice.

🎛️ Define model configurations

• Each ModelConfig defines a model that can be used during the generation process.

• The "model alias" is used to reference the model in the Data Designer config (as we will see below).

• The "model provider" is the external service that hosts the model (see the model config docs for more details).

• By default, the microservice uses build.nvidia.com as the model provider.

🏗️ Initialize the Data Designer Config Builder

• The Data Designer config defines the dataset schema and generation process.

• The config builder provides an intuitive interface for building this configuration.

• The list of model configs is provided to the builder at initialization.

🧑‍🎨 Designing our data

• We will again create a product review dataset, but this time we will use structured outputs and Jinja expressions.

• Structured outputs let you specify the exact schema of the data you want to generate.

• Data Designer supports schemas specified using either json schema or Pydantic data models (recommended).

We'll define our structured outputs using Pydantic data models

💡 Why Pydantic?

• Pydantic models provide better IDE support and type validation.

• They are more Pythonic than raw JSON schemas.

• They integrate seamlessly with Data Designer's structured output system.

Next, let's design our product review dataset using a few more tricks compared to the previous notebook.

⚡ Advanced Feature:

• Conditional parameters let you modify sampler behavior based on other column values.

• Notice how review_style changes when target_age_range == '18-25'.

Next, we will use more advanced Jinja expressions to create new columns.

Jinja expressions let you:

• Access nested attributes: {{ customer.first_name }}

• Combine values: {{ customer.first_name }} {{ customer.last_name }}

• Use conditional logic: {% if condition %}...{% endif %}

🔁 Iteration is key – preview the dataset!

1. Use the preview method to generate a sample of records quickly.

2. Inspect the results for quality and format issues.

3. Adjust column configurations, prompts, or parameters as needed.

4. Re-run the preview until satisfied.

📊 Analyze the generated data

• Data Designer automatically generates a basic statistical analysis of the generated data.

• This analysis is available via the analysis property of generation result objects.

🆙 Scale up!

• Happy with your preview data?

• Use the create method to submit larger Data Designer generation jobs.

⏭️ Next Steps

Check out the following notebook to learn more about:

• Seeding synthetic data generation with an external dataset