🧑‍⚕️ NeMo Data Designer: Realistic Patient Data & Physician Notes

📚 What you'll learn

This notebook demonstrates how to use NeMo Data Designer to generate realistic patient data including physician notes.
We'll leverage both structured data generation and LLM capabilities to create a comprehensive medical dataset.

The dataset includes:

• Policy and claim details
• Policyholder and claimant information (PII)
• Claim descriptions and adjuster notes with embedded PII
• Medical information for relevant claims

👋 IMPORTANT – Environment Setup

• If you haven't already, follow the instructions in the README to install the necessary dependencies.

• You may need to restart your notebook's kernel after setting up the environment.

• In this notebook, we assume you have a self-hosted instance of Data Designer up and running.

• For deployment instructions, see the Installation Options section of the NeMo Data Designer documentation.

📦 Import the essentials

• The data_designer module of nemo_microservices exposes Data Designer's high-level SDK.

• The essentials module provides quick access to the most commonly used objects.

⚙️ Initialize the NeMo Data Designer Client

• NeMoDataDesignerClient is responsible for submitting generation requests to the microservice.

🎛️ Define model configurations

• Each ModelConfig defines a model that can be used during the generation process.

• The "model alias" is used to reference the model in the Data Designer config (as we will see below).

• The "model provider" is the external service that hosts the model (see the model config docs for more details).

• By default, the microservice uses build.nvidia.com as the model provider.

🏗️ Initialize the Data Designer Config Builder

• The Data Designer config defines the dataset schema and generation process.

• The config builder provides an intuitive interface for building this configuration.

• The list of model configs is provided to the builder at initialization.

🌱 Loading Seed Data

• We'll use the symptom-to-diagnosis dataset as our seed data.

• This dataset contains patient symptoms and corresponding diagnoses which will help generate realistic medical scenarios.

🌱 Why use a seed dataset?

• Seed datasets let you steer the generation process by providing context that is specific to your use case.

• Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.

• During generation, prompt templates can reference any of the seed dataset fields.

💡 About datastores

• You can use seed datasets from either the Hugging Face Hub or a locally deployed datastore.

• By default, we use the local datastore deployed with the Data Designer microservice.

• The datastore endpoint is specified in the deployment configuration.

👋 Note: At this time, we only support using a single file as the seed. If you have multiple files you would like to use as
seeds, it is recommended you consolidated these into a single file.

💡 Tip

• If the dataset already exists in the datastore, you can create the seed dataset reference directly.

• See the seed dataset docs for more info.

For example:

from nemo_microservices.data_designer.essentials import SeedDatasetReference

# Create reference to existing dataset in the datastore.
seed_dataset_reference = SeedDatasetReference(
    dataset="data-designer/demo/gretelai_symptom_to_diagnosis.csv",
    datastore_settings={"endpoint": "http://localhost:3000/v1/hf"},
)

🎲 Creating Person Samplers

• We create persona samplers to simulate details about the patient and the doctor

• The persona samplers allow you to sample realistic details of individuals using a model trained on the US Census.
  If the locale of the persona you are generating is anything other than en_US, then the personas will be generated using Faker

🏗️ Defining Data Structure

Now we'll define the structure of our dataset by adding columns for patient information, dates, and medical details. We'll use:

• uuid for patient identification
• Patient personal information (first_name, last_name, dob, patient_email)
• Medical timeline information (symptom_onset_date, date_of_visit)
• Physician information (physician)

🦜 LLM-Generated Physician Notes

The final and most complex column uses an LLM to generate realistic physician notes. We provide:

• Context about the patient and their condition
• Patient summary from our seed data
• Clear formatting instructions

This will create detailed medical notes that reflect the patient's diagnosis and visit information.

🔁 Iteration is key – preview the dataset!

1. Use the preview method to generate a sample of records quickly.

2. Inspect the results for quality and format issues.

3. Adjust column configurations, prompts, or parameters as needed.

4. Re-run the preview until satisfied.

📊 Analyze the generated data

• Data Designer automatically generates a basic statistical analysis of the generated data.

• This analysis is available via the analysis property of generation result objects.

🆙 Scale up!

• Happy with your preview data?

• Use the create method to submit larger Data Designer generation jobs.