🎨 NeMo Data Designer 101: Seeding Synthetic Data Generation with an External Dataset

Note: In order to run this notebook, you must have the NeMo Data Designer microservice deployed locally via docker compose. See the deployment guide for more details.

In this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.

If this is your first time using Data Designer, we recommend starting with the first notebook in this 101 series.

💾 Install dependencies

IMPORTANT 👉 If you haven't already, follow the instructions in the README to install the necessary dependencies. Note you may need to restart your kernel after setting up the environment.

⚙️ Initialize the NeMo Data Designer (NDD) Client

• The NDD client is responsible for submitting generation requests to the Data Designer microservice.

• In this notebook, we connect to the managed service of data designer. Alternatively, you can connect to your own instance of data designer by following the deployment instructions here.

• If you have an instance of data designer running locally, you can connect to it as follows

  data_designer_client = DataDesignerClient(client=NeMoMicroservices(base_url="http://localhost:8080"))
  

🏗️ Initialize the Data Designer Config Builder

• The Data Designer config defines the dataset schema and generation process.

• The config builder provides an intuitive interface for building this configuration.

• You must provide a list of model configs to the builder at initialization.

• This list contains the models you can choose from (via the model_alias argument) during the generation process.

Note: The NeMo Data Designer Managed service has access to specific models. Please visit https://build.nvidia.com/nemo/data-designer to see the latest list of which models are available.

🏥 Download a seed dataset

• For this notebook, we'll change gears and create a synthetic dataset of patient notes.

• To steer the generation process, we will use an open-source symptom-to-diagnosis dataset.

• In this dataset, the input_text represents the patient_summary and the output_text represents the diagnosis

Note: At this time, we only support using a single file as the seed. If you have multiple files you would like to use as seeds, it is recommended you consolidated these into a single file.

🎨 Designing our synthetic patient notes dataset

• We set the seed dataset using the with_seed_dataset method.

• We use the shuffle sampling strategy, which shuffles the seed dataset before sampling.

• We set with_replacement=False, which limits our max number of records to 853, which is the number of records in the seed dataset.

👀 Preview the dataset

• Iteration is key to generating high-quality synthetic data.

• Use the preview method to generate 10 records for inspection.

⏭️ Next Steps

Check out the following notebooks to learn more about:

• Using Custom Model Configs