🎨 NeMo Data Designer: Japanese Commonsense Reasoning Dataset Generation

📚 Overview

This notebook generates synthetic datasets for the following tasks using NeMo Data Designer:

• jcommonsenseqa: Japanese commonsense question answering

Seed Data: Uses nvidia/Nemotron-Personas-Japan directly as the dataset

👋 Important – Environment Setup

• Ensure that NeMo Data Designer installation and configuration are completed
• Ensure that the local LLM server is running

📦 Import Required Modules

⚙️ Initialize NeMo Data Designer Client

🎛️ Define Model Configuration

📊 Prepare Seed Data

Load persona data from nvidia/Nemotron-Personas-Japan and pass it to Data Designer as a pandas DataFrame.

Define Target Count and Category Breakdown

Define target of 2000 total seeds with category-specific breakdowns.

• SEED_TARGET: 2000 total seeds
• WeakA: geo(250), tools(100), public(200), other(150) = 400 total
• WeakB (weakness reinforcement): finance(400), safety(350), vocab(350) = 1100 total
• Typical: Remaining 500 seeds
• Bias suppression: Max 10 per occupation, max 12 per prefecture

Handle Missing Values

Fill missing values in required columns with empty strings. Create columns with empty strings if they don't exist.

Text Construction

Combine multiple columns to construct text for classification.

• _all_text: Combine all columns
• _core_text: Combine core columns only (primary target for keyword matching)
• _core_len: Character count of core text

Create Duplicate Suppression Key

Create a key (_attr_key) for duplicate detection based on attribute combinations. This prevents selecting multiple similar personas.

Exclude completely empty keys (all fields empty).

Exclude by Negative Keywords

Exclude personas containing inappropriate keywords (extreme expressions, crime-related, etc.) unsuitable for JCommonsenseQA. Evaluate _core_text and remove matching entries.

Define Category Keyword Dictionaries

Define keywords for transportation/movement, daily life/housework, and tools.

• geo_kw: A_Transportation/Movement (trains, stations, buses, walking, etc.)
• life_kw: F_Daily Life/Housework (cooking, cleaning, shopping, etc.)
• tools_kw: B_Tools/Usage (knives, vacuum cleaners, stationery, etc.)

Define keywords for public facilities/manners, culture/etiquette, and finance.

• public_kw: D_Public Facilities/Manners (lines, order, priority seats, etc.)
• culture_kw: D_Public Facilities/Manners (etiquette, ceremonies, etc.)
• finance_kw: C_Payment/Money (accounting, banking, card payments, etc.)

Calculate Keyword Scores

Score how many keywords from each category are contained in the persona text.

• Primarily calculate scores using _core_text
• geo/tools only: recalculate by adding supplementary text (travel_persona, hobbies, etc.)
  • This prevents depletion of geo and tools categories and suppresses misclassification to public

Exclude Abnormal Scores and Estimate Categories

Exclude data with abnormally high scores (containing unnaturally many keywords) and estimate the most suitable category for each persona.

• Select the category with the highest score
• In case of ties, decide by priority (finance > safety > vocab > ...)

Determine Neutral Data

Classify personas with few keyword hits and shorter length as Neutral.

Conditions:

• Core text length is 260 characters or less
• Keyword hit count is 0
• Does not contain definition keywords (such as '~とは')

Limit Neutral to 50 entries to prevent too many thin seeds.

Create Sampling Pools

Create sampling pools for each category.

• typical_pool: Neutral and thin data (max_score ≤ 2)
• weakB_pool: Reinforcement targets (finance, safety, vocab)
• geo_pool, tools_pool, public_pool, other_pool: Each sub-category of WeakA

Define Sampling Function with Caps

Function that samples while suppressing bias by occupation and prefecture.

Operation:

1. First sample while respecting caps
2. If insufficient, relax caps to fill the remainder
3. Always ensure the specified count is met

Sample WeakB Categories

Sample weakness reinforcement targets (WeakB).

• finance: 400 entries
• safety: 350 entries
• vocab: 350 entries

Sample a total of 1100 entries, excluding already selected data from subsequent sampling.

WeakA - Sample Geo/Tools

Sample transportation/movement and tools categories from WeakA.

• geo (Transportation/Movement): 250 entries
• tools (Tools): 100 entries

WeakA - Sample Public/Other

Sample the remainder of WeakA.

• public (Public Facilities/Manners): 200 entries
• other (culture/life): 150 entries
  • For Other, prioritize those with public facility-related keywords
  • Suppress those with religion-related keywords (penalty)
  • This prevents category D from being biased toward religion

Sample Typical and Final Adjustments

Fill the remaining slots (approx. 500 entries) from the Typical category.

Process:

1. Sample remaining count from Typical pool
2. Combine all parts
3. If insufficient, add from unused data
4. If excess, adjust to 2000 entries
5. Always ensure exactly 2000 entries

Assign Themes and Check Distribution

Map categories to JCommonsenseQA themes (A-F, N) and check the final distribution.

Themes:

• A: Transportation/Movement
• B: Tools/Usage
• C: Payment/Money
• D: Public Facilities/Manners
• E: Safety/Danger
• F: Daily Life/Housework
• N: Neutral

Create and Save Final Output Data

Select columns needed for prompt generation and create the final seed data.

Output Columns:

• uuid, occupation, prefecture, region, marital_status
• age_band, skills_and_expertise_list
• jc_theme, jc_category, _attr_key

Theme-Based Topic Category Assignment

This code probabilistically assigns topic_category to each seed persona based on JCommonsenseQA themes (A-F, N).

Main Features

1. Define weights for each theme

   • Define assignment probabilities for topic categories (transportation, public places, daily life, etc.) for each theme (transportation/movement, tools/usage, etc.)
   • Example: A_Transportation/Movement → Transportation 60%, Public places 25%, Daily life 15%

2. Deterministic random number generation

   • stable_u01(): Always generates the same 0-1 value from UUID/attribute key (using MD5 hash)
   • Guarantees complete reproducibility as it always returns the same result for the same input

3. Weighted probability selection

   • pick_weighted(): Selects categories based on weights using cumulative probability method
   • Determines appropriate topic category based on 0-1 random value

4. Automatic key selection

   • Uses uuid column if it exists, otherwise uses _attr_key
   • Identifier for assigning unique stable random numbers to each row

Process Flow

Row data → Get jc_theme → Get weights for theme 
→ Generate hash value from UUID (0-1) → Weighted selection → Assign topic_category

Execution Result

A topic_category column is added to each seed persona, assigning topic categories (transportation, public places, daily life, shopping, school, meals, workplace) according to the theme.

CSV Text Normalization and Unterminated Quote Detection

Clean and normalize dataframe text in advance to prevent issues with unterminated quotes and control characters during CSV file output.

Overview

This code combines 4 normalization processes to clean up data:

1. Unify newline codes - Convert CRLF/CR to LF for cross-platform compatibility
2. Remove control characters - Delete invisible characters that hinder CSV parsing (preserve tabs and newlines)
3. Unicode safety - Handle corrupted characters and isolated surrogates
4. Quote escaping - Double quotes as needed (usually handled automatically by csv.writer)

Apply the clean_cell() function to all cells in all columns to convert the entire dataframe to a safe state.

Unterminated Quote Detection

The is_potential_unterminated_quote() function warns when the number of double quote occurrences in each row is odd. While not a complete detection, it functions as an inexpensive primary screening before CSV writing and can detect potential syntax errors early.

Filter Unterminated Quotes and Output CSV

Detect and exclude problematic rows with potential unterminated quotes, outputting only clean seed data to CSV file. Apply the is_potential_unterminated_quote() function to all rows to identify suspicious rows (odd number of double quotes), and extract only safe rows by logical inversion of the mask. Then remove judgment flag columns used in intermediate processing (is_neutral, _has_definition, has_neg_jc) to clean up the data, and save it in CSV format as the final seed data. This prevents CSV syntax errors and parsing failures, ensuring safety in downstream processing.

Note: The variable name is suspects, but it actually contains clean data after excluding suspicious rows.

🏗️ Define Data Structures

Data Structure for jcommonsenseqa

📝 Configuration 1: Using Seed Data

Uses the persona dataset directly as seed, and Data Designer automatically samples columns from the dataset.

Add jcommonsenseqa Generated Columns

Seed data columns can be directly referenced in the format {{ column_name }}.

🔍 Quality Evaluation Setup

Evaluate data quality using LLM-as-a-Judge

🔁 Generate Preview

First check quality with a small amount of data

🆙 Generate Production Data

If no issues in preview, generate a large-scale dataset

📊 Analyze Results

📈 Quality Comparison

Compare quality with and without seed data

💾 Save jcommonsenseqa Data

📋 Summary

What We Did

1. ✅ Correctly configured nvidia/Nemotron-Personas-Japan as seed data
2. ✅ Generated data by directly referencing seed data columns
3. ✅ Generated synthetic data for jcommonsenseqa
4. ✅ Created with seed data
5. ✅ Quality evaluation using LLM-as-a-Judge
6. ✅ Generated quality comparison report