Exploring Model Graders for Reinforcement Fine-Tuning

This guide is for developers and ML practitioners who already know their way around OpenAIʼs APIs, have a basic understanding of reinforcement fine-tuning (RFT), and wish to use their fine-tuned models for research or other appropriate uses. OpenAI’s services are not intended for the personalized treatment or diagnosis of any medical condition and are subject to our applicable terms.

Reinforcement fine-tuning (RFT) of reasoning models consists in running reinforcement learning on of top the models to improve their reasoning performance by exploring the solution space and reinforcing strategies that result in a higher reward. RFT helps the model make sharper decisions and interpret context more effectively.

In this guide, weʼll walk through how to apply RFT to the OpenAI o4-mini reasoning model, using a task from the life sciences research domain: predicting outcomes from doctor-patient transcripts and descriptions, which is a necessary assessment in many health research studies. We'll use a subset of the medical-o1-verifiable-problem dataset. You will learn key steps to take in order to succesfully run RFT jobs for your use-cases.

Here’s what we’ll cover:

• 1. Setup
• 2. Gathering the dataset
• 3. Benchmarking the base model
• 4. Defining your grader
• 5. Training
• 6. Using your fine-tuned model

1. Setup

Even strong reasoning models can miss the mark when it comes to expert-level behavior-especially in domains like medicine, where nuance and exactness matter. Imagine a model trying to extract ICD-10 codes from a transcript: even if it understands the gist, it may not use the precise terminology expected by medical professionals.

Other great candidates for RFT include topics like ledger normalization or tiering fraud risk- settings in which you want precise, reliable, and repeatable reasoning. Checkout our RFT use-cases guide for great examples.

In our case, weʼll focus on teaching o4-mini to become better at predicting the outcomes of clinical conversations and descriptions. Specifically, we want to see if RFT can boost the accuracy of the prediction.

Along the way, weʼll talk about how to write effective graders, how they guide the modelʼs learning, and how to watch out for classic reward-hacking pitfalls.

2. Gathering the Dataset

Letʼs start off by loading the dataset from Hugging Face. Weʼre interested in samples framed as a description of a patient case with an associated question, followed by the correct answer. These represent real world transcripts where a physician is summarizing a case and assigning an outcome. For any use-case, verifying the accuracy of the gold level answers is critical and requires careful consideration. Here, we will trust the dataset quality.

One of the advantages of RFT is that it doesnʼt need thousands of samples to start making a difference. Thanks to trajectory sampling and the feedback loop during training, the model learns not just correct behaviors, but also patterns to avoid. This means we can see solid gains even with small datasets.

For this run, weʼll randomly sample 100 training and 100 test examples and slightly normalize them.

We'll convert these samples to jsonl format, as expected by the reinforcement finetuning API.

Next up: we’ll see how the base model performs out of the box-and where there’s room to grow.

3. Benchmarking the Base Model

Before we fine-tune anything, we need to know where we’re starting from. Benchmarking gives us a clear picture of the model’s initial strengths and weaknesses-so we can later measure how far it’s come.

We’ll first lean on two simple yet powerful evaluators:

1. clinical_phrase_binary_grader - an exact-match checker.
2. clinical_phrase_grader - a softer, token-based similarity grader.

This combination lets us track both strict correctness and partial lexical overlap. The binary grader gives a crisp 0 or 1: did the model produce an exact match? The softer one gives more nuance-how close did the output come to the gold answer? We use both because outcomes are often phrased in multiple valid ways. For instance, a model might respond with “gouty arthritis” instead of “gout.” While a human evaluator could consider this partially acceptable, a strict string match would not. Combining exact and fuzzy scoring ensures a more accurate and fair assessment of model outputs.

We build a helper function to preprend the examples with a system prompt.

Then build a helper function to generate and store the model's predictions.

To generate the predictions, first make sure your API key is set:

export OPENAI_API_KEY=...

We now have predictions that are ready to be evaluated.
We'll build a helper function that allows us to easily swap in different scoring methods,

and then run the evaluations.

Visualizing the results allows us to spot trends and failure modes.

As observed above, typical failure modes fall into three categories:

1. Small differences and formatting issues, score >=0.8.
2. Partial lexical match, 0.3 < score < 0.8.
3. Lexically off-base, score < 0.3.

We can visualize the full score distribution on the training set.

Note: In practice, analyzing model errors at scale often involves a mix of manual review and automated methods-like tagging failure types or clustering predictions by score and content. That workflow is beyond the scope of this guide, but it's a valuable next step once you've identified broad patterns.

Let's compare with other models and prompts, and visualize scores.

We can see that the modelʼs performance has clear limits. In practice, iterating on the prompt often helps boost baseline results and get more out of the base model. However, in this case, our prompt engineering didnʼt lead to meaningful improvements-so we excluded those runs from the analysis.

A key requirement for RFT to work is that the base model demonstrates it can successfully complete the task for at least some examples right out of the gate. The initial accuracy of ~0.6 is a strong signal that RFT can boost performance. If the model never succeeds on your tasks, there is no training signal to hill climb on.

This evaluation process prepares us for the next step: guiding the model with structured, high-quality feedback from a grader.

4. Defining Your Grader

The grader defines the reward function that shapes model behavior during RFT. It provides examples of desired outputs-and penalizes undesirable ones. Designing an effective grader requires both principled structure and thoughtful domain insight, and is perhaps the most important task for successful RFT.

In this section, we will present 3 graders, show how they should be set up to fit the API, and discuss the results they yielded. We will then show how to actually launch an RFT task.

String based grader

We began with a dual grader using our earlier evaluation functions since it provides a distribution of scores that will be aligned with the lexical proximity of the prediction to the reference answer. It provided a starting point, but the signal wasnʼt rich enough for o4-mini to truly learn and improve, and a first experiment showed stagnant reward during the RFT run. For the API calls, you should build the python grading function as shown below.

Here is a snapshot of its training curves, where the green curve is the traning set reward and the blue curve is the test set reward:

Model Grader 1

To address this limitation, we introduced a more advanced approach: the model grader. A model-based grader lets us embed semantic understanding and nuance into the feedback. Thatʼs especially powerful when domain-specific synonyms or fuzzy reasoning are in play.

We used gpt-4.1 as our grader model, guided by a rubric that emphasized semantic fidelity: clinical synonymy, correct disease categorization, and conceptual alignment. Rather than focusing on superficial phrasing-e.g., "Is this the same string?"-the grader aimed to answer, "Does this reflect the correct outcome or phenomenon?"

To ensure the grader aligned with expert expectations, we evaluated it on a subset of base model predictions. For any production use-case, domain expert reviewers should verify that model assigned scores reflect preferred answer orderings and align with domain judgment. This typically involves confirming that the model grader correctly ranks predictions according to their validity. In the scope of this cookbook, we approximated this evaluation by using OpenAI o3 to check whether higher-quality predictions were consistently rewarded relative to their alternatives.

From these discussions of o3 , we iteratively update the model grader until the results are aligned.

To be submitted through the API, this is how the dictionary is built.

Accordingly, we set up the model grader locally to check the results of the models we will fine-tune next.

While the rubric initially delivered sensible feedback, the model soon uncovered a loophole and began reward-hacking. Scores shot up-sometimes by 20-30 percentage points-not because clinical accuracy improved but because the model padded its “one phrase” answers with synonyms, doses, and full management plans. You might see begin warfarin therapy **and** continue unfractionated heparin for ≥5 days, overlapping until the INR is in the therapeutic range (2–3) or chewable aspirin 325 mg stat plus nitroglycerin… instead of the required continue unfractionated heparin or aspirin respectively. Although the system prompt is explicit-“respond with exactly one phrase: the single most likely outcome or phenomenon”-these verbose outputs inflate lexical_similarity scores without precisely adding prediction value. This experience highlights the need to continuously inspect model outputs and remain vigilant for reward-hacking behaviours that can quietly distort evaluation metrics.

Here is a snapshot of its training curves (green is training reward, blue is test reward):

Model Grader 2

To mitigate this reward-hack, we refined the grader prompt by clarifying expectations, enforcing stricter output constraints, and supplying contrastive examples of correct versus incorrect behavior. Once again, we've iterated with o3, leveraging predictions from the base o4-mini and the previous fine-tuned model hacking examples, to design and validate our grader. Another important point of this updated grader is the reduction of the weight of the lexical_similarity, to ensure that clinical_similarity prevails.

The final result was a high-signal, domain-sensitive grader that guided the model toward more appropriate and concise predictions.

Note on cost: LLM graders incur token usage charges in addition to training compute. To manage costs effectively, we recommend:

1. Testing your grader locally on base model completions (and optionally synthetic ones) to ensure it aligns with your rubric or human preferences. When available, use flex processing for more efficient evaluation.
2. Starting with a small-scale RFT run to validate grader alignment and detect potential reward-hacking before scaling up.

Let's look at how to launch the training in the next step!

5. Training

Once your prompt and grader are finalized, you can proceed to training. This section shows how to launch RFT using your final grader-but naturally, you would have already run similar commands when experimenting with earlier grader versions to evaluate their performance.

We make sure the grader passed API test,

and upload the training and test sets to the OpenAI file system.

Let's now define the hyper-parameters for our run. We will be fine-tuning o4-mini, with the medium reasoning effort. This parameter will impact the duration by limiting the number of tokens the model uses to reason. We tune with a moderate compute multiplier and reasonable number of epochs, prioritizing efficiency and fast iteration. Additionally, we set the eval_samples parameter to 3 to make the validation curves more robust given the stochasticity of o4-mini’s outputs. Averaging across multiple samples reduces noise and helps reveal consistent patterns of learning.

You’ll want to tailor these depending on your budget, desired generalization, and dataset difficulty.

We are now ready to launch the run!

On the dashboard you can observe the reward plots - they let you watch overall performance improve across steps, while the per-grader charts break down specific components in the case of a multi_grader. Reasoning token usage trends (often decreasing as the model gets more confident) and step duration metrics give insight into efficiency. Grader latency and error count plots help ensure your grader stays performant and bug-free during the run.

Here is a snapshot of our training curves, where the green and orange curves are for the training set, while tbe blue and red curves are for the test subset:

During training, evaluation runs on the test set are logged directly to the Evaluation API. You can head there to track how your samples perform and get a sense of how predictions evolve over time.

6. Using Your Fine-Tuned Model

When training completes, you can call your new model by its model_id and benchmark its improvements. Expect sharper predictions!

Model's prediction scores

Let's compute the scores of our base and fine-tuned models for comparison.

We can now visualize them!

We see about a 5-point boost in accuracy after fine-tuning. Looking at the first few errors, the model tends to harshly penalize answers that are close but not clinically identical-like unfractionated heparin vs. enoxaparin. It also dings longer answers, even when they’re correct, like contrast enhanced computed tomography of the abdomen.

Looking at the distruibution of scores, we observe that RFT helped shift the model’s predictions out of the mid-to-low score zone (0.2-0.6) and into the high range (0.8-1.0). Since the grader emphasizes clinical similarity over lexical match, this shift reflects stronger medical reasoning-not just better phrasing-according to our expert grader. As seen in the (0.0-0.1) range, a handful of already weak predictions fell even further, hinting at a residual knowledge gap.

Note that, because the earlier combined_grader was designed to reward lexical correctness, its accuracy didnʼt improve much-which is expected. That gap reinforces why validating your model grader is critical, and why you should monitor for reward-hacking. In our case, we used o3 to spot-check grading behavior, but domain expert review is essential.

Model's reasoning

Another important point in the analysis of the fine-tuned model are the reasoning summaries. The model may provide key information throughout these summaries, and exploring them to understand where the model fails can drive updates in the model's and the grader's system prompts. Below, we show examples of such chain of thought summaries that the model produced to show its way of answering the question:

The fine-tuned model spends more reasoning tokens to think through the question. Let's visualize an example thanks to the reasoning summaries.

Base o4‑mini’s reasoning zooms straight to “abdominal CT scan,” mostly worrying about lowercase formatting and giving only a cursory “often the initial choice” justification. The finetuned model, meanwhile, first links the patient’s steatorrhea and alcohol history to chronic pancreatitis or cancer, weighs CT against MRCP and EUS, and explains why a contrast‑enhanced abdominal CT best reveals calcifications and structural change. The latter seems more careful, and seems to have learnt to break down the case description even more.

To push the scores further

Both the baseline o3 and our fine-tuned o4-mini sometimes scored zero on the same samples-a red flag that the reference labels may be wrong. Before adding more compute, invest in data quality: have a domain expert relabel the noisy slice, analyze the model's reasoning, then tighten the grader prompt. Clean, trusted data and methodical updates almost always buys more accuracy than extra epochs.

Conclusion

Weʼve looked at how to design graders that give o4-mini the kind of detailed feedback it needs during RFT. That signal is what helps the model actually learn and improve beyond the baseline. Model graders can be incredibly powerful for this-but only if theyʼre designed carefully. A sloppy grader or sloppy data can send the wrong signals and steer the model in the wrong direction.

You're now ready to apply reinforcement fine-tuning on your own models using the OpenAI API. Weʼre excited to see how you push the boundaries of reasoning and tool use with custom graders and smarter model behavior!

For troubleshooting or next steps, refer to the OpenAI fine-tuning documentation.