Multiturn Conversation Finetuning

Introduction

In this cookbook we demonstrate how you can train your LLM to converse better by finetuning it on multi-step conversational data. This cookbook is part of a technical deep dive blogpost you can read here.

CoQA is a large-scale dataset for building Conversational Question Answering systems. The goal of the CoQA challenge is to measure the ability of machines to understand a text passage and answer a series of interconnected questions that appear in a conversation.

CoQA contains 127,000+ questions with answers collected from 8000+ conversations. Each conversation is collected by pairing two crowdworkers to chat about a passage in the form of questions and answers. CoQA has a lot of challenging phenomena not present in existing reading comprehension datasets, e.g., coreference and pragmatic reasoning.

Install Libraries

Prepare CoQA Dataset for Fine-tuning

Below we load and prepare the CoQA dataset for fine-tuning through the Together AI fine-tuning API.

The code below will format the data to a common conversational format, that can be used for fine-tuning the model.

Lets examine some rows from the CoQA dataset.

We can see that the source passage in which the questions and answers are observed in the story column. The questions column contains multiple questions and the answers column contains a dictionary of answers and also source citation of where the answer starts and ends in the source passage.

The goal of the CoQA challenge is to measure the ability of machines to understand a text passage and answer a series of interconnected questions that appear in a conversation.

This ability can be assessed using two metrics: F1 score, which measures word overlap between predicted and ground truth answers, and Exact Match (EM), which requires the prediction to exactly match one of the ground truth answers.

Format the data to conform with the chat format:

{
  "messages": [
    {"role": "system", "content": "You are a helpful AI chatbot."},
    {"role": "user", "content": "Hello, how are you?"},
    {"role": "assistant", "content": "I'm doing well, thank you! How can I help you?"},
    {"role": "user", "content": "Can you explain machine learning?"},
    {"role": "assistant", "content": "Machine learning is..."}
  ]
}

This list of messages can then be written to a .jsonl file.

To learn more about the conversation format please see our docs.

Fine-tune on Prepared Dataset using Together AI Fine-tuning API

Once the fine-tuning job is completed you will be able to see it in your Together AI dashbaord

You can also look at the WandB plots for the run(if a WANDB key was provided):

Evaluate Fine-tuned Model

For evaluation, CoQA uses two metrics:

• F1 score, which measures word overlap between predicted and ground truth answers
• Exact Match (EM), which requires the prediction to exactly match one of the ground truth answers.

F1 is the primary metric as it better handles free-form answers by giving partial credit for partially correct responses.

Deploy Model and Run Evals

Before we can run the evaluations we need to deploy our finetuned model as a Dedicated Endpoint(DE). After fine-tuning completes, access your model through the Together AI dashboard. Go to Models, select your fine-tuned model, and select Deploy. Choose from the available hardware options - we'll use a single A100-80GB GPU for this example.

For the evaluation above we saw a marked improvement in the LLMs ability to address conversational questions. The exact match score ~12x and the F1 score goes up ~3x after fine-tuning.

To learn more about our fine-tuning API read the docs here!