Notebooks
H
Hugging Face Enterprise Cookbook Argilla

Enterprise Cookbook Argilla

hf-cookbookennotebooks
alph-notebooks/hf-cookbook / enterprise_cookbook_argilla.ipynb

Data Annotation with Argilla Spaces

Authored by: Moritz Laurer

This notebook illustrates the workflow for systematically evaluating LLM outputs and creating LLM training data. You can start by using this notebook to evaluate the zero-shot performance of your favorite LLM on your task without any fine-tuning. If you want to improve performance, you can then easily reuse this workflow to create training data.

Example use case: code generation. In this tutorial, we demonstrate how to create high-quality test and train data for code generation tasks. The same workflow can, however, be adapted to any other task relevant to your specific use case.

In this notebook, we:

  1. Download data for the example task.
  2. Prompt two LLMs to respond to these tasks. This results in "synthetic data" to speed up manual data creation.
  3. Create an Argilla annotation interface on HF Spaces to compare and evaluate the outputs from the two LLMs.
  4. Upload the example data and the zero-shot LLM responses into the Argilla annotation interface.
  5. Download the annotated data.

You can adapt this notebook to your needs, e.g., using a different LLM and API provider for step (2) or adapting the annotation task in step (3).

Install required packages and connect to HF Hub

[ ]
[ ]

Download example task data

First, we download an example dataset containing LLMs' code generation tasks. We want to evaluate how well two different LLMs perform on these code-generation tasks. We use instructions from the bigcode/self-oss-instruct-sc2-exec-filter-50k dataset that was used to train the StarCoder2-Instruct model.

[3]
Dataset structure:
 Dataset({
    features: ['fingerprint', 'sha1', 'seed', 'response', 'concepts', 'prompt', 'instruction', 'id'],
    num_rows: 3
}) 

Example instructions:
 ['Write a Python function named `get_value` that takes a matrix (represented by a list of lists) and a tuple of indices, and returns the value at that index in the matrix. The function should handle index out of range errors by returning None.', 'Write a Python function `check_collision` that takes a list of `rectangles` as input and checks if there are any collisions between any two rectangles. A rectangle is represented as a tuple (x, y, w, h) where (x, y) is the top-left corner of the rectangle, `w` is the width, and `h` is the height.\n\nThe function should return True if any pair of rectangles collide, and False otherwise. Use an iterative approach and check for collisions based on the bounding box collision detection algorithm. If a collision is found, return True immediately without checking for more collisions.']

Prompt two LLMs on the example task

Formatting the instructions with a chat_template

Before sending the instructions to an LLM API, we need to format the instructions with the correct chat_template for each of the models we want to evaluate. This essentially entails wrapping some special tokens around the instructions. See the docs on chat templates for details.

[4]
None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
/home/user/miniconda/lib/python3.9/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.
  warnings.warn(
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.


First prompt formatted for mistralai/Mixtral-8x7B-Instruct-v0.1:

 <s>[INST] Write a Python function named `get_value` that takes a matrix (represented by a list of lists) and a tuple of indices, and returns the value at that index in the matrix. The function should handle index out of range errors by returning None. [/INST] 


First prompt formatted for meta-llama/Meta-Llama-3-70B-Instruct:

 <|begin_of_text|><|start_header_id|>user<|end_header_id|>

Write a Python function named `get_value` that takes a matrix (represented by a list of lists) and a tuple of indices, and returns the value at that index in the matrix. The function should handle index out of range errors by returning None.<|eot_id|><|start_header_id|>assistant<|end_header_id|>

Sending the instructions to the HF Inference API

Now, we can send the instructions to the APIs for both LLMs to get outputs we can evaluate. We first define some parameters for generating the responses correctly. Hugging Face's LLM APIs are powered by Text Generation Inference (TGI) containers. See the TGI OpenAPI specifications here and the explanations of different parameters in the Transformers Generation Parameters docs.

[5]

Now, we can make a standard API request to the Serverless Inference API (docs). Note that the Serverless Inference API is mostly for testing and is rate-limited. For testing without rate limits, you can create your own API via the HF Dedicated Endpoints (docs). See also our corresponding tutorials in the Open Source AI Cookbook.

[!TIP] The code below will be updated once the Inference API recipe is finished.

[6]
  0%|          | 0/3 [00:00<?, ?it/s]
  0%|          | 0/3 [00:00<?, ?it/s]
---First generation of mistralai/Mixtral-8x7B-Instruct-v0.1:
Here's a Python function that meets your requirements:

```python
def get_value(matrix, indices):
    try:
        return matrix[indices[0]][indices[1]]
    except IndexError:
        return None
```

This function takes a matrix (represented by a list of lists) and a tuple of indices as input. It first tries to access the value at the given indices in the matrix. If the indices are out of range, it catches the `IndexError` exception and returns `None`.


---First generation of meta-llama/Meta-Llama-3-70B-Instruct:
Here is a Python function that does what you described:
```
def get_value(matrix, indices):
    try:
        row, col = indices
        return matrix[row][col]
    except IndexError:
        return None
```
Here's an explanation of how the function works:

1. The function takes two arguments: `matrix` (a list of lists) and `indices` (a tuple of two integers, representing the row and column indices).
2. The function tries to access the value at the specified indices using `matrix[row][col]`.
3. If the indices are out of range (i.e., `row` or `col` is greater than the length of the corresponding dimension of the matrix), an `IndexError` exception is raised.
4. The `except` block catches the `IndexError` exception and returns `None` instead of raising an error.

Here's an example usage of the function:
```
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

print(get_value(matrix, (0, 0)))  # prints 1
print(get_value(matrix, (1, 1)))  # prints 5
print(get_value(matrix, (3, 0)))  # prints None (out of range)
print(get_value(matrix, (0, 3)))  # prints None (out of range)
```
I hope this helps! Let me know if you have any questions.

Store the LLM outputs in a dataset

We can now store the LLM outputs in a dataset together with the original instructions.

[7]
Dataset({
,    features: ['instructions', 'response_model_1', 'response_model_2'],
,    num_rows: 3
,})

Create and configure your Argilla dataset

We use Argilla, a collaboration tool for AI engineers and domain experts who need to build high-quality datasets for their projects.

We run Argilla via a HF Space, which you can set up with just a few clicks without any local setup. You can create the HF Argilla Space by following these instructions. For further configuration on HF Argilla Spaces, see also the detailed documentation. If you want, you can also run Argilla locally via Argilla's docker containers (see Argilla docs).

Argilla login screen

Programmatically interact with Argilla

Before we can tailor the dataset to our specific task and upload the data that will be shown in the UI, we need to first set up a few things.

Connecting this notebook to Argilla: We can now connect this notebook to Argilla to programmatically configure your dataset and upload/download data.

[8]
[ ]

Write good annotator guidelines

Writing good guidelines for your human annotators is just as important (and difficult) as writing good training code. Good instructions should fulfill the following criteria:

  • Simple and clear: The guidelines should be simple and clear to understand for people who do not know anything about your task yet. Always ask at least one colleague to reread the guidelines to make sure that there are no ambiguities.
  • Reproducible and explicit: All information for doing the annotation task should be contained in the guidelines. A common mistake is to create informal interpretations of the guidelines during conversations with selected annotators. Future annotators will not have this information and might do the task differently than intended if it is not made explicit in the guidelines.
  • Short and comprehensive: The guidelines should as short as possible, while containing all necessary information. Annotators tend not to read long guidelines properly, so try to keep them as short as possible, while remaining comprehensive.

Note that creating annotator guidelines is an iterative process. It is good practice to do a few dozen annotations yourself and refine the guidelines based on your learnings from the data before assigning the task to others. Versioning the guidelines can also help as the task evolves over time. See further tips in this blog post.

[5]

Cumulative ratings vs. Likert scales: Note that the guidelines above ask the annotators to do cumulative ratings by adding points for explicit criteria. An alternative approach are "Likert scales", where annotators are asked to rate responses on a continuous scale e.g. from 1 (very bad) to 3 (mediocre) to 5 (very good). We generally recommend cumulative ratings, because they force you and the annotators to make quality criteria explicit, while just rating a response as "4" (good) is ambiguous and will be interpreted differently by different annotators.

Tailor your Argilla dataset to your specific task

We can now create our own code-llm task with the fields, questions, and metadata required for annotation. For more information on configuring the Argilla dataset, see the Argilla docs.

[ ]

After running the code above, you will see the new custom code-llm dataset in Argilla (and any other dataset you might have created before).

Load the data to Argilla

At this point, the dataset is still empty. Let's load some data with the code below.

[ ]

The Argilla UI for annotation will look similar to this:

Argilla UI

Annotate

That's it, we've created our Argilla dataset and we can now start annotating in the UI! By default, the records will be completed when they have 1 annotation. Check these guides, to know how to automatically distribute the annotation task and annotate in Argilla.

Important: If you use Argilla in a HF Space, you'd to activate persistent storage so that your data is safely stored and not automatically deleted after a while. For production settings, make sure that persistent storage is activated before making any annotations to avoid data loss.

Download annotated data

After annotating, you can pull the data from Argilla and simply store and process them locally in any tabular format (see docs here). You can also download the filtered version of the dataset (docs).

[ ]
[ ]

Next Steps

That's it! You've created synthetic LLM data with the HF inference API, created a dataset in Argilla, uploaded the LLM data into Argilla, evaluated/corrected the data, and after annotation you have downloaded the data in a simple tabular format for downstream use.

We have specifically designed the pipeline and the interface for two main use-cases:

  1. Evaluation: You can now simply use the numeric scores in the score_response_1 and score_response_2 columns to calculate which model was better overall. You can also inspect responses with very low or high ratings for a detailed error analysis. As you test or train different models, you can reuse this pipeline and track improvements of different models over time.
  2. Training: After annotating enough data, you can create a train-test split from the data and fine-tune your own model. You can either use highly rated response texts for supervised fine-tuning with the the TRL SFTTrainer, or you can directly use the ratings for preference-tuning techniques like DPO with the TRL DPOTrainer. See the TRL docs for the pros and cons of different LLM fine-tuning techniques.

Adapt and improve: Many things can be improved to tailor this pipeline to your specific use-cases. For example, you can prompt an LLM to evaluate the outputs of the two LLMs with instructions very similar to the guidelines for human annotators ("LLM-as-a-judge" approach). This can help further speed up your evaluation pipeline. See our LLM-as-a-judge recipe for an example implementation of LLM-as-a-judge and our overall Open-Source AI Cookbook for many other ideas.