Annotate Text Data Transformers Via Active Learning
Annotate text data using Active Learning with Cleanlab
Authored by: Aravind Putrevu
In this notebook, I highlight the use of active learning to improve a fine-tuned Hugging Face Transformer for text classification, while keeping the total number of collected labels from human annotators low. When resource constraints prevent you from acquiring labels for the entirety of your data, active learning aims to save both time and money by selecting which examples data annotators should spend their effort labeling.
What is Active Learning?
Active Learning helps prioritize what data to label in order to maximize the performance of a supervised machine learning model trained on the labeled data. This process usually happens iteratively — at each round, active learning tells us which examples we should collect additional annotations for to improve our current model the most under a limited labeling budget. ActiveLab is an active learning algorithm that is particularly useful when the labels coming from human annotators are noisy and when we should collect one more annotation for a previously annotated example (whose label seems suspect) vs. for a not-yet-annotated example. After collecting these new annotations for a batch of data to increase our training dataset, we re-train our model and evaluate its test accuracy.
In this notebook, I consider a binary text classification task: predicting whether a specific phrase is polite or impolite.
Active learning with ActiveLab is much better than random selection when it comes to collecting additional annotations for Transformer models. It consistently produces much better models with approximately 50% less error rate, regardless of the total labeling budget.
The rest of this notebook walks through the open-source code you can use to achieve these results.
Setting up the environment
Collecting and Organizing Data
Here we download the data that we need for this notebook.
Classifying the Politeness of Text
We are using Stanford Politeness Corpus as the Dataset.
It is structured as a binary text classification task, to classify whether each phrase is polite or impolite. Human annotators are given a selected text phrase and they provide an (imperfect) annotation regarding its politeness: 0 for impolite and 1 for polite.
Training a Transformer classifier on the annotated data, we measure model accuracy over a set of held-out test examples, where I feel confident about their ground truth labels because they are derived from a consensus amongst 5 annotators who labeled each of these examples.
As for the training data, we have:
X_labeled_full: our initial training set with just a small set of 100 text examples labeled with 2 annotations per example.X_unlabeled: large set of 1900 unlabeled text examples we can consider having annotators label.extra_annotations: pool of additional annotations we pull from when an annotation is requested for an example
Visualize Data
<ipython-input-6-d9c8ad254414>:5: DeprecationWarning: Sampling from a set deprecated
since Python 3.9 and will be removed in a subsequent version.
{k:extra_annotations[k] for k in random.sample(extra_annotations.keys(), 5)}
{'4235a537': {'a6': 0.0, 'a12': 0.0, 'a98': 0.0, 'a99': 0.0, 'a119': 0.0},
, '3d961d64': {'a68': 0.0, 'a70': 0.0, 'a79': 0.0, 'a99': 0.0, 'a199': 1.0},
, '4a5e75dc': {'a60': 1.0, 'a102': 1.0, 'a130': 1.0, 'a148': 1.0, 'a174': 1.0},
, '369a8b74': {'a65': 1.0, 'a68': 1.0, 'a71': 1.0, 'a157': 0.0, 'a161': 0.0},
, '356a4a74': {'a61': 0.0, 'a70': 1.0, 'a139': 0.0, 'a145': 1.0, 'a198': 1.0}} View Some Examples From Test Set
Impolite examples:
Polite examples:
Impolite Examples:
| text | |
|---|---|
| 120 | And wasting our time as well. I can only repeat: why don't you do constructive work by adding contents about your beloved Makedonia? |
| 150 | Rather than tell me how wrong I was to close certain afd's maybe your time would be better spent dealing with the current afd backlog |
| 326 | This was supposed to have been moved to |
Polite Examples:
| text | |
|---|---|
| 498 | Hi there, I've raised the possibility of unprotecting the tamazepam page |
| 132 | Due to certain Edits the page alignment has changed. Could you please help? |
| 131 | I'm glad you're pleased with the general appearance. Before I label all the streets, is the text size, font style, etc OK? |
Helper Methods
The following section contains all of the helper methods needed for this notebook.
get_idx_to_label is designed for use in active learning scenarios, particularly when dealing with a mixture of labeled and unlabeled data. Its primary goal is to determine which examples (from both labeled and unlabeled datasets) should be selected for additional annotations based on their active learning scores.
get_idx_to_label_random is designed for an active learning context where the selection of data points for additional annotation is done randomly rather than based on a model's uncertainty or learning scores. This approach might be used as a baseline to compare against more sophisticated active learning strategies or in scenarios where it's unclear how to score examples.
Below are some utility methods which helps us to compute standard deviation, selecting a specific annotator who has previously annotated the example, and some token functions to Tokenize text examples.
get_trainer function here is designed to set up a training environment for a text classification task using DistilBERT, a distilled version of the BERT model that is lighter and faster.
get_pred_probs function performs out-of-sample prediction probability computation for a given dataset using cross-validation, with additional handling for unlabeled data.
get_annotator function determines the most appropriate annotator to collect a new annotation from for a specific example, based on a set of criteria while get_annotation focused on collecting an actual annotation for a given example from a chosen annotator, it also deletes the collected annotation from the pool to prevent it from being selected again.
Run the following cell to hide the HTML output from the next model training block.
Methodology Used
For each active learning round we:
- Compute ActiveLab consensus labels for each training example derived from all annotations collected thus far.
- Train our Transformer classification model on the current training set using these consensus labels.
- Evaluate test accuracy on the test set (which has high-quality ground truth labels).
- Run cross-validation to get out-of-sample predicted class probabilities from our model for the entire training set and unlabeled set.
- Get ActiveLab active learning scores for each example in the training set and unlabeled set. These scores estimate how informative it would be to collect another annotation for each example.
- Select a subset (n = batch_size) of examples with the lowest active learning scores.
- Collect one additional annotation for each of the n selected examples.
- Add the new annotations (and new previously non-annotated examples if selected) to our training set for the next iteration.
I subsequently compare models trained on data labeled via active learning vs. data labeled via random selection. For each random selection round, I use majority vote consensus instead of ActiveLab consensus (in Step 1) and then just randomly select the n examples to collect an additional label for instead of using ActiveLab scores (in Step 6).
More intuition on Activelab Consensus labels and Active learning scores are shared further in the notebook.
Model Training and Evaluation
I first tokenize my test and train sets, and then initialize a pre-trained DistilBert Transformer model. Fine-tuning DistilBert with 300 training steps produced a good balance between accuracy and training time for my data. This classifier outputs predicted class probabilities which I convert to class predictions before evaluating their accuracy.
Use Active Learning Scores to Decide what to Label Next
During each round of Active Learning, we fit our Transformer model via 3-fold cross-validation on the current training set. This allows us to get out-of-sample predicted class probabilities for each example in the training set and we can also use the trained Transformer to get out-of-sample predicted class probabilities for each example in the unlabeled pool. All of this is internally implemented in the get_pred_probs helper method. The use of out-of-sample predictions helps us avoid bias due to potential overfitting.
Once I have these probabilistic predictions, I pass them into the get_active_learning_scores method from the open-source cleanlab package, which implements the ActiveLab algorithm. This method provides us with scores for all of our labeled and unlabeled data. Lower scores indicate data points for which collecting one additional label should be most informative for our current model (scores are directly comparable between labeled and unlabeled data).
I form a batch of examples with the lowest scores as the examples to collect an annotation for (via the get_idx_to_label method). Here I always collect the exact same number of annotations in each round (under both the active learning and random selection approaches). For this application, I limit the maximum number of annotations per example to 5 (don’t want to spend effort labeling the same example over and over again).
Adding new Annotations
The combined_example_ids are the ids of the text examples we want to collect an annotation for. For each of these, we use the get_annotation helper method to collect a new annotation from an annotator. Here, we prioritize selecting annotations from annotators who have already annotated another example. If none of the annotators for the given example exist in the training set, we randomly select one. In this case, we add a new column to our training set which represents the new annotator. Finally, we add the newly collected annotation to the training set. If the corresponding example was previously non-annotated, we also add it to the training set and remove it from the unlabeled collection.
We’ve now completed one round of collecting new annotations and retrain the Transformer model on the updated training set. We repeat this process in multiple rounds to keep growing the training dataset and improving our model.
Results
After running 25 rounds of active learning (labeling batches of data and retraining the Transformer model), collecting 25 annotations in each round. I repeated all of this, the next time using random selection to choose which examples to annotate in each round — as a baseline comparison. Before additional data are annotated, both approaches start with the same initial training set of 100 examples (hence achieving roughly the same Transformer accuracy in the first round). Because of inherent stochasticity in training Transformers, I ran this entire process five times (for each data labeling strategy) and report the standard deviation (shaded region) and mean (solid line) of test accuracies across the five replicate runs.
We see that choosing what data to annotate next has drastic effects on model performance. Active learning using ActiveLab consistently outperforms random selection by a significant margin at each round. For example, in round 4 with 275 total annotations in the training set, we obtain 91% accuracy via active learning vs. only 76% accuracy without a clever selection strategy of what to annotate. Overall, the resulting Transformer models fit on the dataset constructed via active learning have around 50% of the error-rate, no matter the total labeling budget!
When labeling data for text classification, you should consider active learning with the re-labeling option to better account for imperfect annotators.