Identify Worker Labeling Efficiency using SageMaker GroundTruth

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

Introduction

Welcome to our example on identifying worker labeling efficiency for a SageMaker GroundTruth Labeling job. Before running this notebook, please make sure that all the instructions prior to the section 'Setup the Automated Accuracy Logic' from this blog have been followed.

We first walk through some permissions and utility methods required for setting up. We then follow up with the logic of extracting individual worker responses and annotations from the output manifest and golden manifest to automatically calculate worker accuracy.

Import the necessary libraries

Permissions

As a sanity check, let us verify that this notebook is in the same region as the s3 bucket setup earlier following the instructions in the blog.

Initialize the s3 client

Utility methods

Calculate worker accuracy

Given the worker annotation and the true annotation, we calculate the accuracy of the worker. -1 is returned if we do not know the true annotation.

Extract and process responses from golden manifest file and output manifest file.

For each data object, we fetch the worker response, the golden manifest annotation (if available) and the output manifest annotation. We then calculate the worker accuracy against the output manifest annotation and the golden manifest annotation. This is stored in a dictionary indexed by worker ID.

Calculate the worker accuracy metrics and write to the given output location

This method is the starting point to this script and invokes other methods defined above. A sample invocation is shown in the following cells.

Below is a sample invocation of the script. Please replace with your own inputs.

• bucket: Please provide your bucket name. Our example bucket name is 'gec-sagemaker-blog'.
  

• labeling_job_output_location: Please make sure this is the same as provided while creating the labeling job. In our example this is 's3://gec-sagemaker-blog/output'.
  

• labeling_job_name: We can get the labeling job name from the Labeling jobs section under the SageMaker console.
  

• golden_answers: This is the golden manifest file. Each line is a key value pair in JSON format. The key is the line number of the data object in the input manifest file provided while creating the labeling job. The value is the true label of the data object. This is an optional parameter.
  

• worker_metrics_output: This s3 path of the output JSON file storing worker metrics. This file gets generated automatically. In our example this is 's3://gec-sagemaker-blog/worker_metrics.json'. Each entry in this JSON file is of the form where public.us-east-1.PDC is the worker ID, the total number of objects annotated by this worker is 3. We know the true labels for 2 out of these 3 objects. The golden manifest file need not store the true labels of all data objects. The worker correctly annotated the 2 data objects present in the golden manifest file with respect to the true annotation. The worker agreed with other workers for all 3 data objects.

    'public.us-east-1.PDC': {
            'Total Number Of Objects Annotated': 3,
            'Number Of Golden Standard Objects Annotated': 2,
            'Average Golden Standard Accuracy': 1.0,
            'Average Accuracy Compared To Other Workers': 1.0,
    }
   
  

We first run a sanity check to ensure that the bucket setup earlier as part of the blog instructions exists in the same region as this notebook. If the assertion fails, please cross check that the notebook is in the same region as the bucket. We then call the write_worker_metrics function with the inputs defined above.

Conclusion

In this notebook, we walked through analyzing the accuracy of workers for a completed Text Classification labeling job. We calculated the worker accuracy against the consoldiated output annotation and the golden annotation (when available). This shows us a measure of how often a worker agrees with the other workers as well as the the correctness of a worker with respect to the true annotation.

This script is setup to calculate worker accuracy for our classification modalities. However, it is possible to use a similar logic for other Ground Truth modalities as well. We use different accuracy metrics for different modalities, so code snippet calculating the worker accuracy metric would likely need to be updated. In case of classification, it is 0 or 1 but for modalities like bounding box we could use metrics like Intersection over Union. Please note that other utility methods like processing the worker responses will also need to be updated accordingly.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.