Audit and Improve Video Annotation Quality Using Amazon SageMaker Ground Truth

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This notebook walks through how to evaluate the quality of video annotations received from SageMaker Ground Truth annotators using several metrics.

The standard functionality of this notebook works with the standard Conda Python3/Data Science kernel; however, there is an optional section that uses a PyTorch model to generate image embeddings.

Start by importing the required libraries and initializing the session and other variables used in this notebook. By default, the notebook uses the default Amazon S3 bucket in the same AWS Region you use to run this notebook. If you want to use a different S3 bucket, make sure it is in the same AWS Region you use to complete this tutorial, and specify the bucket name for bucket.

Prerequisites

Create some of the resources you need to launch a Ground Truth audit labeling job in this notebook. To execute this notebook, you must create the following resources:

• A work team: A work team is a group of workers that complete labeling tasks. If you want to preview the worker UI and execute the labeling task, you must create a private work team, add yourself as a worker to this team, and provide the following work team ARN. This GIF demonstrates how to quickly create a private work team on the Amazon SageMaker console. To learn more about private, vendor, and Amazon Mechanical Turk workforces, see Create and Manage Workforces.

• The IAM execution role you used to create this notebook instance must have the following permissions:

  • AmazonSageMakerFullAccess: If you do not require granular permissions for your use case, you can attach the AmazonSageMakerFullAccess policy to your IAM user or role. If you are running this example in a SageMaker notebook instance, this is the IAM execution role used to create your notebook instance. If you need granular permissions, see Assign IAM Permissions to Use Ground Truth for granular policy to use Ground Truth.
  • The AWS managed policy AmazonSageMakerGroundTruthExecution. Run the following code snippet to see your IAM execution role name. This GIF demonstrates how to attach this policy to an IAM role in the IAM console. For further instructions see the: Adding and removing IAM identity permissions section in the AWS Identity and Access Management User Guide.
  • Amazon S3 permissions: When you create your role, you specify Amazon S3 permissions. Make sure that your IAM role has access to the S3 bucket that you plan to use in this example. If you do not specify a S3 bucket in this notebook, the default bucket in the AWS region in which you are running this notebook instance is used. If you do not require granular permissions, you can attach AmazonS3FullAccess to your role.

• The S3 bucket that you use for this demo must have a CORS policy attached. To learn more about this requirement, and how to attach a CORS policy to an S3 bucket, see Video Frame Job Permission Requirements.

Download data

Download a dataset from the Multi-Object Tracking Challenge, a commonly used benchmark for multi-object tracking. Depending on your connection speed, this can take 5–10 minutes. Unzip it and upload it to a bucket in Amazon S3.

Disclosure regarding the Multiple Object Tracking Benchmark:

Multiple Object Tracking Benchmark is created by Anton Milan, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixe. We have not modified the images or the accompanying annotations. You can obtain the images and the annotations here. The images and annotations are licensed by the authors under Creative Commons Attribution-NonCommercial-ShareAlike 3.0 License. The following paper describes Multiple Object Tracking Benchmark in depth: from the data collection and annotation to detailed statistics about the data and evaluation of models trained on it.

MOT17: A Benchmark for Multi-Object Tracking. Anton Milan, Ian Reid, Stefan Roth, Konrad Schindler, Laura Leal-Taixe arXiv:1603.00831

View images and labels

The scene is a street setting with a large number of cars and pedestrians. Grab image paths and plot the first image.

Load labels

The MOT17 dataset has labels for each scene in a single text file. Load the labels and organize them into a frame-level dictionary so you can easily plot them.

View MOT17 annotations

In the existing MOT-17 annotations, the labels include both bounding box coordinates and unique IDs for each object being tracked. By plotting the following two frames, you can see how the objects of interest persist across frames. Since our video has a high number of frames per second, look at frame 1 and then frame 31 to see the same scene with approximately one second between frames. You can adjust the start index, end index, and step values to view different labeled frames in the scene.

Evaluate labels

For demonstration purposes, we've labeled three vehicles in one of the videos and inserted a few labeling anomalies into the annotations. Identifying mistakes and then sending directed recommendations for frames and objects to fix makes the label auditing process more efficient. If a labeler only has to focus on a few frames instead of a deep review of the entire scene, it can drastically improve speed and reduce cost."

We have provided a JSON file containing intentionally flawed labels. For a typical Ground Truth Video job, this file is in the Amazon S3 output location you specified when creating your labeling job. This label file is organized as a sequential list of labels. Each entry in the list consists of the labels for one frame.

For more information about Ground Truth's output data format, see the Output Data section of the Amazon SageMaker Developer Guide.

View annotations

We annotated 3 vehicles, one of which enters the scene at frame 9. View the scene starting at frame 9 to see all of our labeled vehicles.

Analyze tracking data

Put the tracking data into a form that's easier to analyze.

The following function turns our tracking output into a dataframe. You can use this dataframe to plot values and compute metrics to help you understand how the object labels move through the frames.

View label progression plots

The following plots illustrate how the coordinates of a given object progress through the frames of a video. Each bounding box has a left and top coordinate, representing the top-left point of the bounding box. It also has height and width values that represent the other 3 points of the box.

In the following plots, the blue lines represent the progression of our 4 values (top coordinate, left coordinate, width, and height) through the video frames and the orange lines represent a rolling average of these values. If a video has 5 frames per second or more, the objects within the video (and therefore the bounding boxes drawn around them) should have some amount of overlap between frames. Our video has vehicles driving at a normal pace, so our plots should show a relatively smooth progression.

You can also plot the deviation between the rolling average and the actual values of bounding box coordinates. You may want to look at frames in which the actual value deviates substantially from the rolling average.

Plot box sizes

Combine the width and height values to examine how the size of the bounding box for a given object progresses through the scene. For Vehicle 1, we intentionally reduced the size of the bounding box on frame 139 and restored it on frame 141. We also removed a bounding box on frame 217. We can see both of these flaws reflected in our size progression plots.

View box size differential

Now, look at how the size of the box changes from frame to frame by plotting the actual size differential to get a better idea of the magnitude of these changes.

You can also normalize the magnitude of the size changes by dividing the size differentials by the sizes of the boxes to express the differential as a percentage change from the original size of the box. This makes it easier to set thresholds beyond which you can classify this frame as potentially problematic for this object bounding box.

The following plots visualize both the absolute size differential and the size differential as a percentage. You can also add lines representing where the bounding box changed by more than 20% in size from one frame to the next.

If you normalize the size differential, you can use a threshold to identify which frames to flag for review. The preceding plot sets a threshold of 20% change from the previous box size; there a few frames that exceed that threshold.

View the frames with the largest size differential

With the indices for the frames with the largest size differential, you can view them in sequence. In the following frames, you can identify frames including Vehicle 1 where our labeler made a mistake. There was a large difference between frame 216 and frame 217, the subsequent frame, so frame 217 was flagged.

Rolling IoU

IoU (Intersection over Union) is a commonly used evaluation metric for object detection. It's calculated by dividing the area of overlap between two bounding boxes by the area of union for two bounding boxes. While it's typically used to evaluate the accuracy of a predicted box against a ground truth box, you can use it to evaulate how much overlap a given bounding box has from one frame of a video to the next.

Since there are differences from one frame to the next, we would not expect a given bounding box for a single object to have 100% overlap with the corresponding bounding box from the next frame. However, depending on the frames per second (FPS) for the video, there often is only a small change between one frame and the next since the time elapsed between frames is only a fraction of a second. For higher FPS video, we would expect a substantial amount of overlap between frames. The MOT17 videos are all shot at 25 FPS, so these videos qualify. Operating with this assumption, you can use IoU to identify outlier frames where you see substantial differences between a bounding box in one frame to the next.

Identify low overlap frames

With the IoU for your objects, you can set an IoU threshold and identify objects below it. The following code snippet identifies frames in which the bounding box for a given object has less than 50% overlap.

Visualize low overlap frames

With low overlap frames identified by the IoU metric, you can see that there is an issue with Vehicle 2 on frame 102. The bounding box for Vehicle 2 does not go low enough and clearly needs to be extended.

Embedding comparison (optional)

The preceding two methods work because they are simple and are based on the reasonable assumption that objects in high-FPS video won't move too much from frame to frame. They can be considered more classical methods of comparison.

Can we improve upon them? Try something more experimental to identify outliers: Generate embeddings for bounding box crops with an image classification model like ResNet and compare these across frames.

Convolutional neural network image classification models have a final fully connected layer using a softmax function or another scaling activation function that outputs probabilities. If you remove the final layer of your network, your "predictions" are the image embedding that is essentially the neural network's representation of the image. If you isolate objects by cropping images, you can compare the representations of these objects across frames to identify any outliers.

Start by importing a model from Torchhub and using a ResNet18 model trained on ImageNet. Since ImageNet is a very large and generic dataset, the network has learned information about images and is able to classify them into different categories. While a neural network more finely tuned on vehicles would likely perform better, a network trained on a large dataset like ImageNet should have learned enough information to indicate if images are similar.

Note: As mentioned at the beginning of the notebook, if you wish to run this section, you'll need to use a PyTorch kernel.

Generate embeddings

Use your headless model to generate image embeddings for your object crops. The following code iterates through images, generates crops of labeled objects, resizes them to 224x224x3 to work with your headless model, and then predicts the image crop embedding.

View image crops

To generate image crops, use the bounding box label dimensions and then resize the cropped images. Look at a few of them in sequence.

Compute distance

Compare image embeddings by computing the distance between sequential embeddings for a given object.

View outlier frames

In outlier frame crops, you can see that we were able to catch the issue on frame 102, where the bounding box was off-center.

While this method is fun to play with, it's substantially more computationally expensive than the more generic methods and is not guaranteed to improve accuracy. Using such a generic model will inevitably produce false positives. Feel free to try a model fine-tuned on vehicles, which would likely yield better results!

Combining the metrics

Having explored several methods for identifying anomalous and potentially problematic frames, you can combine them and identify all of those outlier frames. While you might have a few false positives, they are likely to be in areas with a lot of action that you might want our annotators to review regardless.

Command line interface

For use outside of a notebook, you can use the following command line interface.

Launch a directed audit job

Take a look at how to create a Ground Truth video frame tracking adjustment job. Ground Truth provides a worker UI and infrastructure to streamline the process of creating this type of labeling job. All you have to do is specify the worker instructions, labels, and input data.

With problematic annotations identified, you can launch a new audit labeling job. You can do this in SageMaker using the console; however, when you want to launch jobs in a more automated fashion, using the Boto3 API is very helpful.

To create a new labeling job, first create your label categories so Ground Truth knows what labels to display for your workers. In this file, also specify the labeling instructions. You can use the outlier frames identified above to give directed instructions to your workers so they can spend less time reviewing the entire scene and focus more on potential problems.

Generate manifests

SageMaker Ground Truth operates using manifests. When you use a modality like image classification, a single image corresponds to a single entry in a manifest and a given manifest contains paths for all of the images to be labeled. Because videos have multiple frames and you can have multiple videos in a single manifest, a manifest is instead organized with a JSON sequence file for each video that contains the paths to frames in Amazon S3. This allows a single manifest to contain multiple videos for a single job.

In this example, the image files are all split out, so you can just grab file paths. If your data is in the form of video files, you can use the Ground Truth console to split videos into video frames. To learn more, see Automated Video Frame Input Data Setup. You can also use other tools like ffmpeg to split video files into individual image frames. The following block stores file paths in a dictionary.

With your image paths, you can iterate through frames and create a list of entries for each in your sequence file.

With your sequence file, you can create your manifest file. To create a new job with no existing labels, you can simply pass in a path to your sequence file. Since you already have labels and instead want to launch an adjustment job, point to the location of those labels in Amazon S3 and provide metadata for those labels in your manifest.

Launch jobs (optional)

Now that you've created your manifests, you're ready to launch your adjustment labeling job. Use this template for launching labeling jobs via boto3. In order to access the labeling job, make sure you followed the steps to create a private work team.

Conclusion

This notebook introduced how to measure the quality of annotations using statistical analysis and various quality metrics like IoU, rolling IoU, and embedding comparisons. It also demonstrated how to flag frames which may not be labeled properly using these quality metrics and how to send those frames for verification and audit jobs using SageMaker Ground Truth.

Using this approach, you can perform automated quality checks on the annotations at scale, which reduces the number of frames humans need to verify or audit. Please try the notebook with your own data and add your own quality metrics for different task types supported by SageMaker Ground Truth. With this process in place, you can generate high-quality datasets for a wide range of business use cases in a cost-effective manner without compromising the quality of annotations.

Cleanup

Use the following command to stop your labeling job.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.