From Unlabeled Data to a Deployed Machine Learning Model: A SageMaker Ground Truth Demonstration for Object Detection

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

1. Introduction
2. Run a Ground Truth labeling job (time: about 4h)
   1. Prepare the data
   2. Specify the category
   3. Create the instruction template
   4. Create a private team to test your task [OPTIONAL]
   5. Define pre-built lambda functions for use in the labeling job
   6. Submit the Ground Truth job request
      1. Verify your task using a private team [OPTIONAL]
   7. Monitor job progress
3. Analyze Ground Truth labeling job results (time: about 20min)
   1. Postprocess the output manifest
   2. Plot class histograms
   3. Plot annotated images
      1. Plot a small output sample
      2. Plot the full results
4. Compare Ground Truth results to standard labels (time: about 5min)
   1. Compute accuracy
   2. Plot correct and incorrect annotations
5. Train an object detector using Ground Truth labels (time: about 15min)
6. Deploy the Model (time: about 20min)
   1. Create Model
   2. Batch Transform
7. Review

Introduction

This sample notebook takes you through an end-to-end workflow to demonstrate the functionality of SageMaker Ground Truth. We'll start with an unlabeled image data set, acquire bounding boxes for objects in the images using SageMaker Ground Truth, analyze the results, train an object detector, host the resulting model, and, finally, use it to make predictions. Before you begin, we highly recommend you start a Ground Truth labeling job through the AWS Console first to familiarize yourself with the workflow. The AWS Console offers less flexibility than the API, but is simple to use.

Cost and runtime

You can run this demo in two modes:

1. Set RUN_FULL_AL_DEMO = True in the next cell to label 1000 images. This should cost about $200 given the current Ground Truth pricing scheme. In order to reduce the cost, we will use Ground Truth's auto-labeling feature. Auto-labeling uses computer vision to learn from human responses and automatically create bounding boxes for the easiest images at a cheap price. The total end-to-end runtime should be about 6h.
2. Set RUN_FULL_AL_DEMO = False in the next cell to label only 100 images. This should cost $26. Since Ground Truth's auto-labeling feature only kicks in for datasets of 1000 images or more, this cheaper version of the demo will not use it. Some of the analysis plots might look awkward, but you should still be able to see good results on the human-annotated 100 images.

Prerequisites

To run this notebook, you can simply execute each cell in order. To understand what's happening, you'll need:

• An S3 bucket you can write to -- please provide its name in the following cell. The bucket must be in the same region as this SageMaker Notebook instance. You can also change the EXP_NAME to any valid S3 prefix. All the files related to this experiment will be stored in that prefix of your bucket.
• The S3 bucket that you use for this demo must have a CORS policy attached. To learn more about this requirement, and how to attach a CORS policy to an S3 bucket, see CORS Permission Requirement.
• Familiarity with Python and numpy.
• Basic familiarity with AWS S3.
• Basic understanding of AWS Sagemaker.
• Basic familiarity with AWS Command Line Interface (CLI) -- ideally, you should have it set up with credentials to access the AWS account you're running this notebook from.

This notebook has only been tested on a SageMaker notebook instance. The runtimes given are approximate. We used an ml.m4.xlarge instance in our tests. However, you can likely run it on a local instance by first executing the cell below on SageMaker and then copying the role string to your local copy of the notebook.

Run a Ground Truth labeling job

This section should take about 4 hours to complete.

We will first run a labeling job. This involves several steps: collecting the images we want annotated, creating instructions, and writing a labeling job specification. In addition, we highly recommend that you run a (free) mock job using a private workforce before you submit any job to the public workforce. This notebook will explain how to do that as an optional step. Using a public workforce, this section should take about 4 hours. However, this will vary depending on the availability of workers.

Prepare the data

We will first download images and labels of a subset of the Google Open Images Dataset. These labels were carefully verified. Later, we will compare Ground Truth annotations to these labels. Our dataset will consist of images of various species of bird.

If you chose RUN_FULL_AL_DEMO = False, then we will choose a subset of 100 images from this dataset. This is a diverse dataset of interesting images, and it should be fun for the human annotators to work with. You are free to ask the annotators to annotate any images you wish as long as the images do not contain adult content. In this case, you must adjust the labeling job request this job produces; please check the Ground Truth documentation.

We will copy these images to our local BUCKET and create a corresponding input manifest. The input manifest is a formatted list of the S3 locations of the images we want Ground Truth to annotate. We will upload this manifest to our S3 BUCKET.

Disclosure regarding the Open Images Dataset V4:

Open Images Dataset V4 is created by Google Inc. We have not modified the images or the accompanying annotations. You can obtain the images and the annotations here. The annotations are licensed by Google Inc. under CC BY 4.0 license. The images are listed as having a CC BY 2.0 license. The following paper describes Open Images V4 in depth: from the data collection and annotation to detailed statistics about the data and evaluation of models trained on it.

A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, T. Duerig, and V. Ferrari. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. arXiv:1811.00982, 2018. (link to PDF)

After running the cell above, you should be able to go to s3://BUCKET/EXP_NAME/images in the S3 console and see 1000 images (or 100 if you have set RUN_FULL_AL_DEMO = False). We recommend you inspect these images! You can download them to a local machine using the AWS CLI.

Specify the categories

To run an object detection labeling job, you must decide on a set of classes the annotators can choose from. At the moment, Ground Truth only supports annotating one OD class at a time. In our case, the singleton class list is simply ["Bird"]. To work with Ground Truth, this list needs to be converted to a .json file and uploaded to the S3 BUCKET.

You should now see class_labels.json in s3://BUCKET/EXP_NAME/.

Create the instruction template

Part or all of your images will be annotated by human annotators. It is essential to provide good instructions. Good instructions are:

1. Concise. We recommend limiting verbal/textual instruction to two sentences and focusing on clear visuals.
2. Visual. In the case of object detection, we recommend providing several labeled examples with different numbers of boxes.

When used through the AWS Console, Ground Truth helps you create the instructions using a visual wizard. When using the API, you need to create an HTML template for your instructions. Below, we prepare a very simple but effective template and upload it to your S3 bucket.

NOTE: If you use any images in your template (as we do), they need to be publicly accessible. You can enable public access to files in your S3 bucket through the S3 Console, as described in S3 Documentation.

Testing your instructions

It is very easy to create broken instructions. This might cause your labeling job to fail. However, it might also cause your job to complete with meaningless results if, for example, the annotators have no idea what to do or the instructions are misleading. At the moment the only way to test the instructions is to run your job in a private workforce. This is a way to run a mock labeling job for free. We describe how in Verify your task using a private team [OPTIONAL].

It is helpful to show examples of correctly labeled images in the instructions. The following code block produces several such examples for our dataset and saves them in s3://BUCKET/EXP_NAME/.

You should now be able to find your template in s3://BUCKET/EXP_NAME/instructions.template.

Create a private team to test your task [OPTIONAL]

This step requires you to use the AWS Console. However, we highly recommend that you follow it, especially when creating your own task with a custom dataset, label set, and template.

We will create a private workteam and add only one user (you) to it. Then, we will modify the Ground Truth API job request to send the task to that workforce. You will then be able to see your annotation job exactly as the public annotators would see it. You could even annotate the whole dataset yourself!

To create a private team:

1. Go to AWS Console > Amazon SageMaker > Labeling workforces
2. Click "Private" and then "Create private team".
3. Enter the desired name for your private workteam.
4. Select "Create a new Amazon Cognito user group" and click "Create private team."
5. The AWS Console should now return to AWS Console > Amazon SageMaker > Labeling workforces.
6. Click on "Invite new workers" in the "Workers" tab.
7. Enter your own email address in the "Email addresses" section and click "Invite new workers."
8. Click on your newly created team under the "Private teams" tab.
9. Select the "Workers" tab and click "Add workers to team."
10. Select your email and click "Add workers to team."
11. The AWS Console should again return to AWS Console > Amazon SageMaker > Labeling workforces. Your newly created team should be visible under "Private teams". Next to it you will see an ARN which is a long string that looks like arn:aws:sagemaker:region-name-123456:workteam/private-crowd/team-name. Copy this ARN into the cell below.
12. You should get an email from no-reply@verificationemail.com that contains your workforce username and password.
13. In AWS Console > Amazon SageMaker > Labeling workforces > Private, click on the URL under Labeling portal sign-in URL. Use the email/password combination from the previous step to log in (you will be asked to create a new, non-default password).

That's it! This is your private worker's interface. When we create a verification task in Verify your task using a private team below, your task should appear in this window. You can invite your colleagues to participate in the labeling job by clicking the "Invite new workers" button.

The SageMaker Ground Truth documentation has more details on the management of private workteams.

Define pre-built lambda functions for use in the labeling job

Before we submit the request, we need to define the ARNs for four key components of the labeling job: 1) the workteam, 2) the annotation consolidation Lambda function, 3) the pre-labeling task Lambda function, and 4) the machine learning algorithm to perform auto-annotation. These functions are defined by strings with region names and AWS service account numbers, so we will define a mapping below that will enable you to run this notebook in any of our supported regions.

See the official documentation for the available ARNs:

• Documentation for available pre-human ARNs for other workflows.
• Documentation for available annotation consolidation ANRs for other workflows.
• Documentation for available auto-labeling ARNs for other workflows.

Submit the Ground Truth job request

The API starts a Ground Truth job by submitting a request. The request contains the full configuration of the annotation task, and allows you to modify the fine details of the job that are fixed to default values when you use the AWS Console. The parameters that make up the request are described in more detail in the SageMaker Ground Truth documentation.

After you submit the request, you should be able to see the job in your AWS Console, at Amazon SageMaker > Labeling Jobs. You can track the progress of the job there. This job will take several hours to complete. If your job is larger (say 100,000 images), the speed and cost benefit of auto-labeling should be larger.

Verify your task using a private team [OPTIONAL]

If you chose to follow the steps in Create a private team, you can first verify that your task runs as expected. To do this:

1. Set VERIFY_USING_PRIVATE_WORKFORCE to True in the cell below.
2. Run the next two cells. This will define the task and submit it to the private workforce (you).
3. After a few minutes, you should be able to see your task in your private workforce interface Create a private team. Please verify that the task appears as you want it to appear.
4. If everything is in order, change VERIFY_USING_PRIVATE_WORKFORCE to False and rerun the cell below to start the real annotation task!

Monitor job progress

A Ground Truth job can take a few hours to complete (if your dataset is larger than 10000 images, it can take much longer than that!). One way to monitor the job's progress is through AWS Console. In this notebook, we will use Ground Truth output files and Cloud Watch logs in order to monitor the progress.

You can re-evaluate the next cell repeatedly. It sends a describe_labeling_job request which should tell you whether the job is completed or not. If it is, then 'LabelingJobStatus' will be 'Completed'.

The next cell extracts detailed information on how your job is doing. You can re-evaluate it at any time. It should give you:

• The number of human and machine-annotated images across the iterations of your labeling job.
• The training curves of any neural network training jobs launched by Ground Truth (only if you are running with RUN_FULL_AL_DEMO=True).
• The cost of the human- and machine-annotated labels.

To understand the pricing, study this document carefully. In our case, each human label costs $0.08 + 5 * $0.036 = $0.26 and each auto-label costs $0.08. If you set RUN_FULL_AL_DEMO=True, there is also the added cost of using SageMaker instances for neural net training and inference during auto-labeling. However, this should be insignificant compared to the other costs.

If RUN_FULL_AL_DEMO==True, then the job will proceed in multiple iterations.

• Iteration 1: Ground Truth will send out 10 images as 'probes' for human annotation. If these are successfully annotated, proceed to Iteration 2.
• Iteration 2: Send out a batch of MaxConcurrentTaskCount - 10 (in our case, 190) images for human annotation to obtain an active learning training batch.
• Iteration 3: Send out another batch of 200 images for human annotation to obtain an active learning validation set.
• Iteration 4a: Train a neural net to do auto-labeling. Auto-label as many data points as possible.
• Iteration 4b: If there is any data leftover, send out at most 200 images for human annotation.
• Repeat Iteration 4a and 4b until all data is annotated.

If RUN_FULL_AL_DEMO==False, only Iterations 1 and 2 will happen.

Analyze Ground Truth labeling job results

This section should take about 20 minutes to complete.

Once the job has finished, we can analyze the results. Evaluate the following cell and verify the output is 'Completed' before continuing.

The plots in the Monitor job progress section form part of the analysis. In this section, we will gain additional insights into the results, which are contained in the output manifest. You can find the location of the output manifest under AWS Console > SageMaker > Labeling Jobs > [name of your job]. We will obtain it programmatically in the cell below.

Postprocess the output manifest

Now that the job is complete, we will download the output manifest manfiest and postprocess it to create a list of output_images with the results. Each entry in the list will be a BoxedImage object that contains information about the image and the bounding boxes created by the labeling jobs.

Plot annotated images

In any data science task, it is crucial to plot and inspect the results to check they make sense. In order to do this, we will

1. Download the input images that Ground Truth annotated.
2. Separate images annotated by humans from those annoted via the auto-labeling mechanism.
3. Plot images in the human/auto-annotated classes.

We will download the input images to a LOCAL_IMAGE_DIR you can choose in the next cell. Note that if this directory already contains images with the same filenames as your Ground Truth input images, we will not re-download the images.

If your dataset is large and you do not wish to download and plot all the images, simply set DATASET_SIZE to a small number. We will pick a random subset of your data for plotting.

Plot a small output sample to understand the labeling mechanism

The following cell will create two figures. The first plots N_SHOW images as annotated by humans. The first column shows the original bounding boxes produced by the human labelers working on Amazon Mechanical Turk. The second column shows the result of combining these boxes to produce a consolidated label, which is the final output of Ground Truth for the human-labeled images. Finally, the third column shows the "true" bounding boxes according to the Open Images Dataset for reference.

The second plots N_SHOW images as annotated by the auto-labeling mechanism. In this case, there is no consolidation phase, so only the auto-labeled image and the "true" label are displayed.

By default, N_SHOW = 5, but feel free to change this to any small number.

Plot the resulting bounding boxes to a pdf

Finally, we plot the results to two large pdf files. You can adjust the number of rows_per_page and columns_per_page if you would like. With the default settings, the pdfs will display 25 images per page. Each page will contain images annotated either by human annotators or by the auto-labeling mechanism. The first, ground-truth-od-confidence.pdf, contains images sorted by the confidence Ground Truth has in its prediction. The second, ground-truth-od-miou.pdf, contains the same images, but sorted by the quality of the annotations compared to the standard labels from the Open Images Dataset. See the Compare Ground Truth results to standard labels section for more details.

We will only plot 10 each of the human- and auto-annotated images. You can set N_SHOW to another number if you want to only plot more of the images.

Compare Ground Truth results to standard labels

This section should take about 5 minutes to complete.

Sometimes we have an alternative set of data labels available. For example, the Open Images data has already been carefully annotated by a professional annotation workforce. This allows us to perform additional analysis that compares Ground Truth labels to the standard labels. When doing so, it is important to bear in mind that any image labels created by humans will most likely not be 100% accurate. For this reason, it is better to think of labeling accuracy as "adherence to a particular standard / set of labels" rather than "how good (in absolute terms) are the Ground Truth labels."

Compute mIoUs for images in the dataset

The following cell plots a histogram of the mean intersections-over-unions (mIoUs) between labels produced by Ground Truth and reference labels from the Open Images Dataset. The intersection over union, also known as the Jaccard index, of two bounding boxes is a measure of their similarity. Because each image can contain multiple bounding boxes, we take the mean of the IoUs to measure the success of the labeling for that image.

Visualize the results

It is useful to see what corresponds to a good or bad mIoU in practice. The following cell displays images with the highest and lowest mIoUs vs the standard labels for both the human- and auto-labeled images. As before, the Ground Truth bounding boxes are in blue and the standard boxes are in lime green.

In our example run, the images with the lowest mIoUs demonstrated that Ground Truth can sometimes outperform standard labels. In particular, many of the standard labels for this dataset contain only one large bounding box despite the presence of many small objects in the image.

Understand the relationship between confidence and annotation quality

During both human- and auto-labeling, Ground Truth produces confidence scores associated with its labels. These scores are used internally by Ground Truth in various ways. As an example, the auto-labeling mechanism will only ouput an annotation for an image when the confidence passes a dynamically-generated threshold.

In practice, Ground Truth is often used to annotate entirely new datasets for which there are no standard labels. The following cells show how the confidence acts as a proxy for the true quality of the annotations.

Once again, it is good to see some concrete examples. The next cell displays several of the human- and auto-labeled images with the highest confidence scores across the dataset.

See how the number of objects in the image affects confidence

The next cell produces two subplots:

• The left subplot shows the counts of images with different numbers of objects in the image on a log scale. Notice that humans are assigned to label more of the images with many boxes.

• The right subplot shows how the confidence associated with an image decreases as the number of objects in the image increases.

Train an object detection model using Ground Truth labels

At this stage, we have fully labeled our dataset and we can train a machine learning model to perform object detection. We'll do so using the augmented manifest output of our labeling job - no additional file translation or manipulation required! For a more complete description of the augmented manifest, see our other example notebook.

NOTE: Object detection is a complex task, and training neural networks to high accuracy requires large datasets and careful hyperparameter tuning. The following cells illustrate how to train a neural network using a Ground Truth output augmented manifest, and how to interpret the results. However, we shouldn't expect a network trained on 100 or 1000 images to do a phenomenal job on unseen images!

First, we'll split our augmented manifest into a training set and a validation set using an 80/20 split and save the results to files that the model will use during training.

Next, we'll upload these manifest files to the previously defined S3 bucket so that they can be used in the training job.

Setup

Here we define S3 file paths for input and output data, the training image containing the object detection algorithm, and instantiate a SageMaker session.

Now we create the SageMaker training job.

To check the progess of the training job, you can repeatedly evaluate the following cell. When the training job status reads 'Completed', move on to the next part of the tutorial.

Deploy the Model

Now that we've fully labeled our dataset and have a trained model, we want to use the model to perform inference.

Object detection only supports encoded .jpg and .png image formats as inference input for now. The output is in JSON format, or in JSON Lines format for batch transform.

This section involves several steps:

1. Create Model: Create model for the training output
2. Batch Transform: Create a transform job to perform batch inference.
3. Host the model for realtime inference: Create an inference endpoint and perform realtime inference.

Create Model

Batch Transform

We now create a SageMaker Batch Transform job using the model created above to perform batch prediction.

Download Test Data

First, let's download a test image that has been held out from the training and validation data.

Inspect the results

The following cell plots the predicted bounding boxes for our example image. You'll notice that inside the function get_predictions, we filter the output to only include bounding boxes with a confidence score above a certain threshold (in this case, 0.2). This is because the object detection model we have trained always ouputs a fixed number of box candidates, and we must include a cutoff to eliminate the spurious results.

Realtime Inference

We now host the model with an endpoint and perform realtime inference.

This section involves several steps:

1. Create endpoint configuration - Create a configuration defining an endpoint.
2. Create endpoint - Use the configuration to create an inference endpoint.
3. Perform inference - Perform inference on some input data using the endpoint.
4. Clean up - Delete the endpoint and model

Create Endpoint Configuration

Create Endpoint

The next cell creates an endpoint that can be validated and incorporated into production applications. This takes about 10 minutes to complete.

Perform inference

The following cell transforms the image into the appropriate format for realtime prediction, submits the job, receives the prediction from the endpoint, and plots the result.

Clean up

Finally, let's clean up and delete this endpoint.

Review

We covered a lot of ground in this notebook! Let's recap what we accomplished. First we started with an unlabeled dataset (technically, the dataset was previously labeled by the authors of the dataset, but we discarded the original labels for the purposes of this demonstration). Next, we created a SageMake Ground Truth labeling job and generated new labels for all of the images in our dataset. Then we split this file into a training set and a validation set and trained a SageMaker object detection model. Next, we trained a new model using these Ground Truth results and submitted a batch job to label a held-out image from the original dataset. Finally, we created a hosted model endpoint and used it to make a live prediction for the same held-out image.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.