Amazon JumpStart Object Detection
Introduction to JumpStart - Object Detection
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
Welcome to Amazon SageMaker JumpStart! You can use JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through SageMaker JumpStart API.
In this demo notebook, we demonstrate how to use the JumpStart API for Object Detection. Object Detection refers to localizing objects in an image with a bounding box and predicting the classes of the objects. It can be used for counting objects, determining their exact locations and many more. It is applied to areas including security, surveillance, automated vehicle systems and machine inspection.
In this notebook, we demonstrate two use cases of object detection models:
- How to use pre-trained models trained on COCO dataset to do object detection.
- How to use JumpStart transfer learning algorithm to train an Object Detection model on a custom dataset.
Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.
1. Set Up
Before executing the notebook, there are some initial steps required for set up. This notebook requires latest version of sagemaker and ipywidgets
Permissions and environment variables
To train and host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook instance as the AWS account role with SageMaker access. It has necessary permissions, including access to your data in S3.
2. Run inference on the pre-trained model
Using JumpStart, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset.
2.1. Select a pre-trained model for inference
Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Object Detection models and select a model for inference.
Chose a model for Inference
2.2. Retrieve JumpStart Artifacts & Deploy an Endpoint
We retrieve the deploy_image_uri, deploy_source_uri, and base_model_uri for the pre-trained model. To host the pre-trained base-model, we create an instance of sagemaker.model.Model and deploy it.
2.3. Download an example image for inference
We download an example image from the JumpStart assets S3 bucket.
2.4. Query endpoint and parse response
Input to the endpoint is a single image in binary format. Response of the endpoint is a set of bounding boxes as well as class names and scores for the bounding boxes. JumpStart allows the flexibility in the number of bounding boxes returned. Below, we show to predict two bounding boxes per image by appending ;n_predictions=2 to Accept. To predict xx boxes, one can change it to ;n_predictions=xx or to get all the predicted boxes, one can remove ;n_predictions=2.
2.5. Display model predictions
Next, we display the bounding boxes overlaid on the original image.
2.6. Clean up the endpoint
3. Fine-tune the pre-trained model on a custom dataset
Previously, we saw how to run inference on a pre-trained model. Next, we discuss how a model can be finetuned to a custom dataset with any number of classes.
Transfer learning algorithm removes the object detection head of the pre-trained model and attaches a new randomly initialized head with number of classes same as the custom dataset. The fine-tuning step fine-tunes the last layer parameters while keeping the parameters of the rest of the model frozen, and returns the fine-tuned model. The objective during finetuning is to minimize box prediction error on the input data.
Input to the algorithm must be a directory with sub-directoriey images and a file annotations.json. The input directory should look like below if the training data contains two images. The names of .png files can be anything.
input_directory
|--images
|--abc.png
|--def.png
|--annotations.json
The annotations.json file should have information for bounding_boxes and their class labels. It should have a dictionary with keys "images" and "annotations". Value for the "images" key should be a list of entries, one for each image of the form {"file_name": image_name, "height": height, "width": width, "id": image_id}. Value of the 'annotations' key should be a list of entries, one for each bounding box of the form {"image_id": image_id, "bbox": [xmin, ymin, xmax, ymax], "category_id": bbox_label}.
We provide pennfudanped dataset as a default dataset for fine-tuning the model. PennFudanPed comprises images of pedestrians. The dataset has been downloaded from here.
Citation: @ONLINE {pennfudanped, author = "Liming Wang1, Jianbo Shi2, Gang Song2, and I-fan Shen1", title = "Penn-Fudan Database for Pedestrian Detection and Segmentation", year = "2007", url = "https://www.cis.upenn.edu/~jshi/ped_html/" }
3.1. Retrieve Training Artifacts
Here, we retrieve the training docker container, the training algorithm source, and the pre-trained base model. Note that model_version="*" fetches the latest model.
3.2. Set Training parameters
Now that we are done with all the set up that is needed, we are ready to train our Object Detection model. To begin, let us create a sageMaker.estimator.Estimator object. This estimator will launch the training job.
There are two kinds of parameters that need to be set for training. The first one are the parameters for the training job. These include: (i) Training data path. This is S3 folder in which the input data is stored, (ii) Output path: This the s3 folder in which the training output is stored. (iii) Training instance type: This indicates the type of machine on which to run the training. Typically, we use GPU instances for these training. We defined the training instance type above to fetch the correct train_image_uri.
The second set of parameters are algorithm specific training hyper-parameters.
For algorithm specific hyper-parameters, we start by fetching python dictionary of the training hyper-parameters that the algorithm accepts with their default values. This can then be overridden to custom values.
3.3. Train with Automatic Model Tuning (HPO)
Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose. We will use a HyperparameterTuner object to interact with Amazon SageMaker hyperparameter tuning APIs.
3.4. Start Training
We start by creating the estimator object with all the required assets and then launch the training job. It takes less than 10 mins on the default dataset.
3.5. Deploy and run inference on the fine-tuned model
A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the bounding boxes of an image. We follow the same steps as in 2. Run inference on the pre-trained model. We start by retrieving the jumpstart artifacts for deploying an endpoint. However, instead of base_predictor, we deploy the od_estimator that we fine-tuned.
Next, we download an example pedestrian image from the S3 bucket for inference.
Next, we query the finetuned model, parse the response and display the predictions. Functions for these are implemented in sections 2.4. Query endpoint and parse response and 2.5. Display model predictions
Next, we delete the endpoint corresponding to the finetuned model.
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.
