Building Pipelines with Amazon SageMaker Geospatial Capabilities

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook, we will describe an example of how to build pipelines for automating the processing of geospatial data, using Amazon SageMaker geospatial capabilities and Amazon SageMaker Pipelines.

We will start by making sure the "sagemaker" SDK is updated, and importing a few libraries required.

We will now define a few variables for which we need to create sessions in the SageMaker and Boto3 SDKs.

We will also create the client for SageMaker geospatial capabilities with a Boto session...

Before we continue make sure that your AWS IAM role has the proper permissions for interacting with AWS Lambda, Amazon S3, and Amazon SQS as required.

Also, make sure you have the proper policy and trust relationship added to your role for "sagemaker-geospatial", as specified in the Get Started with Amazon SageMaker Geospatial Capabiltiies documentation.

For demo purposes, you can uncomment and run the following cell to add the policies and trust relationships to your role using the script below, but this policy should be scoped down to improve security for any production deployment, following the least privilege principle.

Let's now create an AWS Lambda function that will work for calling the SageMaker geospatial processes as required. We will use the same function for:

• Starting the EOJs (Cloud Removal and Stacking in our example) - Using a Lambda Step in SageMaker Pipelines
• Checking the status of the EOJs, as these are asynchronous and take a few minutes to complete - Using a Callback Step in SageMaker Pipelines

We will start by writting a script with our code.

We can now use this script for creating our Lambda function.

Now that we have a Lambda function for calling our EOJs, we can implement the Callback steps required in our pipeline.

Note again, we are using SageMaker Pipelines Callback Steps because our EOJs are asynchronous and takes sometime to complete. So we want the Lambda to check the status of the EOJs, and resume the workflows when each EOJ is completed.

For this, we will create an Amazon SQS queue that will be used in our callback.

Now we need to associate the SQS queue as an input trigger for our Lambda function, in this way whenever the Callback Step pushes a message to the queue it would run our Lambda function for checking the status of the EOJ. We do this by creating an Even Source Mapping.

We will now define the parameters to be used in our pipeline, as we want to be able to pass these dynamically whenever we run our geospatial pipeline.

With all the elements in place, we can now start creating our steps with SageMaker Pipelines...

Finally, we can define our pipeline based on the steps and parameters created before.

Let us test our pipeline by defining some values for our parameters and running an execution.

At this point, you can go to the SageMaker Resources tab in the left menu in Studio and check the Pipelines.

You should be able to see our "GeospatialPipeline" in the list and double-cling on it for checking the details of the execution.

Clean-up

Once done, uncomment and run the following cells for deleting any resources that could incur in costs.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.