Vector Enrichment Jobs with Amazon SageMaker Geospatial Capabilities

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

In this notebook we will take a look at the vector enrichment operations available as part of SageMaker's new Geospatial capabilities.

Environment Set-Up

If you have not already done so, please make sure you have opened this notebook in SageMaker Studio in the Oregon (us-west-2) region. The Geospatial capabilities of SageMaker are currently only available in this region.

Please also ensure that you have set:

• the notebook image to Geospatial 1.0
• the kernel to Python 3
• the instance type to ml.m5.4xlarge.

We will start by making sure the "sagemaker" SDK is updated, and then import the libraries we will need for the notebook.

Next let's ensure that our working directory is the same as the 'files' folder in our Studio Notebook environment.

We will now define a few variables that we will use throughout the notebook that relate to AWS resources.

Note, at the time of writing this notebook sagemaker-geospatial is only available in the region "us-west-2".

Make sure you have the proper policy attached to your notebook's execution role, and a trust relationship added to your role for "sagemaker-geospatial", as specified in the Get Started with Amazon SageMaker Geospatial Capabilities documentation.

Map Matching

Map Matching allows you to snap GPS coordinates to road segments. The input should be a CSV file containing the trace ID (route), longitude, latitude and the timestamp attributes. Let's put a CSV in our working directory to send to S3 later.

As you can see, there are four columns. The first is the "id" column, which holds the "trace"/"drive" ID of the route that is being map-matched. Here the only value is 0 because the data only pertains to one single drive. It is possible to supply data that has many routes, in which case the map matching for each route is calculated separately.

The next two columns hold the longitude and latitude of the GPS system at a that particular moment of the route. Finally, we have the timestamp column, which indicates the time at which that particular coordinate was reached on the route, and allows SageMaker to determine the order in which coordinates appear on the route.

Visualizing input data on a road map

Let's take a look at the data on a map. We will use the folium Python library to do this.

First we will need to parse the CSV to extract the longitude and latitude coordinates.

folium expects coordinates as a list of [latitude, longitude].

Finally, we will need to provide folium with the area of interest to zoom to. Let's supply the bounding box of the coordinates we have extracted from the input file.

With all that done, let's render the coordinates on the map.

The points rendered on the map show a coordinate trace through the City of London. Note how some points aren't perfectly sat on the roads that the trace likely follows.

Starting the Vector Enrichment Job

SageMaker expects this data to be available on S3. As such, let's upload that file to our S3 bucket.

Let's now define some methods that will allow us to easily call the Vector Enrichment Job API.

With our helper methods defined, let's initialize the config objects required by the map matching operation.

We can now start the Map Matching operation. This may take a few minutes, but thankfully the function we defined to start the job will print out the status of the job every 30 seconds.

Exporting the Vector Enrichment Job

With the job completed, we would now like to view the results. The results are currently stored within SageMaker, and require an export to S3 for us to easily access them. Let's define a function that will start the export, and similarly print out the export status as it updates.

Let's download the results to our Notebook directory so that we can look at them.

Visualizing map match output on a road map

With the map matching completed, let's view the snapped coordinates on the map in comparison to our input coordinates.

The snapped coordinates live in the waypoints.geojson file. geojson is a commonly used data format for Geospatial data.

In order to plot the waypoint on our folium map, we need to convert them to a list of [latitude, longitude], (here they are in [longitude, latitude]).

Since we want to compare the snapping results to out input, let's plot the snapped line on our existing map.

As before, the red points are our input coordinates, and the new blue ones are the snapped coordinates from the map matching job output. As you can see, the points that were not quite on the adjacent roads have now been snapped perfectly onto the roads.

Okay, so now we have seen the road-snapped coordinates that we found in the waypoints.geojson output file, but what about the links.geojson output file? What does that contain?

The links file contains coordinate information about all the known roads segments along the shortest viable path that allows us to reach all of our waypoints in one drive (if possible).

First let's open the links file, parse it to JSON, and look at the contents.

As you can see it is a pretty long file. What we care about is the stuff in the 'features' member, a list of line features, each describing a road segment. Let's extract the coordinates of each line segment into a list of lists of coordinates. Remember again that folium expects the coordinates as [latitude, longitude] so we need to swap the coordinates around.

We can see that there are many links on the route. Each link is an individual road segment. A road is divided into separate segments at points where it connects to other roads.

For example, a long straight road which has a junction in the middle could be divided into three segments: the part before the junction, the junction, and the part after the junction. This is why long roads that have many connections to side streets contain many individual road segments/links.

Here we have rendered the link data onto the map such that the start of the calculated route is in orange and the end is in pink.

Notice how the path at the start of the route differs from what we might expect when looking at the waypoint data. One might think the westward drive along the north of Hyde Park would be an unbroken straight line. Instead, the route is altered such that we go around the oval at the north-east of Hyde Park in order to get back on Bayswater Road.

This is because the link output takes road directionality into consideration. The connection between eastern Bayswater Road and western Oxford Street is one-directional, allowing only eastward traffic. As such, the shortest viable route that allows the waypoints to be joined requires we take the loop adjacent to north-east Hyde Park.

Let's render the two maps side-by-side so that you can compare.

Reverse Geocoding

Reverse Geocoding is the process of deriving human-readable addresses from coordinates. Let's take a look at how to do this in SageMaker.

The Reverse Geocoding operation in SageMaker requires a CSV of coordinates as input. Let's use the snapped coordinates that we computed in the map match.

Again, we need to upload our CSV to S3 in order for it to be accessible by SageMaker.

Like the map matching job, the reverse geocoding job will require an input file config and a job config to be passed to the start Vector Enrichment Job function.

As with the map matching job, we then pass the config objects to our start Vector Enrichment Job function, and export the results to s3 once it is completed.

Now that the results are exported to S3, let's download them to our notebook's working directory so that we can visualize the results on the map.

The result is a CSV whose first two columns are identical to what we supplied as input, and with extra columns added by the reverse geocoding job that contain human-readable address properties.

Let's parse this CSV so that we can easier work with it.

Now we can show the results on a map. To avoid the map being too saturated with markers, we will only plot every third address.

That's it! Thank you for following this notebook on Vector Enrichment Jobs in SageMaker!

Cleanup

If you want to remove the objects saved in your S3 bucket as part of this notebook, uncomment and run the following code.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.