Document Understanding Solution - Document Summarization

This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.

This solution consists of comphrensive notebooks for five natural language processing tasks Document Summarization, Text Classification, Question and Answering, Name Entity Recognition, and Semantic Relation Extracion. To view each notebook, click Text Classification, Question and Answering, Name Entity Recognition, and Semantic Relation Extraction.

Now let's first look at the Document Summarization. Summarization is useful when you want to distill the information found in a large amount of text down to a few sentences. We use an 'extractive' summarization method in this notebook, that extracts the most important sentences from the document verbatim. We don't cover 'abstractive' summarization here, because it's a lot more challenging and error prone to generate new sentences that summarize the document.

In this notebook, we deploy and use a document summarization model T5-base from the transformers library. Next, we send an example article to the deployed endpoints to get a response (summarization result) and run evaluation metric ROUGE to compare the input article and summmarization result.

Note: When running this notebook on SageMaker Studio, you should make sure the SageMaker JumpStart PyTorch 1.0 image/kernel is used. When running this notebook on SageMaker Notebook Instance, you should make sure the 'sagemaker-soln' kernel is used.

Install required packages to run this notebook

We start by importing a variety of packages that are used throughout the notebook. One of the most important packages is the Amazon SageMaker Python SDK (i.e. import sagemaker). We also import modules from our own custom (and editable) package that can be found at ../package.

Up next, we define the current folder and create a SageMaker client (from boto3). We can use the SageMaker client to call SageMaker APIs directly, as an alternative to using the Amazon SageMaker SDK. We use it at the end of the notebook to delete certain resources that are created in this notebook.

Our text summarization system needs a machine learning model. In this section, we deploy a model to an Amazon SageMaker Endpoint and then invoke the endpoint from the notebook. We use a pre-trained model from the transformers library instead of training a model from scratch, specifically the T5 Base model.

We use the unique solution prefix to name the model and endpoint.

Up next, we need to define the Amazon SageMaker Model which references the source code and the specifies which container to use. Our pre-trained model is from the transformers library which uses PyTorch. As a result, we should use the PyTorchModel from the Amazon SageMaker Python SDK. Using PyTorchModel and setting the framework_version argument, means that our deployed model runs inside a container that has PyTorch pre-installed. Other requirements can be installed by defining a requirements.txt file at the specified source_dir location. We use the entry_point argument to reference the code (within source_dir) that should be run for model inference: functions called model_fn, input_fn, predict_fn and output_fn are expected to be defined. And lastly, you can pass model_data from a training job, but we are going to load the pre-trained model in the source code running on the endpoint. We still need to provide model_data, so we pass an empty archive.

Using this Amazon SageMaker Model, we can deploy a HTTPS endpoint on a dedicated instance. We choose to deploy the endpoint on a single ml.p3.2xlarge instance (or ml.g4dn.2xlarge if unavailable in this region). Our summarization model is transfomer that benefits from GPU optimization, and a ml.p3.2xlarge has a high performance NVIDIA V100 GPU that can reduce inference latency on each request. You can expect this deployment step to take around 5 minutes. After approximately 15 dashes, you can expect to see an exclamation mark which indicates a successful deployment.

When you're trying to update the model for development purposes, but experiencing issues because the model/endpoint-config/endpoint already exists, you can delete the existing model/endpoint-config/endpoint by uncommenting and running the following commands:

When calling our new endpoint from the notebook, we use a Amazon SageMaker SDK Predictor. A Predictor is used to send data to an endpoint (as part of a request), and interpret the response. Our model.deploy command returned a Predictor but, by default, it send and receive numpy arrays. Our endpoint expects to receive (and also sends) JSON formatted objects, so we modify the Predictor to use JSON instead of the PyTorch endpoint default of numpy arrays. JSON is used here because it is a standard endpoint format and the endpoint response can contain nested data structures.

With our model successfully deployed and our predictor configured, we can try out the summarizer out on example inputs. All we need to do is construct a dictionary object with a single key called text and provide the the input string. We call predict on our predictor and we should get a response from the endpoint that contains the summary.

We have the responce and we can print out the summary that has been extracted from the text above.

Evaluate and compare input article and summarization reseult

rougeN measures the number of matching n-grams between our model-generated text (summarization result) and a reference (input text).

An n-gram is simply a grouping of tokens/words. A unigram (1-gram) would consist of a single word. A bigram (2-gram) consists of two consecutive words.

The metric rougeL and rougeLsum measure the longest matching sequences of words by looking for the longest common substrings in the generated and reference summaries. The “sum” in rougeLsum refers to the fact that this metric is computed over a whole summary, while rougeL is computed as the average over individual sentences.

For each metric described above, it computes confidence intervals for precision, recall, and F1-score; these are the low, mid, and high attributes you can see here. For each of precision, recall, and F1-score metrics, higher value indicates better results.

You can try more examples above, but note that this model has been pretrained on a news dataset. You may need to fine-tune this model with your own summarizations to obtain better results.

Clean Up

When you've finished with the summarization endpoint (and associated endpoint-config), make sure that you delete it to avoid accidental charges.

Next Stage

We've just looked at how to analyse documents from a high level. Up next we look at a technique that can be used to classify each sentence in the document, called Text Classification.

Click here to continue with Text Classification.

Notebook CI Test Results

This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.