Document Text Classification
data-scienceinferencearchivedamazon-sagemaker-examplesreinforcement-learningidentify_key_insights_from_textual_documentmachine-learningawsexamplesdeep-learningsagemakerjupyter-notebooktrainingmlops
Export
Document Understanding Solution - Text Classification
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
Text Classification refers to classifying an input sentence to one of the class labels of the training dataset. In this notebook, we demonstrate how to use the JumpStart API for Text Classification. In particular, we demonstrate three use cases of Text Classification:
- How to directly deploy a pretrained Transformer-based text classification model to perform Sentiment Analysis.
- How to fine-tune a pre-trained Transformer model on a custom dataset, and then run inference on the fine-tuned model.
- How to run SageMaker Automatic Model Tuning (a hyperparameter optimization procedure) to find the best model compared with the model fine-tuned in point 2. The performance of the optimal model and model fine-tuned in point 2 is evaluated on a hold-out test data.
Note: When running this notebook on SageMaker Studio, you should make
sure the PyTorch 1.10 Python 3.8 CPU Optimized image/kernel is used. When
running this notebook on SageMaker Notebook Instance, you should make
sure the 'sagemaker-soln' kernel is used.
1. Set Up
Before executing the notebook, there are some initial steps required for setup. This notebook requires latest version of sagemaker and ipywidgets.
[2]
Requirement already satisfied: sagemaker in /opt/conda/lib/python3.9/site-packages (2.157.0) Collecting ipywidgets Using cached ipywidgets-8.0.6-py3-none-any.whl (138 kB) Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (23.0) Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (4.13.0) Requirement already satisfied: pandas in /opt/conda/lib/python3.9/site-packages (from sagemaker) (1.5.3) Requirement already satisfied: cloudpickle==2.2.1 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (2.2.1) Requirement already satisfied: schema in /opt/conda/lib/python3.9/site-packages (from sagemaker) (0.7.5) Requirement already satisfied: tblib==1.7.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (1.7.0) Requirement already satisfied: boto3<2.0,>=1.26.131 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (1.26.135) Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (1.23.5) Requirement already satisfied: protobuf<4.0,>=3.1 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (3.20.2) Requirement already satisfied: platformdirs in /opt/conda/lib/python3.9/site-packages (from sagemaker) (3.5.1) Requirement already satisfied: PyYAML==6.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (6.0) Requirement already satisfied: jsonschema in /opt/conda/lib/python3.9/site-packages (from sagemaker) (4.17.3) Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (1.0.1) Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (0.1.5) Requirement already satisfied: attrs<24,>=23.1.0 in /opt/conda/lib/python3.9/site-packages (from sagemaker) (23.1.0) Requirement already satisfied: google-pasta in /opt/conda/lib/python3.9/site-packages (from sagemaker) (0.2.0) Requirement already satisfied: pathos in /opt/conda/lib/python3.9/site-packages (from sagemaker) (0.3.0) Requirement already satisfied: ipykernel>=4.5.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets) (5.5.6) Requirement already satisfied: ipython>=6.1.0 in /opt/conda/lib/python3.9/site-packages (from ipywidgets) (8.10.0) Collecting jupyterlab-widgets~=3.0.7 Using cached jupyterlab_widgets-3.0.7-py3-none-any.whl (198 kB) Collecting widgetsnbextension~=4.0.7 Using cached widgetsnbextension-4.0.7-py3-none-any.whl (2.1 MB) Requirement already satisfied: traitlets>=4.3.1 in /opt/conda/lib/python3.9/site-packages (from ipywidgets) (5.9.0) Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.9/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (0.6.0) Requirement already satisfied: botocore<1.30.0,>=1.29.135 in /opt/conda/lib/python3.9/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.29.135) Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.9/site-packages (from boto3<2.0,>=1.26.131->sagemaker) (1.0.1) Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.9/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker) (3.13.0) Requirement already satisfied: jupyter-client in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.1.5) Requirement already satisfied: ipython-genutils in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (0.2.0) Requirement already satisfied: tornado>=4.2 in /opt/conda/lib/python3.9/site-packages (from ipykernel>=4.5.1->ipywidgets) (6.2) Requirement already satisfied: pickleshare in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.7.5) Requirement already satisfied: pexpect>4.3 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (4.8.0) Requirement already satisfied: backcall in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.2.0) Requirement already satisfied: matplotlib-inline in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.1.6) Requirement already satisfied: decorator in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (5.1.1) Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.30 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (3.0.36) Requirement already satisfied: stack-data in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.6.2) Requirement already satisfied: jedi>=0.16 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (0.18.2) Requirement already satisfied: pygments>=2.4.0 in /opt/conda/lib/python3.9/site-packages (from ipython>=6.1.0->ipywidgets) (2.14.0) Requirement already satisfied: six in /opt/conda/lib/python3.9/site-packages (from protobuf3-to-dict<1.0,>=0.1.5->sagemaker) (1.16.0) Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/conda/lib/python3.9/site-packages (from jsonschema->sagemaker) (0.19.3) Requirement already satisfied: python-dateutil>=2.8.1 in /opt/conda/lib/python3.9/site-packages (from pandas->sagemaker) (2.8.2) Requirement already satisfied: pytz>=2020.1 in /opt/conda/lib/python3.9/site-packages (from pandas->sagemaker) (2022.7.1) Requirement already satisfied: ppft>=1.7.6.6 in /opt/conda/lib/python3.9/site-packages (from pathos->sagemaker) (1.7.6.6) Requirement already satisfied: dill>=0.3.6 in /opt/conda/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.6) Requirement already satisfied: pox>=0.3.2 in /opt/conda/lib/python3.9/site-packages (from pathos->sagemaker) (0.3.2) Requirement already satisfied: multiprocess>=0.70.14 in /opt/conda/lib/python3.9/site-packages (from pathos->sagemaker) (0.70.14) Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.9/site-packages (from schema->sagemaker) (21.6.0) Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.9/site-packages (from botocore<1.30.0,>=1.29.135->boto3<2.0,>=1.26.131->sagemaker) (1.26.14) Requirement already satisfied: parso<0.9.0,>=0.8.0 in /opt/conda/lib/python3.9/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3) Requirement already satisfied: ptyprocess>=0.5 in /opt/conda/lib/python3.9/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0) Requirement already satisfied: wcwidth in /opt/conda/lib/python3.9/site-packages (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets) (0.2.6) Requirement already satisfied: pyzmq>=13 in /opt/conda/lib/python3.9/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (24.0.1) Requirement already satisfied: jupyter-core>=4.6.0 in /opt/conda/lib/python3.9/site-packages (from jupyter-client->ipykernel>=4.5.1->ipywidgets) (4.9.2) Requirement already satisfied: executing>=1.2.0 in /opt/conda/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (1.2.0) Requirement already satisfied: asttokens>=2.1.0 in /opt/conda/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.2.1) Requirement already satisfied: pure-eval in /opt/conda/lib/python3.9/site-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2) Installing collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets Successfully installed ipywidgets-8.0.6 jupyterlab-widgets-3.0.7 widgetsnbextension-4.0.7 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0 -> 23.1.2 [notice] To update, run: pip install --upgrade pip
[3]
2. Select a pre-trained text classification model
You can continue with the default model, or can choose a different model from the dropdown generated upon running the next cell. A complete list of JumpStart models can also be accessed at JumpStart Models.
[4]
You can also select a different JumpStart model. Here, we download jumpstart model_manifest file from the jumpstart s3 bucket, filter-out all the Text Classification models and select a model.
[5]
All the other available text classification models are as below. huggingface-tc-bert-base-cased huggingface-tc-bert-base-multilingual-cased huggingface-tc-bert-base-multilingual-uncased huggingface-tc-bert-base-uncased huggingface-tc-bert-large-cased huggingface-tc-bert-large-cased-whole-word-masking huggingface-tc-bert-large-uncased huggingface-tc-bert-large-uncased-whole-word-masking huggingface-tc-distilbert-base-cased huggingface-tc-distilbert-base-multilingual-cased huggingface-tc-distilbert-base-uncased huggingface-tc-distilroberta-base huggingface-tc-models huggingface-tc-roberta-base huggingface-tc-roberta-base-openai-detector huggingface-tc-roberta-large huggingface-tc-roberta-large-openai-detector huggingface-tc-xlm-clm-ende-1024 huggingface-tc-xlm-mlm-ende-1024 huggingface-tc-xlm-mlm-enro-1024 huggingface-tc-xlm-mlm-tlm-xnli15-1024 tensorflow-tc-albert-en-base tensorflow-tc-bert-en-cased-L-12-H-768-A-12-2 tensorflow-tc-bert-en-cased-L-24-H-1024-A-16-2 tensorflow-tc-bert-en-uncased-L-12-H-768-A-12-2 tensorflow-tc-bert-en-uncased-L-24-H-1024-A-16-2 tensorflow-tc-bert-en-wwm-cased-L-24-H-1024-A-16-2 tensorflow-tc-bert-en-wwm-uncased-L-24-H-1024-A-16-2 tensorflow-tc-bert-multi-cased-L-12-H-768-A-12-2 tensorflow-tc-electra-base-1 tensorflow-tc-electra-small-1 tensorflow-tc-experts-bert-pubmed-1 tensorflow-tc-experts-bert-wiki-books-1 tensorflow-tc-small-bert-bert-en-uncased-L-10-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-10-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-10-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-10-H-768-A-12 tensorflow-tc-small-bert-bert-en-uncased-L-12-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-12-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-12-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-12-H-768-A-12 tensorflow-tc-small-bert-bert-en-uncased-L-2-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-2-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-2-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-2-H-768-A-12 tensorflow-tc-small-bert-bert-en-uncased-L-4-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-4-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-4-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-4-H-768-A-12 tensorflow-tc-small-bert-bert-en-uncased-L-6-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-6-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-6-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-6-H-768-A-12 tensorflow-tc-small-bert-bert-en-uncased-L-8-H-128-A-2 tensorflow-tc-small-bert-bert-en-uncased-L-8-H-256-A-4 tensorflow-tc-small-bert-bert-en-uncased-L-8-H-512-A-8 tensorflow-tc-small-bert-bert-en-uncased-L-8-H-768-A-12 tensorflow-tc-talking-heads-base tensorflow-tc-talking-heads-large
3. Run inference on the pre-trained text classification model
This is a Text Classification model built upon a Text Embedding model from TensorFlow Hub. It takes a text string as input and classifies the input text as either a positive or negative movie review.
The Text Embedding model which is pre-trained on Wikipedia and BookCorpus datasets returns an embedding of the input text.
The model available for deployment is created by attaching a binary classification layer to the output of the Text Embedding model, and then fine-tuning the entire model on SST2 dataset. The SST2 dataset comprises positive and negative movie reviews.
3.1. Retrieve jumpStart artifacts & deploy an endpoint
We retrieve the deploy_image_uri, deploy_source_uri, and base_model_uri for the pre-trained model. To host the pre-trained model, we create an instance of sagemaker.model.Model and deploy it.
[6]
------!
3.2. Example input sentences for inference
These examples are taken from SST2 dataset downloaded from TensorFlow. Apache 2.0 License. Dataset Homepage.
[7]
3.3. Query endpoint and parse response
Input to the endpoint is a single sentence. Response from the endpoint is a dictionary containing the predicted class label, and a list of class label probabilities.
[8]
Inference: Input text: 'astonishing ... ( frames ) profound ethical and philosophical questions in the form of dazzling pop entertainment' Model prediction: [0.00015312265143107244, 0.9998468773485689] Labels: ['negative', 'positive'] Predicted Label: positive Inference: Input text: 'simply stupid , irrelevant and deeply , truly , bottomlessly cynical ' Model prediction: [0.9997017409141489, 0.00029825908585111375] Labels: ['negative', 'positive'] Predicted Label: negative
3.4. Clean up the endpoint
[9]
4. Finetune the pre-trained model on a custom dataset
Previously, we saw how to run inference on a pre-trained model, which was fine-tuned on SST dataset. Next, we discuss how a model can be finetuned to a custom dataset with any number of classes.
The Text Embedding model can be fine-tuned on any text classification dataset in the same way the model available for inference has been fine-tuned on the SST2 movie review dataset.
The model available for fine-tuning attaches a classification layer to the Text Embedding model and initializes the layer parameters to random values. The output dimension of the classification layer is determined based on the number of classes detected in the input data. The fine-tuning step fine-tunes all the model parameters to minimize prediction error on the input data and returns the fine-tuned model. The model returned by fine-tuning can be further deployed for inference. Below are the instructions for how the training data should be formatted for input to the model.
- Input: A directory containing a 'data.csv' file.
- Each row of the first column of 'data.csv' should have integer class labels between 0 to the number of classes.
- Each row of the second column should have the corresponding text.
- Output: A trained model that can be deployed for inference.
Below is an example of 'data.csv' file showing values in its first two columns. Note that the file should not have any header.
| 0 | hide new secretions from the parental units |
| 0 | contains no wit , only labored gags |
| 1 | that loves its characters and communicates something rather beautiful about human nature |
| ... | ... |
source: TensorFlow Hub. License:Apache 2.0 License.
SST2 dataset is downloaded from TensorFlow. Apache 2.0 License. Dataset Homepage.
4.1. Retrieve jumpStart training artifacts
Here, for the selected model, we retrieve the training docker container, the training algorithm source, the pre-trained model, and a python dictionary of the training hyper-parameters that the algorithm accepts with their default values. Note that the model_version="*" fetches the lates model. Also, we do need to specify the training_instance_type to fetch train_image_uri.
[10]
4.2. Set training parameters
Now that we are done with all the setup that is needed, we are ready to fine-tune our Text Classification model. To begin, let us create a sageMaker.estimator.Estimator object. This estimator launches the training job.
There are two kinds of parameters that need to be set for training.
The first one are the parameters for the training job. These include: (i) Training data path. This is S3 folder in which the input data is stored, (ii) Output path: This the s3 folder in which the training output is stored. (iii) Training instance type: This indicates the type of machine on which to run the training. Typically, we use GPU instances for these training. We defined the training instance type above to fetch the correct train_image_uri.
The second set of parameters are algorithm specific training hyper-parameters.
[11]
For algorithm specific hyper-parameters, we start by fetching python dictionary of the training hyper-parameters that the algorithm accepts with their default values. This can then be overridden to custom values.
[12]
{'epochs': '3', 'adam-learning-rate': '1e-6', 'batch-size': '64', 'reinitialize-top-layer': 'Auto', 'train-only-top-layer': 'False'}
4.3. Download, preprocess, and upload the training data
[13]
download: s3://jumpstart-cache-prod-us-west-2/training-datasets/SST/data.csv to data/sst2/data.csv
[14]
View the first five observations of the training data
[15]
[16]
[17]
[18]
Upload the splitted training data into the S3 bucket. The training data is further splitted into training and validation data during training. The test data is used as hold-out data to evaluate the model performance.
[19]
4.4 Fine-tuning without hyperparameter optimization
We start by creating the estimator object with all the required assets and then launch the training job.
[20]
INFO:sagemaker:Creating training-job with name: sagemaker-soln-documents--tc-finetune-2023-05-18-22-50-48-250
2023-05-18 22:50:48 Starting - Starting the training job... 2023-05-18 22:51:15 Starting - Preparing the instances for training......... 2023-05-18 22:52:25 Downloading - Downloading input data... 2023-05-18 22:53:01 Training - Downloading the training image......... 2023-05-18 22:54:46 Training - Training image download completed. Training in progress....2023-05-18 22:55:11.282934: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler. 2023-05-18 22:55:11.283170: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped. 2023-05-18 22:55:11.290441: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2 2023-05-18 22:55:11.328155: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler. 2023-05-18 22:55:12,950 sagemaker-training-toolkit INFO Imported framework sagemaker_tensorflow_container.training 2023-05-18 22:55:13,610 sagemaker-training-toolkit INFO Invoking user script Training Env: { "additional_framework_parameters": {}, "channel_input_dirs": { "model": "/opt/ml/input/data/model", "training": "/opt/ml/input/data/training" }, "current_host": "algo-1", "framework_module": "sagemaker_tensorflow_container.training:main", "hosts": [ "algo-1" ], "hyperparameters": { "adam-learning-rate": "1e-6", "batch-size": "64", "epochs": "3", "reinitialize-top-layer": "Auto", "train-only-top-layer": "False" }, "input_config_dir": "/opt/ml/input/config", "input_data_config": { "model": { "ContentType": "application/x-sagemaker-model", "TrainingInputMode": "File", "S3DistributionType": "FullyReplicated", "RecordWrapperType": "None" }, "training": { "TrainingInputMode": "File", "S3DistributionType": "FullyReplicated", "RecordWrapperType": "None" } }, "input_dir": "/opt/ml/input", "is_master": true, "job_name": "sagemaker-soln-documents--tc-finetune-2023-05-18-22-50-48-250", "log_level": 20, "master_hostname": "algo-1", "model_dir": "/opt/ml/model", "module_dir": "s3://jumpstart-cache-prod-us-west-2/source-directory-tarballs/tensorflow/transfer_learning/tc/v1.2.2/sourcedir.tar.gz", "module_name": "transfer_learning", "network_interface_name": "eth0", "num_cpus": 8, "num_gpus": 1, "output_data_dir": "/opt/ml/output/data", "output_dir": "/opt/ml/output", "output_intermediate_dir": "/opt/ml/output/intermediate", "resource_config": { "current_host": "algo-1", "current_instance_type": "ml.p3.2xlarge", "current_group_name": "homogeneousCluster", "hosts": [ "algo-1" ], "instance_groups": [ { "instance_group_name": "homogeneousCluster", "instance_type": "ml.p3.2xlarge", "hosts": [ "algo-1" ] } ], "network_interface_name": "eth0" }, "user_entry_point": "transfer_learning.py" } Environment variables: SM_HOSTS=["algo-1"] SM_NETWORK_INTERFACE_NAME=eth0 SM_HPS={"adam-learning-rate":"1e-6","batch-size":"64","epochs":"3","reinitialize-top-layer":"Auto","train-only-top-layer":"False"} SM_USER_ENTRY_POINT=transfer_learning.py SM_FRAMEWORK_PARAMS={} SM_RESOURCE_CONFIG={"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.p3.2xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p3.2xlarge"}],"network_interface_name":"eth0"} SM_INPUT_DATA_CONFIG={"model":{"ContentType":"application/x-sagemaker-model","RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"},"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}} SM_OUTPUT_DATA_DIR=/opt/ml/output/data SM_CHANNELS=["model","training"] SM_CURRENT_HOST=algo-1 SM_MODULE_NAME=transfer_learning SM_LOG_LEVEL=20 SM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main SM_INPUT_DIR=/opt/ml/input SM_INPUT_CONFIG_DIR=/opt/ml/input/config SM_OUTPUT_DIR=/opt/ml/output SM_NUM_CPUS=8 SM_NUM_GPUS=1 SM_MODEL_DIR=/opt/ml/model SM_MODULE_DIR=s3://jumpstart-cache-prod-us-west-2/source-directory-tarballs/tensorflow/transfer_learning/tc/v1.2.2/sourcedir.tar.gz SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"model":"/opt/ml/input/data/model","training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"adam-learning-rate":"1e-6","batch-size":"64","epochs":"3","reinitialize-top-layer":"Auto","train-only-top-layer":"False"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"model":{"ContentType":"application/x-sagemaker-model","RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"},"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"sagemaker-soln-documents--tc-finetune-2023-05-18-22-50-48-250","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://jumpstart-cache-prod-us-west-2/source-directory-tarballs/tensorflow/transfer_learning/tc/v1.2.2/sourcedir.tar.gz","module_name":"transfer_learning","network_interface_name":"eth0","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_group_name":"homogeneousCluster","current_host":"algo-1","current_instance_type":"ml.p3.2xlarge","hosts":["algo-1"],"instance_groups":[{"hosts":["algo-1"],"instance_group_name":"homogeneousCluster","instance_type":"ml.p3.2xlarge"}],"network_interface_name":"eth0"},"user_entry_point":"transfer_learning.py"} SM_USER_ARGS=["--adam-learning-rate","1e-6","--batch-size","64","--epochs","3","--reinitialize-top-layer","Auto","--train-only-top-layer","False"] SM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate SM_CHANNEL_MODEL=/opt/ml/input/data/model SM_CHANNEL_TRAINING=/opt/ml/input/data/training SM_HP_ADAM-LEARNING-RATE=1e-6 SM_HP_BATCH-SIZE=64 SM_HP_EPOCHS=3 SM_HP_REINITIALIZE-TOP-LAYER=Auto SM_HP_TRAIN-ONLY-TOP-LAYER=False PYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/local/lib/python37.zip:/usr/local/lib/python3.7:/usr/local/lib/python3.7/lib-dynload:/usr/local/lib/python3.7/site-packages Invoking script with the following command: /usr/local/bin/python3.7 transfer_learning.py --adam-learning-rate 1e-6 --batch-size 64 --epochs 3 --reinitialize-top-layer Auto --train-only-top-layer False [2023-05-18 22:55:27.579 ip-10-0-220-59.us-west-2.compute.internal:36 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None [2023-05-18 22:55:27.760 ip-10-0-220-59.us-west-2.compute.internal:36 INFO profiler_config_parser.py:102] User has disabled profiler. Model: "functional_1" __________________________________________________________________________________________________ Layer (type) Output Shape Param # Connected to ================================================================================================== input_word_ids (InputLayer) [(None, None)] 0 __________________________________________________________________________________________________ input_mask (InputLayer) [(None, None)] 0 __________________________________________________________________________________________________ input_type_ids (InputLayer) [(None, None)] 0 __________________________________________________________________________________________________ keras_layer (KerasLayer) [(None, 768), (None, 109482241 input_word_ids[0][0] input_mask[0][0] input_type_ids[0][0] __________________________________________________________________________________________________ dropout (Dropout) (None, 768) 0 keras_layer[0][0] __________________________________________________________________________________________________ dense (Dense) (None, 2) 1538 dropout[0][0] ================================================================================================== Total params: 109,483,779 Trainable params: 109,483,778 Non-trainable params: 1 __________________________________________________________________________________________________ [2023-05-18 22:55:46.783 ip-10-0-220-59.us-west-2.compute.internal:36 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json. [2023-05-18 22:55:46.784 ip-10-0-220-59.us-west-2.compute.internal:36 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries. [2023-05-18 22:55:46.785 ip-10-0-220-59.us-west-2.compute.internal:36 INFO hook.py:255] Saving to /opt/ml/output/tensors [2023-05-18 22:55:46.785 ip-10-0-220-59.us-west-2.compute.internal:36 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist. [2023-05-18 22:55:46.786 ip-10-0-220-59.us-west-2.compute.internal:36 INFO hook.py:425] Monitoring the collections: sm_metrics, metrics, losses [2023-05-18 22:55:46.789 ip-10-0-220-59.us-west-2.compute.internal:36 INFO hook.py:425] Monitoring the collections: sm_metrics, metrics, losses Epoch 1/3 845/845 - 264s - loss: 0.5694 - accuracy: 0.7042 - val_loss: 0.3484 - val_accuracy: 0.8717 - batch: 0.0000e+00 Epoch 2/3 845/845 - 263s - loss: 0.3318 - accuracy: 0.8703 - val_loss: 0.2848 - val_accuracy: 0.8895 - batch: 1.0000 Epoch 3/3 845/845 - 263s - loss: 0.2978 - accuracy: 0.8825 - val_loss: 0.2733 - val_accuracy: 0.8945 - batch: 2.0000 2023-05-18 22:55:14.012611: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler. 2023-05-18 22:55:14.012799: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped. 2023-05-18 22:55:14.055607: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler. WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Model.state_updates (from tensorflow.python.keras.engine.training) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. 2023-05-18 23:09:30.355561: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them. WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. WARNING:tensorflow:From /usr/local/lib/python3.7/site-packages/tensorflow/python/training/tracking/tracking.py:111: Layer.updates (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version. Instructions for updating: This property should not be used in TensorFlow 2.0, as updates are applied automatically. 2023-05-18 23:09:38,625 sagemaker-training-toolkit INFO Reporting training SUCCESS 2023-05-18 23:09:45 Uploading - Uploading generated training model 2023-05-18 23:10:51 Completed - Training job completed Training seconds: 1106 Billable seconds: 1106
4.5. Deploy & run Inference on the fine-tuned model
A trained model does nothing on its own. We now want to use the model to perform inference. For this example, that means predicting the class label of an input sentence. We follow the same steps as in 3. Run inference on the pre-trained model. We start by retrieving the jumpstart artifacts for deploying an endpoint. However, instead of base_predictor, we deploy the tc_estimator that we fine-tuned.
[21]
INFO:sagemaker.image_uris:Ignoring unnecessary Python version: py37. INFO:sagemaker:Repacking model artifact (s3://sagemaker-us-west-2-688520471316/TC/output/sagemaker-soln-documents--tc-finetune-2023-05-18-22-50-48-250/output/model.tar.gz), script artifact (s3://jumpstart-cache-prod-us-west-2/source-directory-tarballs/tensorflow/inference/tc/v1.1.1/sourcedir.tar.gz), and dependencies ([]) into single tar.gz file located at s3://sagemaker-us-west-2-688520471316/sagemaker-jumpstart-2023-05-18-23-11-28-349/model.tar.gz. This may take some time depending on model size... INFO:sagemaker:Creating model with name: sagemaker-jumpstart-2023-05-18-23-11-28-349 INFO:sagemaker:Creating endpoint-config with name sagemaker-soln-documents--1eed6b-tc-finetune-endpoint INFO:sagemaker:Creating endpoint with name sagemaker-soln-documents--1eed6b-tc-finetune-endpoint
-----!
Next, we query each of the examples in the test data to get its predicted label.
[22]
[23]
4.6. Compute evaluation metrics
Since it is a binary classification task, we use accuracy score and f1 score as the evaluation metrics.
[24]
[25]
[26]
For accuracy and F1 score, larger value indicates the better performance.
5. Finetune the pre-trained model on a custom dataset with automatic model tuning (AMT)
Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose. We use a HyperparameterTuner object to interact with Amazon SageMaker hyperparameter tuning APIs.
5.1. Fine-tuning with hyperparameter optimization
[27]
[29]
WARNING:sagemaker.estimator:No finished training job found associated with this estimator. Please make sure this estimator is only used for building workflow config INFO:sagemaker:Creating hyperparameter tuning job with name: sagemaker-soln-docum-230518-2328
.......................................................................................................................................................................................................................................................!
5.2. Deploy & run Inference on the fine-tuned model
[30]
INFO:sagemaker.image_uris:Ignoring unnecessary Python version: py37.
2023-05-18 23:48:57 Starting - Preparing the instances for training 2023-05-18 23:48:57 Downloading - Downloading input data 2023-05-18 23:48:57 Training - Training image download completed. Training in progress. 2023-05-18 23:48:57 Uploading - Uploading generated training model 2023-05-18 23:48:57 Completed - Resource retained for reuse
INFO:sagemaker:Repacking model artifact (s3://sagemaker-us-west-2-688520471316/TC/output/sagemaker-soln-docum-230518-2328-003-ec9c36b1/output/model.tar.gz), script artifact (s3://jumpstart-cache-prod-us-west-2/source-directory-tarballs/tensorflow/inference/tc/v1.1.1/sourcedir.tar.gz), and dependencies ([]) into single tar.gz file located at s3://sagemaker-us-west-2-688520471316/sagemaker-jumpstart-2023-05-18-23-49-25-130/model.tar.gz. This may take some time depending on model size... INFO:sagemaker:Creating model with name: sagemaker-jumpstart-2023-05-18-23-49-25-130 INFO:sagemaker:Creating endpoint-config with name sagemaker-soln-documents--tc-hpo-endpoint INFO:sagemaker:Creating endpoint with name sagemaker-soln-documents--tc-hpo-endpoint
------!
[31]
[32]
[33]
[34]
We can see results with hyperparameter optimization shows better performance on the hold-out test data.
5.3. Clean Up the endpoint
When you've finished with the summarization endpoint (and associated endpoint-config), make sure that you delete it to avoid accidental charges.
[35]
INFO:sagemaker:Deleting model with name: sagemaker-jumpstart-2023-05-18-23-11-28-349 INFO:sagemaker:Deleting endpoint configuration with name: sagemaker-soln-documents--1eed6b-tc-finetune-endpoint INFO:sagemaker:Deleting endpoint with name: sagemaker-soln-documents--1eed6b-tc-finetune-endpoint INFO:sagemaker:Deleting model with name: sagemaker-jumpstart-2023-05-18-23-49-25-130 INFO:sagemaker:Deleting endpoint configuration with name: sagemaker-soln-documents--tc-hpo-endpoint INFO:sagemaker:Deleting endpoint with name: sagemaker-soln-documents--tc-hpo-endpoint
Next Stage
We've just looked at how you can query document for specific information. Up next we look at a technique that can be used to query the document for specifics, called Question Answering.
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.