Huggingface Sagemaker-sdk - Run a batch transform inference job with 🤗 Transformers

1. Introduction
2. Run Batch Transform after training a model
3. Run Batch Transform Inference Job with a fine-tuned model using jsonl

Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy two transformer model for inference. In the first example we deploy a trained Hugging Face Transformer model on to SageMaker for inference. In the second example we directly deploy one of the 10 000+ Hugging Face Transformers from the Hub to Amazon SageMaker for Inference.<

Run Batch Transform after training a model

not included in the notebook

After you train a model, you can use Amazon SageMaker Batch Transform to perform inferences with the model. In Batch Transform you provide your inference data as a S3 uri and SageMaker will care of downloading it, running the prediction and uploading the results afterwards to S3 again. You can find more documentation for Batch Transform here

If you trained the model using the HuggingFace estimator, you can invoke transformer() method to create a transform job for a model based on the training job.

batch_job = huggingface_estimator.transformer(
    instance_count=1,
    instance_type='ml.c5.2xlarge',
    strategy='SingleRecord')


batch_job.transform(
    data='s3://s3-uri-to-batch-data',
    content_type='application/json',    
    split_type='Line')

For more details about what can be specified here, see API docs.

Run Batch Transform Inference Job with a fine-tuned model using jsonl

Data Pre-Processing

In this example we are using the provided tweet_data.csv as dataset. The csv contains ~1800 tweets about different airlines. The csv contains 1 column "inputs" with the tweets. To use this csv we need to convert it into a jsonl file and upload it to s3. Due to the complex structure of text are only jsonl file supported for batch transform. As pre-processing we are removing the @ in the beginning of the tweet to get the names/identities correct.

_NOTE: While preprocessing you need to make sure that your inputs fit the max_length.

The created file looks like this

{"inputs": "VirginAmerica What dhepburn said."}
{"inputs": "VirginAmerica plus you've added commercials to the experience... tacky."}
{"inputs": "VirginAmerica I didn't today... Must mean I need to take another trip!"}
{"inputs": "VirginAmerica it's really aggressive to blast obnoxious \"entertainment\"...."}
{"inputs": "VirginAmerica and it's a really big bad thing about it"}
{"inputs": "VirginAmerica seriously would pay $30 a flight for seats that didn't h...."}
{"inputs": "VirginAmerica yes, nearly every time I fly VX this \u201cear worm\u201d won\u2019t go away :)"}
{"inputs": "VirginAmerica Really missed a prime opportunity for Men Without ..."}
{"inputs": "virginamerica Well, I didn't\u2026but NOW I DO! :-D"}
{"inputs": "VirginAmerica it was amazing, and arrived an hour early. You're too good to me."}
{"inputs": "VirginAmerica did you know that suicide is the second leading cause of death among teens 10-24"}
{"inputs": "VirginAmerica I <3 pretty graphics. so much better than minimal iconography. :D"}
{"inputs": "VirginAmerica This is such a great deal! Already thinking about my 2nd trip ..."}
....

Create Inference Transformer to run the batch job

We use the twitter-roberta-base-sentiment model running our batch transform job. This is a RoBERTa-base model trained on ~58M tweets and finetuned for sentiment analysis with the TweetEval benchmark.