Huggingface Sagemaker-sdk - Deploy 🤗 Transformers for inference

Welcome to this getting started guide, we will use the new Hugging Face Inference DLCs and Amazon SageMaker Python SDK to deploy a transformer model for inference. In this example we deploy a trained Hugging Face Transformer model on to SageMaker for inference.

API - SageMaker Hugging Face Inference Toolkit

Using the transformers pipelines, we designed an API, which makes it easy for you to benefit from all pipelines features. The API is oriented at the API of the 🤗 Accelerated Inference API, meaning your inputs need to be defined in the inputs key and if you want additional supported pipelines parameters you can add them in the parameters key. Below you can find examples for requests.

text-classification request body

{
	"inputs": "Camera - You are awarded a SiPix Digital Camera! call 09061221066 fromm landline. Delivery within 28 days."
}

question-answering request body

{
	"inputs": {
		"question": "What is used for inference?",
		"context": "My Name is Philipp and I live in Nuremberg. This model is used with sagemaker for inference."
	}
}

zero-shot classification request body

{
	"inputs": "Hi, I recently bought a device from your company but it is not working as advertised and I would like to get reimbursed!",
	"parameters": {
		"candidate_labels": [
			"refund",
			"legal",
			"faq"
		]
	}
}

Deploy a Hugging Face Transformer model from S3 to SageMaker for inference

There are two ways on how you can deploy you SageMaker trained Hugging Face model from S3. You can either deploy it after your training is finished or you can deploy it later using the model_data pointing to you saved model on s3.

Deploy the model directly after training

If you deploy you model directly after training you need to make sure that all required files are saved in your training script, including the Tokenizer and the Model.

from sagemaker.huggingface import HuggingFace

############ pseudo code start ############

# create HuggingFace estimator for running training
huggingface_estimator = HuggingFace(....)

# starting the train job with our uploaded datasets as input
huggingface_estimator.fit(...)

############ pseudo code end ############

# deploy model to SageMaker Inference
predictor = huggingface_estimator.deploy(initial_instance_count=1, instance_type="ml.m5.xlarge")

# example request, you always need to define "inputs"
data = {
   "inputs": "Camera - You are awarded a SiPix Digital Camera! call 09061221066 fromm landline. Delivery within 28 days."
}

# request
predictor.predict(data)

Deploy the model using model_data