BONUS Amazon JumpStart Upscaling
Introduction to JumpStart - Enhance image quality guided by prompt
This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook.
Welcome to Amazon SageMaker JumpStart! You can use Sagemaker JumpStart to solve many Machine Learning tasks through one-click in SageMaker Studio, or through SageMaker Python SDK.
In this demo notebook, we demonstrate how to use SageMaker Python SDK for Upscaling with state-of-the-art pre-trained Stable Diffusion models. Upscaling is the task of generating high resolution image given a low resolution image and a textual prompt describing the image. An image that is low resolution, blurry, and pixelated can be converted into a high resolution image that appears smoother, clearer, and more detailed. This process, called upscaling, can be applied to both real images and images generated by text-to-image Stable Diffusion models. This can be used to enhance image quality in various industries such as e-commerce and real estate, as well as for artists and photographers. Additionally, upscaling can improve the visual quality of low-resolution images when displayed on high-resolution screens.
Stable Diffusion uses an AI algorithm to upscale images, eliminating the need for manual work that may require manually filling gaps in an image. It has been trained on millions of images and can accurately predict high-resolution images, resulting in a significant increase in detail compared to traditional image upscalers. Additionally, unlike non-deep-learning techniques such as nearest neighbor, Stable Diffusion takes into account the context of the image, using a textual prompt to guide the upscaling process.
Notebook license: This notebook is provided under MIT No Attribution license.
Model lincese: By using this model, you agree to the CreativeML Open RAIL-M++ license.
Note: This notebook was tested on ml.t3.medium instance in Amazon SageMaker Studio with Python 3 (Data Science) kernel and in Amazon SageMaker Notebook instance with conda_python3 kernel.
Note: After you’re done running the notebook, make sure to delete all resources so that all the resources that you created in the process are deleted and your billing is stopped. Code in Clean up the endpoint deletes model and endpoints that are created.
1. Set Up
Before executing the notebook, there are some initial steps required for set up. This notebook requires latest version of sagemaker and ipywidgets.
WARNING: The directory '/root/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag. WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv [notice] A new release of pip is available: 23.0 -> 23.0.1 [notice] To update, run: pip install --upgrade pip
Permissions and environment variables
To host on Amazon SageMaker, we need to set up and authenticate the use of AWS services. Here, we use the execution role associated with the current notebook as the AWS account role with SageMaker access.
2. Retrieve Artifacts & Deploy an Endpoint
Using SageMaker, we can perform inference on the pre-trained model, even without fine-tuning it first on a new dataset. We start by retrieving the deploy_image_uri, deploy_source_uri, and model_uri for the pre-trained model. To host the pre-trained model, we create an instance of sagemaker.model.Model and deploy it. This may take a few minutes.
!
3. Query endpoint and parse response
Input to the endpoint is a prompt, a low resolution image and image generation parameters in json format and encoded in utf-8 format. Output of the endpoint is a json with generated images and the input prompt.
3.1 Download example low resolution image
We start by downloading an example image with low resolution.
Next we write some helper function for querying the endpoint, parsing the response and display generated image.
Below, we put in the example low resolution image and a prompt. You can put in any text and any image and the model generates the corresponding upscaled image. Note that model generates an image of size up to four times the original image. So, putting a very large input image may result in CUDA memory issue. To address this, either input a low resolution image or select an instance type with large CUDA memory such as ml.g5.2xlarge.
Supported Parameters
This model supports many parameters while performing inference. They include:
- prompt: prompt to guide the image generation. Must be specified and can be a string or a list of strings.
- num_inference_steps: number of denoising steps during image generation. More steps lead to higher quality image. If specified, it must a positive integer.
- guidance_scale: higher guidance scale results in image closely related to the prompt, at the expense of image quality. If specified, it must be a float. guidance_scale<=1 is ignored.
- negative_prompt: guide image generation against this prompt. If specified, it must be a string or a list of strings and used with guidance_scale. If guidance_scale is disabled, this is also disabled. Moreover, if prompt is a list of strings then negative_prompt must also be a list of strings.
- seed: fix the randomized state for reproducibility. If specified, it must be an integer.
- noise_level: add noise to latent vectors before upscaling. If specified, it must be an integer.
4. Clean up the endpoint
After you’re done running the notebook, make sure to delete all resources created in the process to ensure that the billing is stopped.
Notebook CI Test Results
This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.
