Stable Diffusion is a cutting-edge technique that makes use of latent diffusion models to produce high-quality images from textual descriptions. Easy-to-use pipelines for deploying and utilizing the Stable Diffusion model, including creating, editing, and upscaling photos, are provided by the Hugging Face diffusers package.

Best way to upscale Stable Diffusion

We will tell how to use the Stable Diffusion Upscale Pipeline from the diffusers library to upscale images produced by stable diffusion in this article. Go over the rationale behind upscaling and show you how to use the Intel Extension for the PyTorch package (a Python package where Intel releases its latest optimizations and features before upstreaming them into open-source PyTorch) to optimize this process for better performance on Intel Xeon Processors.

How can the Stable Diffusion Upscale Pipeline be made more efficient for inference?

Using the Stable Diffusion model, the Stable Diffusion Upscale Pipeline from the Hugging Face diffusers library is intended to improve input image resolution, specifically increasing the resolution by a factor of four. This pipeline employs a number of different components, such as a frozen CLIP text model for text encoding, a Variational Auto-Encoder (VAE) for picture encoding and decoding, a UNet architecture for image latent noise reduction, and multiple schedulers to control the diffusion process during image production.

This pipeline is perfect for bringing out the details in artificial or real-world photos, as it is especially helpful for applications that need to produce high-quality image outputs from lower resolution inputs. In order to balance fidelity to the input text versus image quality, users can define a variety of parameters, including the number of denoising steps. Custom callbacks can also be supported during the inference process to allow for monitoring or modification of the generation.

Tune each of the Stable Diffusion Upscale Pipeline’s component parts separately before combining them to improve the pipeline’s performance. An essential component of this improvement is the PyTorch Intel Extension. With sophisticated optimizations, the addon improves PyTorch and offers a further boost in speed on Intel technology. These improvements make use of the capabilities of Intel CPUs’ Vector Neural Network Instructions (VNNI), Intel Advanced Vector Extensions 512 (Intel AVX-512), and Intel Advanced Matrix Extensions (Intel AMX). The Python API ipex.optimize(), which is accessible through the Intel Extension for PyTorch, automatically optimizes the pipeline module, enabling it to take advantage of these advanced hardware instructions for increased performance efficiency.

Sample Code

Using the Intel Extension for PyTorch for performance enhancements, the code sample below shows how to upscale a picture using the Stable Diffusion Upscale Pipeline from the diffusers library. The pipeline’s U-Net, VAE, and text encoder components are all individually targeted and CPU inference-optimized.

Configuring the surroundings

It is advised to carry out the installations in a virtualized Conda environment. Install the Intel Extension for PyTorch, diffusers, and PyTorch itself:

python -m pip install torch torchvision torchaudio –index-url https://download.pytorch.org/whl/cpu python -m pip install intel-extension-for-pytorch python -m pip install oneccl_bind_pt –extra-index-url https://pytorch-extension.intel.com/release-whl/stable/cpu/us/ pip install transformers pip install diffusers

How to Improve

First, let’s load the sample image that you want to upscale and import all required packages, including the Intel Extension for PyTorch.

Let’s now examine how the PyTorch Intel Extension capabilities can be used to optimize the upscaling pipeline.

To optimize, each pipeline component is targeted independently. Initially, you configure the text encoder, VAE, and UNet to use the Channels Last format. Tensor dimensions are ordered as batch, height, width, and channels using the channel’s last format. Because it better fits specific memory access patterns, this structure is more effective and improves performance. For convolutional neural networks, channels last is especially helpful because it minimizes the requirement for data reordering during operations, which can greatly increase processing speed.

Similar to this, ipex.optimize()} is used by Intel Extension for PyTorch to optimize each component, with the data type set to BFloat16. With the help of Intel AMX, which is compatible with 4th generation Xeon Scalable Processors and up, BFloat16 precision operations are optimized. You can enable Intel AMX, an integrated artificial intelligence accelerator, for lower precision data types like BFloat16 and INT8 by utilizing IPEX'soptimize()} function.

Ultimately, you can achieve the best results for upscaling by employing mixed precision, which combines the numerical stability of higher precision (e.g., FP32) with the computational speed and memory savings of lower precision arithmetic (e.g., BF16). This pipeline automatically applies mixed precision when `torch.cpu.amp.autocast()} is set. Now that the pipeline object has been optimized using Intel Extension for PyTorch, it can be utilized to achieve minimal latency and upscale images.

Configuring an Advanced Environment

This section explains how to configure environment variables and configurations that are optimized for Intel Xeon processor performance, particularly for memory management and parallel processing, to gain even more performance boosts. Some environment variables unique to the Intel OpenMP library are configured by the script ‘env_activate.sh’. Additionally, it specifies which shared libraries are loaded before others by using LD_PRELOAD. In order to ensure that particular libraries are loaded at runtime prior to the application starting, the script constructs the path to those libraries dynamically.

How to configure Advanced Environment on Intel Xeon CPUs for optimal performance:

Install two packages that serve as dependencies to use the script

pip install intel-openmp conda install -y gperftools -c conda-forge

git clone https://github.com/intel/intel-extension-for-pytorch.git cd intel-extension-for-pytorch git checkout v2.3.100+cpu

cd examples/cpu/inference/python/llm

Activate environment variables

source ./tools/env_activate.sh

Run a script with the code from the previous section

python run_upscaler_pipeline.py

Your environment is prepared to run Stable Diffusion Upscale Pipeline, which was performance-flagged in the previous stage to optimize for higher performance. Moreover, additional performance can be obtained by inference utilizing the Intel Extension for Pytorch optimized pipeline.

Weight Only Quantization(WOQ)

A practical guide to Large Language Models (LLMs) quantization. The capabilities, uses, and complexity of large language models (LLMs) have all significantly risen in recent years. With an ever-increasing amount of parameters, weights, and activations, LLMs have become larger and more intelligent.

However, They usually have to compress LLMs without significantly sacrificing their performance in order to increase the number of possible deployment targets and lower the cost of inference. Large neural networks, including language models, may be made smaller using a variety of methods. Quantization is one such crucial method.

WOQ meaning

In machine learning, especially in deep learning, Weight Only Quantization (WOQ) is a technique that minimizes the size of neural network models without compromising their functionality. It entails quantizing just the neural network’s weights the parameters that define the behavior of the model into a format with less precision (e.g., 8-bit instead of 32-bit).

This article provides an example of code that uses the Intel Extension for Transformers tool to conduct Weight Only Quantization (WOQ) on an LLM (Intel/neural-chat-7b model) for both INT8 and INT4.

How does quantization work?

INT8 Vs INT4

The process of switching to lower precision data types, such as float16, INT8 or INT4, from high-precision representation, such as float32, for weights and/or activations is known as quantization. Lower precision may greatly minimize the amount of memory needed.

While this may seem simple in principle, there are a lot of subtleties to consider, and computing data type is the most crucial warning to know. Certain operations need us to scale the representation back to high precision at runtime since not all operations support or have low-precision implementation. Although there is some additional cost, they may lessen its effects by using tools like Intel Neural Compressor, OpenVINO toolkit, and Neural Speed.

Because these runtimes include optimized implementations of several operators for low-precision data types, upscale values to high-precision is not necessary, resulting in improved speed and reduced memory use. If lower-precision data types are supported by your hardware, the performance improvements are substantial. For instance, support for float16 and bfloat16 is included into Intel Xeon Scalable processors of the 4th generation.

Therefore, quantization only serves to lower the model’s memory footprint; nevertheless, it may also introduce some cost during inference. Using optimized runtimes and the newest hardware is required to obtain both memory and performance improvements.

What Does WOQ Mean?

There are several methods for quantizing models. Model weights and activations the output values produced by every neuron in a layer are often quantized. One of these quantization methods, called Weight Only Quantization(WOQ), preserves the original accuracy of the activations while only quantizing the model weights. Faster inference and a reduced memory footprint are the clear advantages. In actual use, WOQ improves performance without appreciably affecting accuracy.

Code Execution

The Intel/neural-chat-7b-v3-3 language model’s quantization procedure is shown in the provided code sample. The model, which is an improved version of Mistral-7B, is quantized using Weight Only Quantization (WOQ) methods made available by the Intel Extension for Transformers.

With only one line of code, developers can easily use the power of Intel technology for their Generative AI workloads. You import AutoModelForCausualLM from Intel Extension for Transformers rather of the Hugging Face transformers library, and everything else stays the same.

1. From intel_extension_for_transformers.transformers import AutoModelForCausalLM

For INT8 quantization, just set load_in_8bit to True.

1. # INT8 quantization 2. Q8_model = AutoModelForCausalLM.from_pretrained( 3. model_name, load_in_8bit=True)

Similarly, for INT4 quantization set load_in_4bit to True.1. # INT4 quantization 2. q4_model = AutoModelForCausalLM.from_pretrained( 3. model_name, load_in_4bit=True)

The Hugging Face transformers library may be used in the same way for implementation.

If you set device to GPU, the aforementioned code snippets will utilize BitandBytes for quantization. This makes your code run much faster without requiring any code changes, regardless of whether you are utilizing a CPU or GPU.

GGUF model in operation

A binary file format called GGUF was created expressly to store deep learning models like LLMs especially for CPU inference. It has several important benefits, such as quantization, efficiency, and single-file deployment. They will be utilizing the model in GGUF format in order to maximize the performance of their Intel hardware.

Generally, one would need to utilize an extra library like Llama_cpp in order to execute models in GGUF format. Still, you may use it Intel Extension for Transformers library to run GGUF models since Neural Speed is built on top of Llama_cpp.1. model = AutoModelForCausalLM.from_pretrained( 2. model_name=“TheBloke/Llama-2-7B-Chat-GGUF”, 3. model_file=“llama-2-7b-chat.Q4_0.gguf” 4. )

Take a look at the code example. The code example demonstrates how to use Intel’s AI Tools, Intel Extension for Transformers, to quantize an LLM model and how to optimize your Intel hardware for Generative AI applications.

INT4 vs INT8

Quantizing LLMs for Inference in INT4/8

Better quantization approaches are becoming more and more necessary as models become bigger. However, what is quantization exactly? Model parameters are represented with less accuracy by quantization. For example, using float16 to represent model weights instead of the widely used float32 may reduce storage needs by half.

Additionally, it improves performance at lesser precision by lowering computational burden. Nevertheless, a drawback of quantization is a little reduction in model accuracy. This happens when accuracy decreases and parameters have less representation power. In essence, quantization allows us to sacrifice accuracy for better inference performance (in terms of processing and storage).

Although there are many other approaches to quantization, this sample only considers Weight Only Quantization (WOQ) strategies. Model weights and activations the output values produced by every neuron in a layer are often quantized. But only the model weights are quantized by WOQ; activations remain unaltered. In actual use, WOQ improves performance without appreciably affecting accuracy.

The transformers library from HuggingFace makes quantization easier by offering clear choices. To enable quantization, users just need to specify the load_in_4bit or load_in_8bit option to True. But there’s a catch: only CUDA GPUs can use this capability. Unfortunately, only CUDA GPU devices can use the BitsandBytes configuration that is automatically built when these arguments are enabled. For consumers using CPUs or non-CUDA devices, this presents a problem.

The Intel team created Intel Extension for Transformers (ITREX), which improves quantization support and provides further optimizations for Intel CPU/GPU architectures, in order to overcome this constraint. Users may import AutoModelForCasualLM from the ITREX library rather of the transformers library in order to use ITREX. This allows users, irrespective of their hardware setup, to effortlessly use quantization and other improvements.

The from_pretrained function has been expanded to include the quantization_config, which now takes in different settings for CUDA GPUs and CPUs to perform quantization, including RtnConfig, AwqConfig, TeqConfig, GPTQConfig, and AutoroundConfig. How things behave when you set the load_in_4bit or load_in_8bit option to True is dependent on how your device is configured.

BitsAndBytesConfig will be used if the CUDA option is selected for your device. RtnConfig, which is specifically tailored for Intel CPUs and GPUs, will be used, nonetheless, if your device is set to CPU. In essence, this offers a uniform interface for using Intel GPUs, CPUs, and CUDA devices, guaranteeing smooth quantization across various hardware setups.

Enhancing deep learning model performance is essential for scalability and efficiency in the rapidly changing field of  artificial intelligence. Intel has been in the forefront of creating frameworks and tools to improve  AI models’ memory efficiency and speed of execution, especially with Intel Extension for PyTorch and Intel Extension for Transformers.

Comprehending the AI Stack

There are several layers in the  AI stack, and each is essential to optimizing LLMs. The hardware layer, which consists of Intel Xeon CPUs, Intel Data Centre GPUs, Intel Arc GPUs, and Intel Gaudi AI accelerators, is fundamental.

The acceleration libraries, such as Intel oneAPI Collective Communications Library (oneCCL) and Intel oneAPI Deep Neural Network Library (oneDNN), sit above this layer and offer optimized kernels with Intel optimized instruction sets for effective processing. The highest layer is made up of resource-efficient frameworks such as PyTorch that interface with the hardware and libraries underneath to optimize model performance.

Important Optimization Methods

Optimizing operators is essential to improving LLM performance. Using enhanced instruction sets such as Intel enhanced Vector Extensions (Intel AVX), Intel Advanced Matrix Extensions (Intel AMX), and Intel Xe Matrix Extensions (Intel XMX), Intel replaces the default operation kernels with highly-optimized Intel oneDNN kernels. The accuracy-flexible design of this optimization ensures that applications can operate at maximum speed and precision by supporting a variety of data types, from FP32 to INT4.

Graph optimizations reduce the amount of memory accesses needed during computation, which further enhances efficiency. For example, memory access times can be reduced by combining layers (e.g., Conv+ReLU+Sum) with bandwidth-limited operations (e.g., activation functions, ReLU, or Tanh).

This method works especially well for models such as ResNet-50, where a large amount of processing time is dedicated to bandwidth-constrained tasks. Specific fusion methods, including as linear post-ops fusion and multi-head attention fusion, are used in the context of LLMs with Intel Extension for PyTorch in JIT/Torch script mode to improve performance.

Memory management is essential for maximizing LLM performance because they frequently require large amounts of memory. By pre-filling key/value pairs before to the onset of autoregressive decoding and utilising pre-allocated buffers throughout the decoding stage, the Segment KV Cache approach maximizes memory use.

This technique increases efficiency by lowering the requirement for in-the-moment memory changes. Similar to this, the Indirect Access KV Cache efficiently manages memory by utilising beam index history and pre-allocated buffers, which lowers the overhead related to memory access during inference.

Model compression uses quantization algorithms, which successively decrease weight and activation precision from FP32 to lower precision forms like INT8 or INT4. This reduction minimizes the size of the model, increases inference speed, and lowers the required for memory bandwidth. Smooth Quant is a post-training quantization technique that shifts the quantization difficulty from activations to weights. This allows for the preservation of model accuracy while mitigating activation outliers and optimizing hardware utilization.

A big part of optimization is also played by custom operators. The goal of weight-only quantization is to increase input and output activation precision by quantizing the model’s weights alone. With minimal influence on accuracy, this technique maximizes computational performance by utilising weight-only quantization-optimized bespoke GEMM (General Matrix Multiply) kernels. Performance can be further optimized by using Explicit SIMD (ESIMD) extensions, which provide more precise control over hardware features.

Intel Extension for PyTorch

APIs for implementing these optimizations on CPU and GPU based training and inference are provided by the Intel Extension for PyTorch. You may make sure that your models are optimized to operate well on Intel hardware by making use of these APIs. To make it easier for developers to execute these optimizations, the extension comes with environment configurations and scripts that are intended to maximize hardware utilization.

Another essential element of Intel’s optimization approach are the Intel Gaudi  AI accelerators. Deep learning applications perform better because to the integration of PyTorch with the Intel Gaudi software suite, which effectively transfers neural network topologies onto Gaudi hardware. This integration also supports important kernel libraries and optimizations.

Intel Extension for Transformers

https://community.intel.com/t5/image/serverpage/image-id/56748i59BB048F0E369A11/image-size/large?v=v2&px=999&whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright

Several plugins for widely used pipelines, like audio processing and retrieval-augmented generation (RAG), can be integrated with Neural Chat. By integrating the required optimizations straight into the pipeline setup, it makes the deployment of optimized chatbots easier.

Neural Velocity and Dispersed Interpretation

https://community.intel.com/t5/image/serverpage/image-id/56751iE6BB93D0A520220B/image-size/large?v=v2&px=999&whitelist-exif-data=Orientation%2CResolution%2COriginalDefaultFinalSize%2CCopyright

DeepSpeed

These optimizations are further expanded across numerous nodes or GPUs via Intel’s support for distributed inference via DeepSpeed. DeepSpeed now supports Intel GPUs thanks to the Intel Extension for DeepSpeed. It includes the following parts:

Implementation of the DeepSpeed Accelerator Interface

Implementation of DeepSpeed op builder for XPU

Code for DeepSpeed op builder kernel

With the help of oneCCL, this Intel-optimized extension distributes compute jobs well, lowering memory footprint and increasing throughput overall. Scaling  AI applications across heterogeneous computer systems requires this capacity.

Utilising Optimizations in Real-World Applications

It’s actually very easy to implement these optimizations using Intel’s tools, as you can use the extensions for the PyTorch and Transformers frameworks. For example, Intel Extension for Transformers improves model compression methods such as weight-only and smooth quantization right inside the well-known Transformers API. By setting the quantization parameters and using the integrated APIs, you may optimize models with ease.

In a similar vein, the Intel Extension for Transformers and PyTorch offers an adaptable framework for optimizing deep learning models other than LLMs. This update provides GPU-centric capabilities like tensor parallelism and  CPU optimizations like NUMA management and graph optimization’s to enable fine-tuning and deployment across a variety of hardware configurations.

In summary

You may significantly increase the effectiveness and performance of your AI models by utilising Intel’s extensive hardware stack, accelerated libraries, and optimized frameworks. These optimizations cut the energy and operating expenses associated with running large-scale  AI applications in addition to improving computational performance and reducing latency.

Using the getting started samples from the Intel Extension for PyTorch and Intel Extension for Transformers, you can investigate these optimizations on the Intel Tiber Developer Cloud. You can make sure your LLMs are operating at optimal performance on Intel hardware by incorporating these strategies.