#oneapi

Kernel_Compiler

The kernel_compiler, which was first released as an experimental feature in the fully SYCL2020 compliant Intel oneAPI DPC++/C++ compiler 2024.1 is one of the new features. Here’s another illustration of how Intel advances the development of LLVM and SYCL standards. With the help of this extension, OpenCL C strings can be compiled at runtime into kernels that can be used on a device.

For offloading target hardware-specific SYCL kernels, it is provided in addition to the more popular modes of Ahead-of-Time (AOT), SYCL runtime, and directed runtime compilation.

Generally speaking, the kernel_compiler extension ought to be saved for last!

Nonetheless, there might be some very intriguing justifications for leveraging this new extension to create SYCL Kernels from OpenCL C or SPIR-V code stubs.

Let’s take a brief overview of the many late- and early-compile choices that SYCL offers before getting into the specifics and explaining why there are typically though not always better techniques.

Three Different Types of Compilation 

The ability to offload computational work to kernels running on another compute device that may be installed on the machine, such as a GPU or an FPGA, is what SYCL offers your application. Are there thousands of numbers you need to figure out? Forward it to the GPU!

Power and performance are made possible by this, but it also raises more questions:

Which device are you planning to target? In the future, will that change?

Could it be more efficient if it were customized to parameters that only the running program would know, or do you know the complete domain parameter value for that kernel execution? SYCL offers a number of choices to answer those queries:

Ahead-of-Time (AoT) Compile: This process involves compiling your kernels to machine code concurrently with the compilation of your application.

SYCL Runtime Compilation: This method compiles the kernel while your application is executing and it is being used.

With directed runtime compilation, you can set up your application to generate a kernel whenever you’d want.

Let’s examine each one of these:

1. Ahead of Time (AoT) Compile

You can also precompile the kernels at the same time as you compile your application. All you have to do is specify which devices you would like the kernels to be compiled for. All you need to do is pass them to the compiler with the -fsycl-targets flag. Completed! Now that the kernels have been compiled, your application will use those binaries.

AoT compilation has the advantage of being easy to grasp and familiar to C++ programmers. Furthermore, it is the only choice for certain devices such as FPGAs and some GPUs.

An additional benefit is that your kernel can be loaded, given to the device, and executed without the runtime stopping to compile it or halt it.

Although they are not covered in this blog post, there are many more choices available to you for controlling AoT compilation. For additional information, see this section on compiler and runtime design or the -fsycl-targets article in Intel’s GitHub LLVM User Manual.

SPIR-V

2. SYCL Runtime Compilation (via SPIR-V) 

If no target devices are supplied or perhaps if an application with precompiled kernels is executed on a machine with target devices that differ from what was requested, this is SYCL default mode.

SYCL automatically compiles your kernel C++ code to SPIR-V (Standard Portable Intermediate form), an intermediate form. When the SPIR-V kernel is initially required, it is first saved within your program and then sent to the driver of the target device that is encountered. The SPIR-V kernel is then converted to machine code for that device by the device driver.

The default runtime compilation has the following two main benefits:

First of all, you don’t have to worry about the precise target device that your kernel will operate on beforehand. It will run as long as there is one.

Second, if a GPU driver has been updated to improve performance, your application will benefit from it when your kernel runs on that GPU using the new driver, saving you the trouble of recompiling it.

However, keep in mind that there can be a minor cost in contrast to AoT because your application will need to compile from SPIR-V to machine code when it first delivers the kernel to the device. However, this usually takes place outside of the key performance route, before parallel_for loops the kernel.

In actuality, this compilation time is minimal, and runtime compilation offers more flexibility than the alternative. SYCL may also cache compiled kernels in between app calls, which further eliminates any expenses. See kernel programming cache and environment variables for additional information on caching.

However, if you prefer the flexibility of runtime compilation but dislike the default SYCL behavior, continue reading!

3. Directed Runtime Compilation (via kernel_bundles) 

You may access and manage the kernels that are bundled with your application using the kernel_bundle class in SYCL, which is a programmatic interface.

Here, the kernel_bundle techniques are noteworthy.build(), compile(), and link(). Without having to wait until the kernel is required, these let you, the app author, decide precisely when and how a kernel might be constructed.

Additional details regarding kernel_bundles are provided in the SYCL 2020 specification and in a controlling compilation example.

Specialization Constants 

Assume for the moment that you are creating a kernel that manipulates an input image’s numerous pixels. Your kernel must use a replacement to replace the pixels that match a specific key color. You are aware that if the key color and replacement color were constants instead of parameter variables, the kernel might operate more quickly. However, there is no way to know what those color values might be when you are creating your program. Perhaps they rely on calculations or user input.

Specialization constants are relevant in this situation.

The name refers to the constants in your kernel that you will specialize in at runtime prior to the kernel being compiled at runtime. Your application can set the key and replacement colors using specialization constants, which the device driver subsequently compiles as constants into the kernel’s code. There are significant performance benefits for kernels that can take advantage of this.

The Last Resort – the kernel_compiler 

All of the choices that as a discussed thus far work well together. However, you can choose from a very wide range of settings, including directed compilation, caching, specialization constants, AoT compilation, and the usual SYCL compile-at-runtime behavior.

Using specialization constants to make your program performant or having it choose a specific kernel at runtime are simple processes. However, that might not be sufficient. Perhaps all your software needs to do is create a kernel from scratch.

Here is some source code to help illustrate this. Intel made an effort to compose it in a way that makes sense from top to bottom.

When is It Beneficial to Use kernel_compiler? 

Some SYCL users already have extensive kernel libraries in SPIR-V or OpenCL C. For those, the kernel_compiler is a very helpful extension that enables them to use those libraries rather than a last-resort tool.

Download the Compiler 

Download the most recent version of the Intel oneAPI DPC++/C++ Compiler, which incorporates the kernel_compiler experimental functionality, if you haven’t already. Purchase it separately for Windows or Linux, via well-known package managers only for Linux, or as a component of the Intel oneAPI Base Toolkit 2024.

The open-source project known as llama.cpp is a lightweight LLM framework that is gaining greater and greater popularity. Given its performance and customisability, developers, scholars, and fans have formed a strong community around the project. Since its launch, GitHub has over 600 contributors, 52,000 stars, 1,500 releases, and 7,400 forks. More hardware, including Intel GPUs seen in server and consumer products, is now supported by llama.cpp as a result of recent code merges. Hardware support for GPUs from other vendors and CPUs (x86 and ARM) is now combined with Intel’s GPUs.

Georgi Gerganov designed the first implementation. The project is mostly instructional in nature and acts as the primary testing ground for new features being developed for the machine learning tensor library known as ggml library. Intel is making  AI more accessible to a wider range of customers by enabling inference on a greater number of devices with its latest releases. Because Llama.cpp is built in C and has a number of other appealing qualities, it is quick.

16-bit float compatibility

Support for integer quantisation (four-, five-, eight-, etc.)

Absence of reliance on outside parties

There are no runtime memory allocations.

Intel GPU SYCL Backend

GGM offers a number of backends to accommodate and adjust for various hardware. Since oneAPI supports GPUs from multiple vendors, Intel decided to construct the SYCL backend using their direct programming language, SYCL, and high-performance BLAS library, oneMKL. A programming model called SYCL is designed to increase hardware accelerator productivity. It is an embedded, single-source language with a domain focus that is built entirely on C++17.

All Intel GPUs can be used with the SYCL backend. Intel has confirmed with:

Flex Series and Data Centre GPU Max from Intel

Discrete GPU Intel Arc

Intel Arc GPU integrated with the Intel Core Ultra CPU

In Intel Core CPUs from Generations 11 through 13: iGPU

Millions of consumer devices can now conduct inference on Llama since llama.cpp now supports Intel GPUs. The SYCL backend performs noticeably better on Intel GPUs than the OpenCL (CLBlast) backend. Additionally, it supports an increasing number of devices, including CPUs and future processors with  AI accelerators. For information on using the SYCL backend, please refer to the llama.cpp tutorial.

Utilise the SYCL Backend to Run LLM on an Intel GPU

For SYCL, llama.cpp contains a comprehensive manual. Any Intel GPU that supports SYCL and oneAPI can run it. GPUs from the Flex Series and Intel Data Centre GPU Max can be used by server and cloud users. On their Intel Arc GPU or iGPU on Intel Core CPUs, client users can test it out. The 11th generation Core and later iGPUs have been tested by Intel. While it functions, the older iGPU performs poorly.

The memory is the only restriction. Shared memory on the host is used by the iGPU. Its own memory is used by the dGPU. For llama2-7b-Q4 models, Intel advise utilising an iGPU with 80+ EUs (11th Gen Core and above) and shared memory that is greater than 4.5 GB (total host memory is 16 GB and higher, and half memory could be assigned to iGPU).

Put in place the Intel GPU driver

There is support for Windows (WLS2) and Linux. Intel suggests Ubuntu 22.04 for Linux, and this version was utilised for testing and development.

Linux:sudo usermod -aG render username sudo usermod -aG video username sudo apt install clinfo sudo clinfo -l

Output (example):Platform #0: Intel(R) OpenCL Graphics -- Device #0: Intel(R) Arc(TM) A770 Graphics

orPlatform #0: Intel(R) OpenCL HD Graphics -- Device #0: Intel(R) Iris(R) Xe Graphics \[0x9a49\]

Set the oneAPI Runtime to ON

Install the Intel oneAPI Base Toolkit first in order to obtain oneMKL and the SYCL compiler. Turn on the oneAPI runtime next:

First, install the Intel oneAPI Base Toolkit to get the SYCL compiler and oneMKL. Next, enable the oneAPI runtime:

Linux: source /opt/intel/oneapi/setvars.sh

Windows: “C:\Program Files (x86)\Intel\oneAPI\setvars.bat\” intel64

Run sycl-ls to confirm that there are one or more Level Zero devices. Please confirm that at least one GPU is present, like [ext_oneapi_level_zero:gpu:0].

Build by one-click:

Linux: ./examples/sycl/build.sh

Windows: examples\sycl\win-build-sycl.bat

Note, the scripts above include the command to enable the oneAPI runtime.

Run an Example by One-Click

Download llama-2–7b.Q4_0.gguf and save to the models folder:

Linux: ./examples/sycl/run-llama2.sh

Windows: examples\sycl\win-run-llama2.bat

Note that the scripts above include the command to enable the oneAPI runtime. If the ID of your Level Zero GPU is not 0, please change the device ID in the script. To list the device ID:

Linux: ./build/bin/ls-sycl-device or ./build/bin/main

Windows: build\bin\ls-sycl-device.exe or build\bin\main.exe

Synopsis

All Intel GPUs are available to LLM developers and users via the SYCL backend included in llama.cpp. Kindly verify whether the Intel laptop, your gaming PC, or your cloud virtual machine have an iGPU, an Intel Arc GPU, or an Intel Data Centre GPU Max and Flex Series GPU. If so, llama.cpp’s wonderful LLM features on Intel GPUs are yours to enjoy. To add new features and optimise SYCL for Intel GPUs, Intel want developers to experiment and contribute to the backend. The oneAPI programming approach is a useful project to learn for cross-platform development.

PyTorch 2.4 Launches to Speed Up AI Tasks with Initial Support for Intel GPUs. In order to further speed up AI tasks, PyTorch 2.4 now offers initial support for the  Intel Data Centre GPU Max Series, which integrates  Intel GPUs and the SYCL software stack into the standard PyTorch stack.

Advantages

With Intel GPU support, customers have more options for GPUs and can use a consistent front-end and back-end GPU programming model. Workloads can now be deployed and operated on Intel GPUs with little coding required. To support streaming devices, this version generalizes the PyTorch device and runtime (device, stream, event, generator, allocator, and guard). The generalization facilitates not only PyTorch’s deployment on widely available hardware but also the integration of many hardware back ends.

Integrated PyTorch provides continuous software support, standardized software distribution, and consistent product release schedules, all of which will improve the experience for users of Intel GPUs.

An Overview of Support for Intel GPUs

Eager mode and graph mode are supported in the PyTorch built-in front end thanks to Intel GPU support that has been up streamed into the program. The SYCL programming language is now utilized to implement popular Aten operators in the eager mode. OneAPI Math Kernel Library (oneMKL) and oneAPI Deep Neural Network Library (oneDNN) are used to highly optimize the most performance-critical graphs and operators. To perform the optimization for Intel GPUs and to integrate Triton, the graph mode (torch.compile) now has an enabled Intel GPU back end.

PyTorch 2.4 now includes the necessary parts of Intel GPU support: Aten operators, oneDNN, Triton, Intel GPU source build, and integration of Intel GPU tool chains. In the meantime, PyTorch Profiler which is built on an integration between Kineto and oneMKL is being actively worked on in front of the forthcoming PyTorch 2.5 release. The front-end and back-end enhancements for Intel GPUs that are currently being implemented into PyTorch are depicted in Figure 1.Image Credit To Intel

PyTorch 2.4 Features

Apart from offering essential functionalities for training and inference on the Intel Data Centre GPU Max Series, the PyTorch 2.4 release for Linux maintains the same user interface as other supported hardware for PyTorch.

Using an Intel GPU, PyTorch 2.4 features include:

Workflows for inference and training.

The core eager functions as well as torch.compile are supported, and both eager and compile modes can fully run a Dynamo Hugging Face benchmark.

Data types like automated mixed precision (AMP), BF16, FP32, and so on.

operates on the Intel Data Centre GPU Max Series and Linux.

PyTorch 2.5

The first Intel GPU from the Intel Data Centre GPU Max Series is now available in the PyTorch ecosystem for AI workload acceleration thanks to the Intel GPU on PyTorch 2.4 first support (prototype) release.

In order to achieve beta quality in the PyTorch 2.5 release, they are constantly improving the functionality and performance of the Intel GPU support. Intel Client GPUs will be added to the list of GPUs supported for AI PC use cases as the product develops further. They’re also investigating more features for PyTorch 2.5, like:

Eager Mode: Completely execute Dynamo Torchbench and TIMM eager mode, and implement additional Aten operators.

Torch.compile: Optimise performance while running Dynamo Torchbench and TIMM benchmark compile mode in full.

To support Intel GPU, enable torch.profile under the profiler and utilities section.

Distribution of PyPI wheels.

Support for Windows and the  Intel Client GPU Series.

They invite the community to assess these latest additions to PyTorch’s  Intel GPU support.

Intel Extensions For PyTorch

The most recent performance enhancements for Intel devices are added to PyTorch using the Intel Extension. The Intel XeMatrix Extensions (XMX) AI engines on Intel discrete GPUs and Intel Advanced Vector Extensions 512 (Intel AVX-512) Vector Neural Network Instructions (VNNI) and Intel Advanced Matrix Extensions (Intel AMX) on Intel CPUs are utilized in optimizations. Additionally, the PyTorch xpu device, in conjunction with Intel Extension for PyTorch, facilitates simple GPU acceleration for Intel discrete GPUs.

Workloads and models for Generative AI (GenAI) have become increasingly common in today’s technological environment. These GenAI applications are mostly driven by large language models, or LLMs. The Intel Extension for PyTorch has added special optimizations for a few Large Language Models (LLMs) as of version 2.1.0. See Large Language Models (LLMs) section for additional details on LLM optimizations.

For Python programs, the extension can be loaded as a module, and for C++ projects, it can be linked as a library. It can be dynamically enabled in Python programs by importing intel_extension_for_pytorch.

Buildings

Eager Mode: Custom Python modules (including fusion modules), optimum optimizers, and INT8 quantization APIs are added to the PyTorch frontend in the eager mode. Using extended graph fusion passes, eager-mode models can be transformed into graph mode to further increase performance.

Graph Mode: Performance is enhanced by fusions’ reduction of operator/kernel invocation overhead in the graph mode. In PyTorch, the graph mode typically produces better results from optimization techniques like operation fusion than the eager mode does.

They are enhanced by the Intel Extension for PyTorch, which offers more thorough graph optimizations. Supported graph modes are PyTorch Torchscript and TorchDynamo. They advise you to use torch.jit.trace() instead of torch.jit.script() when using Torchscript since it typically supports a larger variety of workloads. The ipex backend can deliver strong performance with TorchDynamo.

CPU Optimization: Based on the detected instruction set architecture (ISA), Intel Extension for PyTorch automatically assigns operators to underlying kernels on the CPU. The addon makes use of the Intel hardware’s vectorization and matrix acceleration units. For enhanced performance, the runtime extension provides weight sharing and more precise thread runtime management.

Intel GPU

GPU Optimisation: The PyTorch dispatching method is used to implement and register optimized operators and kernels on the GPU. The intrinsic vectorization and matrix calculating capabilities of Intel GPU hardware enhance certain operators and kernels. The DPC++ compiler, which supports both the most recent SYCL standard and several extensions to the SYCL standard, is used by the Intel Extension for PyTorch for GPU. These extensions are located in the sycl/doc/extensions directory.

Encouragement

GitHub issues are used by the team to keep track of bugs and enhancement requests. Check to see whether your issue has previously been reported on GitHub before making a proposal or bug report.

Hyperdimensional Computing The high-dimensional and distributed information representation found in the brain serves as the model for Hyperdimensional Computing (HDC), a machine learning technique. Hypervectors serve as the foundation for HDC, which, like other AI techniques, is implemented through training and inference stages.

HDC Hypervectors: Instruction and Interpretation Hypervectors, which are enormous vectors, are how Hyperdimensional Computing represents data. Hypervectors can be examined for similarities and differences using the fundamental Hyperdimensional Computing operations (similarity, bundling, binding, and permutation), and an encoding system can be created.

Training: In training, input feature vectors are transformed into hypervectors, which are compared and grouped into classes according to how similar they are or are not to each other. Ultimately, a class hypervector that represents every class element is created by bundling the hypervectors within each class. An HDC-based classification system can be developed thanks to this class hypervector.

Inference: Each new feature vector is similarly converted to a hypervector once training has succeeded in creating class hypervectors from a series of input feature vectors. This allows for a quick, effective, and reliable classification system to be established by comparing the new hypervector to the pre-existing class hypervectors and determining whether the data is similar.

The inherent advantages of a classification system based on HDC are as follows: Efficiency: Compared to conventional techniques, hypervectors provide the efficient and compact representation of complicated patterns, improving processing and classification jobs’ efficiency.

Minimal Power: Hyperdimensional Computing can be implemented in hardware with minimal power consumption because of its simplicity and the binary nature of its operations (XOR, AND, etc.). This is especially helpful for edge computing applications, wearable technology, and Internet of Things devices where energy efficiency is critical.

Extremely Parallel: Hyperdimensional Computing‘s distributed architecture permits parallel processing, which is comparable to the brain’s capacity for parallel processing. For classification problems, this can greatly reduce computing times.

Quick Learning: Unlike deep learning models, which frequently need a large amount of training data, Hyperdimensional Computing may do one-shot or few-shot learning, in which the system learns from a relatively small number of samples. Because of this feature, HDC is very beneficial in situations when data is limited or changing quickly.

Robust and Reliable: Due to its great dimensionality, Hyperdimensional Computing is intrinsically resistant to mistakes and noise. Reliable categorization is possible even in noisy situations because little adjustments or distortions in the input data do not dramatically alter the overall representation. Furthermore, the manipulation of hypervectors involves a large number of simple, repetitive operations, which makes them very suitable to hardware platform acceleration.

The efficient, low-power, highly parallel, and hardware-implementable nature of Hyperdimensional Computing applications makes them perfect for running on Altera FPGAs, which are also extremely parallel and efficient.

Utilising Altera FPGAs and oneAPI Intel oneAPI Base ToolKit simplifies cross-architecture, high-performance application development. The Intel oneAPI Base Toolkit implements the industry standard and works with FPGAs, CPUs, GPUs, and AI accelerators.

The development process is made simpler by oneAPI, which is one of its primary advantages. Developers don’t need to learn multiple programming languages to construct SYCL/C++ apps that operate on diverse architectures when they use oneAPI. This saves a tonne of time and effort for developers because they can write code only once and have it operate on multiple processors. They can also create software applications and deploy them on the most economical and efficient platform without having to modify the code.

With Intel oneAPI Base Toolkit, Hyperdimensional Computing applications written in SYCL/C++ can be immediately implemented into Altera FPGAs.

To get started with oneAPI for FPGAs, see Boosting Productivity with High-Level Synthesis in Intel Quartus with oneAPI for information on tutorial examples, self-start learning videos, and reference designs.

Hyperdimensional Computing Picture Categorization using Altera FPGAs Image classification is one scenario where HDC is applied in a practical setting. An HDC image classification system was implemented on an Altera FPGA for training and inference in this example, which was developed by Ian Peitzsch at the Centre for Space, High-Performance, and Resilient Computing (SHREC).

While the data flow is similar in both scenarios, the computations differ. Streaming feature vectors are sent to the FPGA from the host. Every vector is directed to one of eight compute units in the training flow and one of twenty-five compute units in the inference flow.

Training: The FPGA receives a stream of feature vectors from the host. The parallel encode compute units (CUs) receive each vector. Following the bundling of the output-encoded partial hypervectors, a label is produced. The class associated with the label receives the bundle of hypervectors. Following the processing of all the training data, the class hypervectors are streamed to the host in order to be normalised. Inference: The FPGA receives a stream of feature vectors from the host. Every vector is sent to CUs that are parallel encoded. To create a single hypervector, the data is routed to the classification kernel. Every class hypervector is compared to this one, and the prediction belongs to the class that has the greatest resemblance. After then, the host receives a stream of the prediction.

HDC classification algorithms are a perfect fit for Altera FPGAs due to intrinsic properties of both oneAPI and FPGAs, in addition to the fact that Hyperdimensional Computing applications are often quite accessible to hardware implementation.

The SYCL language’s Universal Shared Memory (USM) feature is supported by oneAPI. In the code-based and final hardware, SYCL USM uniquely permits the host and accelerator to share the same memory, unlike C or C++ alternatives. This lowers system latency to enhance overall performance and permits the simple, industry-standard coding technique of using a pointer to explicitly access data, whether on the host or the accelerator. The bottleneck is at the encoding step. The parallel nature of a programmable FPGA enables the usage of many compute units in parallel because each dimension of a hypervector can be encoded independently. When utilising several compute units, the inference time is greatly decreased by this parallel processing. During training and inference, parallel processing was employed in the encoding stage, and Altera FPGA’s USM feature guaranteed a low latency solution.

At AI Inference, Altera FPGAs Perform Well This Hyperdimensional Computing image classification algorithm was built on CPU, GPU, and FPGA resources in a real-world evaluation1.

Cascade Lake-based Intel Xeon Platinum 8256 CPU (3.8GHz, 4 Cores). The eleventh generation Intel UHD 630 GPU. FPGA Intel Stratix 10 GX. An Hyperdimensional Computing classification model with 2000 hyperdimensions of 32-bit floating-point values was utilised in all implementations. All three implementations obtained similar accuracy of about 94-95% using the NeuralHD retraining algorithm.

The outcomes highlight the advantages of utilising an FPGA during the inference phase. The total amount of time spent training the model is the important metric. Latency is the key performance indicator during inference.

Through training, the FPGA provided an x18 speed-up and the Intel GPU reached a 60x speed-up over the CPU. On the other hand, the accuracy of the CPU and FPGA implementations was about 97%, whereas the GPU implementation only managed to reach about 94% accuracy.

Due to memory constraints in this training scenario, the FPGA could only employ 8 CU in parallel during encoding. More parallel CUs and a quicker FPGA training period might be possible with a bigger FPGA. The evaluation was carried out using fixed resources on Intel DevCloud, and a larger FPGA was not accessible.

Benefits of FPGA AI Latency With a 3× speed increase over the CPU, the FPGA exhibits the lowest latency during the inference stage. It is demonstrated that the GPU is overkill for inference, requiring significantly more time than the FPGA or even the CPU (demonstrating a slow-down rather than an acceleration).

These findings demonstrate the advantages of utilising an FPGA for AI inference once more. Altera FPGAs are the best option for quick and effective AI inference in data centre and edge applications because they combine the advantages of performance and generally lower costs with license-free oneAPI development tools. Any AI algorithm is trained/developed once and then implemented for inference on a mass scale.

AI Reference Kits Features

With a concentration on AI/ML research, Dev Aryan Khanna is an Intel Student Ambassador for oneAPI. His most recent project was the Healthcare AI Reference Kits Companion, an intelligent healthcare application that combines text-based symptom analysis and image classification with cutting-edge deep learning models. The application aids in the early diagnosis of illnesses using X-ray pictures and symptom descriptions. For model optimization, it makes use of oneAPI libraries and Intel AI Reference Kits.

Concerning the Healthcare AI Companion Initiative

The intrinsic constraint of current healthcare solutions designed exclusively for particular GPUs served as the driving force behind the creation of the Healthcare AI Reference Kits Companion. This was a serious issue since it made it more difficult to adjust to Intel CPUs, which affected the customer base worldwide. By creating an intelligent healthcare tool that is designed for a variety of architectures and focuses on achieving effective performance on Intel hardware, the project seeks to close this gap.

In order to construct the Healthcare AI Reference Kits Companion, oneAPI tools with cross-architecture were utilized. With the use of these tools, developers may create single-language and platform apps that are easily adapted to and optimized for a variety of architectures, including CPUs from Intel. These instruments guarantee flexibility, effectiveness, and ease of use, which is consistent with the project’s objective of offering a complete and adjustable healthcare solution.

Use of Intel Software and Hardware Optimizations

Intel Deep Neural Network (oneDNN) Library via oneAPI

How: Showcase how oneDNN may increase CNN layers’ performance for image-based illness identification.

What: Convolutional layers are optimized by the library, which is essential for improving the effectiveness of illness identification in X-ray pictures.

When: During the training and inference phases of the project life cycle, in particular.

The Data Analytics Library (oneDAL) of Intel oneAPI

How: OneDAL was used to expedite feature engineering operations inside the data preprocessing pipeline.

What: Although not the main goal, oneDAL helps to improve feature engineering and guarantee data quality, which are essential for precise disease detection models.

When: Just before the model is trained, during the data preparation stage.

PyTorch Intel Extension

How: The comparison graph shows how the Intel Extension for PyTorch was able to accelerate model training. Mixed-precision training is also made possible by the PyTorch optimizations.

What: The Intel Extension for PyTorch allows for mixed-precision training without compromising accuracy, optimizes model training, and speeds up deep learning workloads on Intel CPUs.

when: Mainly while the model is being trained.

The Intel Student Ambassador’s Success Story

Indian undergraduate Dev Aryan Kanna attends Guru Gobind Singh Indraprastha University. His experience as an Intel Student Ambassador for oneAPI has been enlightening. As part of the Intel Student Ambassador Program, he has organized workshops, hackathons, and worked on projects that have improved his own learning and given the student attendees invaluable practical experience. With the program, he has access to a wealth of resources, including the newest Intel hardware, which has enhanced his knowledge of Intel technologies and stoked his love for creative problem-solving.

His ambition to use technology to better healthcare has been inspired by the Healthcare AI Reference Kits Companion project, which placed first runner-up in a recent Intel Student Ambassador Hackathon. He hopes to continue making a significant contribution to the nexus of artificial intelligence and healthcare with continued access to Intel’s capabilities.

A cutting-edge strategy for improving single-threaded CPU speed is Single Instruction Multiple Data (SIMD).

FPGAs are known for high-performance computing via customizing circuits for algorithms. Their tailored and optimized hardware accelerates difficult computations.

SIMD and FPGAs seem unrelated, yet this blog article will demonstrate their compatibility. By enabling data parallel processing, FPGAs can boost processing performance with SIMD. For many computationally intensive activities, FPGA adaptability and SIMD efficiency are appealing.

High-performance SIMDified programming

SIMD parallel processing applies a single instruction to numerous data objects. Special hardware extensions can execute the same instruction on several data objects simultaneously.

SIMDified processing uses data independence to boost software application performance by rewriting application code to use SIMD instructions extensively.

Key advantages of SIMDified processing include:

Increased performance: SIMDified processing boosts computationally intensive software applications.

Integrability: Intrinsics and dedicated data types make SIMDified processing desirable.

SIMDified processing is available on many current processors, giving it a viable option for computational speed improvement.

Despite its benefits, SIMDified processing is not ideal for many applications. Applications with minimal data parallelism will not benefit from SIMDified processing. It is a convincing method for improving data-intensive software applications.

SIMD Portability Supports Heterogeneity

SIMD registers and instructions make up SIMD instruction sets. SIMD intrinsics in C/C++ are the best low-level programming method for performance.

Low-level programming in heterogeneous settings with different hardware platforms, operating systems, architectures, and technologies is difficult due to hardware capabilities, data parallelism, and naming standards.

Specialized implementations limit portability between platforms, hence SIMD abstraction libraries provide a common SIMD interface and abstract SIMD functions. These libraries use C++ template metaprogramming and function template specializations to translate to SIMD intrinsics and potential compensations for missing functions, which must be implemented.

C/C++ libraries let developers construct SIMD-hardware-oblivious application code and SIMD extension code with minimum overhead. Separating SIMD-hardware-oblivious code with a SIMD abstraction library simplifies both sides.

This method has promoted many SIMD libraries and abstraction layers to solve problems:

Examples of SIMD libraries

Google Highway (open-source)

Xsimd (C++ wrapper for SIMD instances)

Such libraries allow SIMDified code to be designed once and specialized for the target SIMD platform by the SIMD abstraction library. Libraries and varied design environments suit SIMD instructions and abstraction.

Accelerating with FPGAs

FPGAs speed software at low cost and power. Traditional FPGAs required a strong understanding of digital design concepts and specific languages like VHDL or Verilog. FPGA-based solutions are harder to access and more specialized than CPU or GPU-based computing platforms due to programming complexity and code portability. Intel oneAPI changes this.

Intel oneAPI is a software development kit that unifies CPU, GPU, and FPGA programming. It supports C++, Fortran, Python, and Data Parallel C++ (DPC++) for heterogeneous computing to improve performance, productivity, and development time.

Since Intel oneAPI can target FPGAs from SYCL/C++, software developers are increasingly interested in using them for data processing. FPGAs can be used with SIMDified applications by adding them as a backend to the SIMD abstraction library. This allows SIMD applications with FPGAs.

SIMD and FPGAs go together Annotations let the Intel DPC++ compiler synthesis C++ code into circuits and auto-vectorize data-parallel processing. Annotating and implementing code arrays as registers on an FPGA removes data access constraints and allows parallel processing from sink to source. This enables SIMD performance acceleration using FPGAs straightforward and configurable.

SIMD abstraction libraries are a logical choice for FPGA SIMD processing. As noted, the libraries support Intel and ARM SIMD instruction set extensions. TSL abstraction library simplifies FPGA SIMD instruction implementation in the following example. The scalar code specifies loading registers, and the pragma unroll attribute tells the DPC++ Compiler to implement all pathways in parallel in the generic element-wise addition example below.

This simple element-wise example has no dependencies, and comparable implementations will work for SIMD instructions like scatter, gather, and store. Optimization can also accelerate complex instructions.

A horizontal reduction requires a compile-time adder tree of depth ld(N), where N is the number of entries. Unroll pragmas with compile-time constants can implement adder trees in a scalable manner, as shown in the following code example.

Software that calls a library of comparable SIMD components can expedite SIMD instructions on Intel FPGAs by adding the examples above.

Intel FPGA Board Support Package adds system benefits. Intel FPGAs use a BSP to describe hardware interfaces and offer a kernel shell.

The BSP enables SYCL Universal Shared Memory (USM), which frees the CPU from data transfer management by exchanging data directly with the accelerator. FPGAs can be coprocessors.

The pre-compiled BSP generates only kernel logic live, reducing runtime.

Intel FPGAs are ideal for SIMD and streaming applications like current composable databases because to their C++/SYCL compatibility, CPU data transfer offloading, and pre-compiled BSPs.

SIMD/FPGA simplicity At SiMoDSIGMOD 2023 in Seattle, USA, Dirk Habich, Alexander Krause, Johannes Pietrzyk, and Wolfgang Lehner of TU Dresden presented their paper “Simplicity done right for SIMDified query processing on CPU and FPGA” on using FPGAs to accelerate SIMD instructions. The work, supported by Intel’s Christian Färber, illustrates how practical and efficient developing a SIMDified kernel in an FPGA is while achieving top performance.

The paper evaluated FPGA acceleration of SIMD instructions using a dual-socket 3rd-generation Intel Xeon Scalable processor (code-named “Ice Lake”) with 36 cores and a base frequency of 2.2 GHz and a BitWare IA-840f acceleration card with an Intel Agilex 7 AGF027 FPGA and 4x 16 GB DDR4 memories.

First, they gradually increased the SIMD instance register width to see how it affected maximum acceleration bandwidth. The first instance, a simple aggregation, revealed that the FPGA accelerator’s bandwidth improves with data width doubling until the global bandwidth saturates an ideal acceleration case.

The second scenario, a filter-count kernel with a data dependency in the last stage of the adder tree, demonstrated similar behavior but saturates earlier at the PCIe link width. Both scenarios demonstrate the considerable speeding gains of natively parallel instructions on a highly parallel architecture and suggest that wide memory accesses could sustain the benefits.

Final performance comparisons compared the FPGA and CPU. CPU and FPGA received the same multi-threaded AVX512-based filter-count kernel. As expected, per-core CPU bandwidth decreased as thread count and CPU core count grew. FPGA performance was peak across all workloads.

Based on this work, the TU Dresden and Intel team researched how to use TSL to use an FPGA as a bespoke SIMD processor.