#rocm

하정우 AI 수석은 AI 시대의 일자리 소멸 우려가 거짓말이 아니라고 명확히 밝혔습니다. 한국 AI 산업과 일자리 변화, AMD GPU와 ROCm 소프트웨어의 역할까지 살펴봅니다. 목차 AI 시대 한국 일자리 변화와 정부 대응 AMD 리사수 회장 방한과 GPU 공급 다변화 ROCm과 엔비디아 CUDA 경쟁 구도 하정우 AI 수석이 제시한 AI 시대 대응 전략 AI 시대 일자리 변화와 기술 도입 비교 AI 시대 한국 일자리 변화와 정부 대응 하정우 AI 수석의 일자리 전망 하정우 AI 수석은 AI 기술 발전으로 인한 일자리 소멸 우려가 단순한 과장이 아니라고 밝혔습니다. AI가 반복적이고 단순 업무를 대체하며, 일부 직종에서는 고용 감소가 불가피합니다. 다만, 새로운 일자리 창출과 직무 재설계가…

AMD Instinct Accelerators

AMD Instinct MI300X Accelerators with ROCm Software: Boost Your LLMs

Although large language models (LLMs) appear to be widely available and unrestricted, fierce rivalry exists behind the scenes for the AMD Instinct GPUs resources required to run them. For those wishing to develop and implement LLMs and their visual counterparts, there are substantial obstacles due to cost, availability, and performance limits.

These models place heavy demands on memory and processing power due to their reliance on processing billions of parameters at once. Their impressive powers are made possible by their huge scale, but this also makes it difficult to deploy them economically. TCO issues can also arise from AI inferencing, which uses trained models to produce and provide predictions or outputs. But the AMD Instinct MI300X accelerator aids in removing these obstacles and maximising LLM potential.

AMD MI300X accelerator Vs Nvidia H200

Huge memory bandwidth and the ability to accommodate larger models

Large datasets and computations required by LLMs require high bandwidth, which is essential for enabling quicker processing, lower latency, and improved overall performance. With a peak memory bandwidth of up to 5.3 TB/s, the AMD MI300X accelerator outperforms the Nvidia H200 by a wide margin.

The MI300X does not require splitting models of this magnitude across many GPUs because it can support models with up to 80 billion parameters on a single GPU because to its 192 GB of HBM3 memory. On the other hand, the Nvidia H200, which has 141 GB of HBM2e memory, could need to split models, which would complicate matters and reduce data transfer efficiency.

More of the model can be stored closer to the computing units thanks to the AMD Instinct GPUs huge memory capacity, which lowers latency and boosts performance. Furthermore, the MI300X’s enormous memory capacity allows it to manage numerous large models on a single GPU, which solves the problem of dividing these models between AMD Instinct GPUs and the related execution complexity that accompany this operation.

The MI300X is a great option for handling the rigorous requirements of LLMs since it minimises potential inefficiencies in data transfer, which simplifies implementation and improves performance.

Due of its massive memory capacity and high bandwidth, the MI300X GPU can complete tasks on a single GPU that the H200 would require numerous AMD Instinct GPUs to complete. This can reduce expenses and ease deployment. These features can increase performance while simplifying the management of many GPUs. It might require less GPUs to run a model like ChatGPT on the MI300X than it would on the H200. making it a fantastic choice for businesses looking to implement cutting-edge  AI models.

Using Flash Attention to Improve LLM inference

Flash Attention, a significant advancement in optimising LLM inference on GPUs, is supported by AMD Instinct GPUs such the MI300X. Conventional attention techniques cause bottlenecks because they require numerous reads and writes to high-bandwidth memory (HBM). To combat this, Flash Attention reduces data transmission and boosts speed by consolidating processes like activation and dropout into a single step. LLMs especially benefit from this optimisation since it enables quicker and more effective processing.

AMD Instinct MI300X

Performance of floating point operations

One key indicator of LLM performance is floating point operation performance. Up to 1.3 PFLOPS of half-precision floating point (FP16) and 163.4 TFLOPS of single-precision floating point (FP32) performance are provided by the MI300X. These performance thresholds contribute to the precise and efficient operation of the intricate calculations involved in LLMs. Deep-learning models rely on complex numerical computations for tasks like matrix multiplications and tensor operations, for which this performance is also important.

The MI300X can handle several operations at once because of its advanced parallelism-supporting architecture. The MI300X can easily manage the large amount of parameters in LLMs thanks to its 304 compute units, which allows it to carry out complicated tasks.

AMD ROCm

An ideal open software stack for developing and transferring LLMs

For  AI and HPC applications, the AMD ROCm software platform offers a solid and open basis. ROCm makes AI-specific libraries, tools, and frameworks available so that  AI developers may easily take advantage of the MI300X GPU’s capabilities. Code created on CUDA can be easily ported to ROCm with little modifications from developers, ensuring efficiency and compatibility.

Leading AI frameworks like PyTorch and TensorFlow are supported by upstream ROCm software, making millions of Hugging Face and other LLMs functional right out of the box. Additionally, it makes it easier to integrate libraries like Hugging Face and frameworks like PyTorch with AMD GPUs, making the integration of LLMs on the MI300X simple. With AMD Instinct GPUs, this integration guarantees developers to optimise application performance and give optimal performance for LLM inference.

AMD ROCm GPUs

Making a tangible difference

To improve LLM inference models and address real-world issues, AMD works in an open ecosystem with industry partners like Microsoft, Hugging Face, and the OpenAI Triton team. AMD Instinct GPUs, such as the MI300X, are used by the Microsoft Azure cloud platform to improve enterprise AI services. Another noteworthy MI300X implementation by Microsoft and OpenAI is ChatGPT-4, which demonstrates how well AMD GPUs can manage demanding  AI workloads.

Hugging Face collaborates with the OpenAI Triton team to integrate cutting-edge tools and frameworks, while utilising AMD technology to optimise models and accelerate inference times.

In conclusion, because the AMD Instinct MI300X accelerator can handle issues with availability, speed, and cost, it’s a great option for implementing big language models. By offering a dependable, effective substitute and a robust ROCm ecosystem, AMD supports companies in maintaining stable  AI operations and achieving peak performance.

ROCm: What is it?

ROCm is an open-source stack for graphics processing unit (GPU) compute that is mainly made up of open-source software. GPU programming is made possible by ROCm, which is a set of drivers, development tools, and APIs that range from low-level kernel to end-user programs.

Heterogeneous-computing Interface for Portability (HIP) powers ROCm, which comes with all the required libraries, debuggers, and compilers for open source applications. It also supports programming models like OpenMP and OpenCL. It is completely integrated with PyTorch and TensorFlow, two machine learning (ML) frameworks.

AMD ROCm 6.1

With the AMD ROCm 6 open-source software platform, AMD hopes to maintain its commitment to open-source and device-independent solutions while creating an environment that maximises the performance and potential of AMD Instinct accelerators. Consider ROCm 6 as the link that will allow your most ambitious AI concepts to be implemented successfully. In the current market, it gives developers the opportunity to create at their own speed, testing and deploying applications across a wide range of GPU architectures, and it delivers outstanding interoperability with key industry frameworks.

The most recent platform upgrade from AMD, ROCm 6.1, adds a host of new features for both academics and developers. In order to stay up with the quick developments in AI frameworks, AMD will examine how ROCm 6.1 builds on the fundamental advantages of ROCm 6 by supporting the most recent AMD Instinct and Radeon GPUs, boosting optimisations across a wide range of computational domains, and extending ecosystem support. The goal of ROCm 6.1’s new features and updates is to enhance application performance and reliability so that AI and HPC developers can push the boundaries of what is feasible.

Presenting rocDecode, a video processing tool Thanks to the new ROCm library, AMD GPUs now have high-performance video decoding capabilities directly on the GPU thanks to the Video Core Next (VCN) specialised media engines. These hardware-based decoders are effective at handling video streams.

By enabling direct decoding of compressed video into visual memory, rocDecode reduces the amount of data transferred via the PCIe bus and gets rid of typical bottlenecks in video processing. With real-time applications like video scaling, colour conversion, and augmentation which are critical for advanced analytics, inferencing, and machine learning training this feature enables rapid post-processing with the ROCm HIP framework.

The efficiency and scalability of video decoding activities are maximised with rocDecode. The API fully utilises all of the VCNs on a GPU device by permitting the development of numerous decoder instances that can run concurrently. The capacity to process in parallel ensures that even large-volume video streams can be simultaneously decoded and processed. To put it succinctly, rocDecode strengthens the video processing pipeline, providing power efficiency and performance increases that are necessary for contemporary AI and HPC applications.

MIGraphX adds Flash Attention and PyTorch backend The AMD graph inference engine is called MIGraphX. MIGraphX is a command-line programme called migraphx-driver and is available through C++ and Python APIs. Its purpose is to speed up deep learning neural networks. Because of this flexibility, developers may incorporate sophisticated model inference features into their applications with ease.

With support for Flash Attention, which increases the memory efficiency of well-known models like BERT, GPT, and Stable Diffusion, ROCm 6.1 enhances performance for transformer-based models and contributes to the faster, more power-efficient processing of complicated neural networks.

A new Torch-MIGraphX library is also included in ROCm 6.1, allowing the PyTorch workflows to directly incorporate MIGraphX capabilities. It defines an immediate-use “migraphx” backend for the torch.compile API. A variety of data types, such as FP32, FP16, and INT8, are supported by the Torch-MIGraphX library to meet various computing requirements.

Better MIOpen Library performance AMD’s open-source MIOpen deep learning primitives library is made especially to improve GPU performance. It has a full suite of tools to maximise GPU launch overheads and memory bandwidth using cutting-edge methods like fusion and auto-tuning infrastructure. This infrastructure adapts algorithms to optimise convolutions for different filter and input sizes, and it handles a wide range of issue setups efficiently.

The goal of MIOpen’s most recent upgrades is to improve performance, especially for convolutions and inference. ROCm 6.1 features Find 2.0 fusion plans, which are intended to maximise system resource utilisation and enhance the library’s capacity to carry out inference jobs more effectively. The convolution kernels for the Number of samples, Height, Width, and Channels (NHWC) format have been enhanced by AMD. The new heuristics especially optimise efficiency for this format, allowing better handling and processing of convolution operations across multiple applications. NHWC prioritises the height and width dimensions, followed by channels.

New Composable Kernel Library Architecture Support The Composable Kernel (CK) library now has expanded architecture support thanks to ROCm 6.1, providing extremely effective capabilities on a larger variety of AMD GPUs. The addition of stochastic rounding to the FP8 rounding mechanism is a major update in this version. By simulating more realistic data behaviour, this rounding technique improves model convergence and provides a more accurate and dependable means of handling data in machine learning models.

Enlarged hipSparse Computations using SPARSELt To speed up deep learning tasks, ROCm 6.1 adds extensions to hipSPARSELt that allow structured sparsity matrices. Support for configurations in which ‘B’ denotes the sparse matrix and ‘A’ the dense matrix in Sparse Matrix-Matrix Multiplication (SPMM) is noteworthy in this release. The library’s capabilities were previously restricted to multiplications where the sparse matrix was represented by the letter “A” and the dense matrix by the letter “B.” This addition expands the library’s capabilities. The performance and versatility of SPMM operations are improved by support for various matrix configurations, which further optimises deep learning computations.

Higher-Level Tensor Functions using hipTensor The AMD-specific C++ library hipTensor uses the Composable Kernel Library’s primitives to speed up tensor operations. hipTensor was created by AMD to take advantage of general-purpose kernel languages like HIP C++. In cases where complicated tensor computations are needed, hipTensor optimises the way tensor primitives are executed.

HipTensor’s most recent version adds support for 4D tensor contraction and permutation. A critical operation in many tensor-based computations, permutations on 4D tensors can now be efficiently carried out by users with ROCm 6.1. 4D contractions for F16, BF16, and Complex F32/F64 data formats are now supported by the library. With this additional functionality, hipTensor can now optimise a wider range of operations, enabling more complicated and varied manipulations of tensor data many of which are necessary for sophisticated computing activities like training neural networks and running complex simulations.

AMD wants to provide you with the newest in high-performance computing through the ROCm platform. Every upgrade in ROCm 6.1 has been created to increase productivity, optimise processes, and assist you in reaching your objectives more quickly by offering useful, strong tools that unleash your creative potential.

https://rocm.github.io/install.html

So, I have recently discovered this piece of software that could replace CUDA and Nvidia in the GPU Machine Learning framework space. As of now it only runs on Ubuntu and Fedora. So, I will have to install a second OS on my machine.