#llama32

It is possible to run local LLMs on AMD GPUs by using Ollama. The most recent version of Llama 3.2, which went live on September 25, 2024, is the subject of this tutorial.

Llama 3.2 from Meta is compact and multimodal, featuring 1B, 3B, 11B, and 90B models. Here is a step-by-step installation instruction for Ollama on Linux and Windows operating systems using Radeon GPUs, along with information on running these versions on different AMD hardware combinations.

Supported AMD GPUs

Ollama’s product may be used with both older and current AMD GPU models since it supports a variety of them.

Ollama Installation & Setup Guide.

Linux

Ubuntu 22.04.4.

AMD GPUs using the most recent version of AMD ROCm.

To install ROCm 6.1.3.

Use a single command to install Ollama.

Windows

System prerequisites

Windows 10 or Later.

Installed drivers for AMD GPUs that are supported.

After installation, just launch PowerShell and execute it.

Run, llama, llama3.2.

That’s all; you’re set to speak with your local LLM.

AMD ROCm Supported GPUs

Use ROCm to install Radeon software on Linux

The amdgpu-install script helps you install a cohesive collection of stack components, including the ROCm Software Stack and other Radeon software for Linux components.

Makes the installation of the AMD GPUs stack easier by using command line arguments that let you select the following and by encapsulating the distribution-specific package installation logic.

The AMD GPUs stack’s use case (graphics or workstation) that has to be installed.

Combination of elements (user selection or Pro stack).

Carries out post-install inspections to confirm that the installation went well.

Installs the uninstallation script, which enables you to use a single command to delete the whole AMD GPU stack from the computer.

AMD Radeon GPUs

Ollama supports the following AMD GPUs:

Linux Support

Linux Overrides

Not all AMD GPUs are supported by the AMD ROCm library, which Ollama makes use of. You can sometimes cause the system to attempt using a nearby, comparable LLVM target. For instance, the Radeon RX 5400 is gfx1034 (also referred to as 10.3.4), however ROCm does not yet support this target. gfx1030 is the closest support. With x.y.z syntax, you may utilize the environment variable HSA_OVERRIDE_GFX_VERSION. For instance, you may change the server’s environment variable HSA_OVERRIDE_GFX_VERSION=”10.3.0″ to force the system to operate on the RX 5400.

A future version of ROCm v6 is anticipated to support a greater number of GPU families due to AMD’s ongoing efforts to improve it.

GPU Selection

To restrict Ollama to use a subset of your system’s AMD GPUs, you may set HIP_VISIBLE_DEVICES to a list of GPUs separated by commas. The list of devices having rocminfo is shown. You may use an incorrect GPU ID (such as “-1”) to compel CPU utilization while ignoring the GPUs.

Permission for Containers

SELinux may restrict containers’ access to AMD GPU hardware in various Linux editions. To enable containers to utilize devices, execute sudo setsebool container_use_devices=1 on the host system.

Metal: GPUs made by Apple

Through the Metal API, Ollama facilitates GPU acceleration on Apple devices.

In summary

Ollama’s broad support for AMD GPUs is evidence of how widely available executing LLMs locally is becoming. Users may run models like Llama 3.2 on their own hardware with a variety of choices, ranging from high-end AMD Instinct accelerators to consumer-grade AMD Radeon RX graphics cards. More customization, privacy, and experimentation are possible in AI applications across a range of industries thanks to this adaptable strategy for enabling creative LLMs throughout the extensive AI portfolio.

Dive into the latest advancements in AI technology with Llama 3.2. Explore its multimodal capabilities, combining text and voice for seamless user interactions. Discover how this innovative model can transform industries and enhance user experiences.