Radeon RX 7900 XTX

An Nvidia GPU is the most common answer, but personally i've done all my LLM use locally using mainly Mistral Small 3.1/3.2-based models and llama.cpp with an AMD RX 7900 XTX GPU. [...] Or, more often, i'd fix bugs in the LLM's code :-P. FWIW GPU aside, my PC isn't particularly new - it is a 5-6 year old PC that was the cheapest money could buy originally and became "decent" at the time i upgraded it 5 years ago and i only added the GPU around Christmas as prices were dropping since AMD was about to release the new GPUs. [...]

I do use 2 amd gpus and I get high 40 for generation, 500 for pp and low 20/100 by the end of the context of 256k. llama-server --host 0.0.0.0 --port 8089 -m Qwen3.8-27B-UD-Q8u.gguf --spec-type draft-mtp,ngram-mod --spec-draft-n-max 3 --spec-draft-n-min 1 if you have an igpu and want to exclude or just use some gpus you can use --device Vulkan3,Vulkan2,Vulkan1 in my case vulkan because of amd, you can see your devices with llama-server2 --list-devices Available devices: Vulkan0: AMD Radeon Graphics (RADV RAPHAELMENDOCINO) (33515 MiB, 29349 MiB free) Vulkan1: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 4911 MiB free) Vulkan2: AMD Radeon RX 7900 XTX (RADV NAVI31) (24560 MiB, 7681 MiB free)

For some anecdata, I've set up Qwen3.5 on a RX 7900XTX last weekend. [...]

[...] I have a machine in my home with a single RX 7900 XTX in it. [...]

If you don't mind going through the eldritchian horror that is building ROCm from source[0], QwenQwen3-30B-A3B-Q6K (6bit quantization of the LLM mentioned in the article which in practice shouldn't be much different) works decently fast on a RX 7900 XTX using koboldcpp and llama.cpp. [...] If you're on Debian AFAIK AMD is paying someone to experience the pain in your place, so that is an option if you're building something from scratch, but my openSUSE Tumbleweed installation predates the existence of llama.cpp by a few years and i'm not subjecting myself to the horror that is Python projects (mis)managed by AI developers[1] :-P. [...]

For those wondering about AMD performance, I did some very brief testing. I have an Radeon RX 7900 XTX graphics card and a Ryzen 7800X3D running Fedora 41 with Proton Experimental on a 1920x1200 display. I booted into the Ravenholm level and immediately got mid-50 FPS with no obvious stuttering. The framerate jumped over 70 once I got inside the first couple buildings. [...]

[...] "Almost" because it actually isn't frozen, if there is any output/animation/etc going on it plays fine, but once i try to move the mouse it freezes and everything updates at a single frame per couple of seconds - sometimes freezing completely. [...] AFAICT (from the behavior) something isn't properly saved/restored and communicating with the GPU (the mouse cursor is a hardware cursor thus needs to send commands to the GPU to update its position) causes some sort of issue. [...] Weirdly enough this only happens after i replaced my RX 5700 XT with a RX 7900 XTX so it might be something GPU (or GPU arch) specific. I've been considering plugging my laptop and see if there is something i can figure out (GPU aside the PC is usable, but i guess if this a kernel side thing i'd need a second computer connected to it to debug it), but as this isn't something i've tried before (though i know someone who has and said it isn't anything special) my annoyance still hasn't gone over the "i need to get to the bottom of this" threshold :-P. [...]

While Vulkan can be a good fallback, for LLM inference at least, the performance difference is not as insignificant as you believe. [...] Note: if you're building llama.cpp, all you have to do is swap GGMLHIPBLAS=1 and GGMLVULKAN=1 so the extra effort is just installing ROCm? (vs the Vulkan devtools) ROCm: CUDAVISIBLEDEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q40.gguf ggmlcudainit: GGMLCUDAFORCEMMQ: no ggmlcudainit: GGMLCUDAFORCECUBLAS: no ggmlcudainit: found 1 ROCm devices: Device 0: Radeon RX 7900 XTX, compute capability 11.0, VMM: no | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | | llama 7B Q40 | 3.56 GiB | 6.74 B | ROCm | 99 | pp512 | 3258.67 ± 29.23 | | llama 7B Q40 | 3.56 GiB | 6.74 B | ROCm | 99 | tg128 | 103.31 ± 0.03 | build: 31ac5834 (3818) Vulkan: GGMLVKVISIBLEDEVICES=1 ./llama-bench -m /models/gguf/llama-2-7b.Q40.gguf | model | size | params | backend | ngl | test | t/s | | ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: | ggmlvulkan: Found 1 Vulkan devices: Vulkan0: Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | llama 7B Q40 | 3.56 GiB | 6.74 B | Vulkan | 99 | pp512 | 1077.49 ± 2.00 | | llama 7B Q40 | 3.56 GiB | 6.74 B | Vulkan | 99 | tg128 | 71.83 ± 0.06 | build: 31ac5834 (3818) EDIT: HN should really support markdown...

[...] Building a PC is - in many ways - not that different than it was in 2005. [...] So I spent a lot of time researching and picking out parts, and then building the system. [...] I went with an MSI MEG x670e "Godlike" motherboard, AMD Ryzen 9 7950X CPU, MSI Gaming Radeon RX 7900 XTX GPU, and 64GB of DDR5 RAM. [...]

I bought replaylee but it ran like total dogwater on my pc :/ The demo worked fine though. Had to refund it. Here's my specs: Intel core i5-10600k 4.1GHz 64GB 2666MHz memory AMD Radeon RX 7900 XTX 24GB Game was installed on Sata SSD Running at 4k

[...] I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. [...]

I only ask because I've been running local models (using Ollama) on my RX 7900 XTX for the last year and a half or so and haven't had a single problem that was ROCm specific that I can think of. It's probably using the Vulkan backend, that is pretty stable and performance is good.

But for how long? The subsidized phase is probably short, and then what? I run Qwen 3.5 27 Dense om my old AMD RX7900XTX at about 45 t/s and barely use my Claude Code subscription anymore.

I squeezed it into 24 GiB VRAM (since I have RX7900XTX): -- Q5KM Unsloth quantization on Linux llama.cpp -- context 81k, flash attention on, 8-bit K/V caches -- pp 625 t/s, tg 30 t/s

I run this model on my AMD RX7900XTX with 24GB VRAM with up to 4 concurrent chats and 512K context window in total. It is very fast (100 t/s) and feels instant and very capable, and I have used Claude Code less and less these days.

It will, I’m doing DP to HDMI 4:4:4 4K@120Hz (and expecting HDR in the near future) from an RX 7900XTX to an LG C3 on Linux. I’m using the Club3D active adapter, which is the only one I found in reviews to reliably work. And it does, 0 problems whatsoever.

I bought an RX 7900 XTX with 24GB, and it’s everything I expected of it. [...] Cheap too, compared to a lot of what I’m seeing.

[...] Using an RX 7900 XTX card, FWIW.

I use an RX 7900 XTX myself and that's the only thing I have direct experience with. [...]

I was able to install (AMD provided) ROCm and Ollama on Ubuntu 22.04.5 with an RX 7900 XTX with no real problems to speak of, and I can execute LLMs using Ollama on ROCm just fine. Take that FWIW.

[...] I'm really curious how he wrote that program, as I have a Rx 7900 XTX and would love to use it for this competition haha

Only the RX 7900 XTX has 24 GB of VRAM at its price point. If I went with an NVIDIA card, I would either have to spend over 50% more on the card, or use much worse models to fit on their 16 GB cards.

There is even less reason to own a PS5, given that Sony doesn't see the PC as being competition given how expensive good value GPUs are these days. I own like two gaming handhelds that can connect to a Radeon RX 7900 XTX, which will be more powerful than a PS5 Pro, except for ease of use […]

Even 200B for 3k USD is really good, 128 GB of memory for that price is surprising! Right now using a two RTX 2080 setup for a pilot project, where it runs ollama and qwen2.5-coder (14B quantized version), a more serious step up from there on a budget would realistically be an RX 7900 XTX (24 GB, though I've had issues with setting up ROCm) for 1k EUR or RTX A5000 (24 GB) for 2.5k EUR in the current market. Honestly, I don't even need the best performance for what I'm trying to do, even an Arc A770 (16 GB) for 350 EUR would be enough to iterate, except that it's not actually supported by ollama and lots of stuff out there: (I know ollama isn't the only solution, but it sucks when the tools you like aren't available)

It seems to use an old, 2 year old version of ROCm (5.4.2) which I'm doubtful would support my RX 7900 XTX. I personally found it easiest to just use the latest rocm/pytorch image and run what I need from there

Radeon RX 7900 XTX is RDNA3 but I wonder if llama.cpp is using the Vulkan matrix instructions wmma and mfma. I have not noticed any remarkable differences between Vulkan and ROCm when using IREE but it's not a turnkey solution yet[1]. [1] <