GeForce RTX 4090

[...] Docker helps but image build/push overhead adds another 5-10 min to the loop. [...] Worth building. On the infrastructure layer: we built GhostNexus ( to address 1 and 2 — per-second billing, <30s cold starts on RTX 4090 hardware, Python SDK with 3 lines to submit a job. Might be worth using as the GPU backend if you don't want to manage the infra layer yourself. [...]

I work in AI and I'm surrounded by RTX-4090 and H100 servers but for much of the day to day AI training I use my RTX-970 in the desktop on my desk for convenience and it works just fine for most cases. Literally 2 meters from my desk is a 2x RTX-4090 server and many times I just use my 8 year old GPU anyway so you don't need it.

Two whole years later and NVIDIA RTX 4090 remains a device guaranteed to decimate your credit card, and in all likelihood melt your power cables in the process...

[...] (My daily driver for the last decade have been various flavours of linux) High end Windows 10/11 machines (like my Alienware I9 w/ 64GB mem + RTX4090) seem a bit sluggish. [...] I've install debian onto my fairly ancient Lenovo X201 Tablet and it is exceptionally snappy.

[...] I already had an RTX 4090 and I forked ninfer [0] with the obsession to squeeze everything out of this card for this model. [...] Im currently measuring a pareto front in J/tok in order to set power limits of this card without sacrificing too much performance. Since we are talking about full power draw of 480W which is fine during the day (with solar panels) but during night when the sun doesn't shine (even with a battery) I'd like to limit this a little bit. [...]

[...] for the past few days, i ran this model on my rtx 4090 + rtx 3070 and told it to check all the bills, invoices, contracts for me and my small company. [...] out of all the models i've used locally qwen3.8:27b blows everything out of the water. my setup Logical CUDA0 = RTX 4090, logical CUDA1 = RTX 3070 export CUDAVISIBLEDEVICES=0,1 cd /projects/misc/llama.cpp/ exec ./build/bin/llama-server -hf ggml-org/Qwen3.8-27B-GGUF:Q4KM --mmproj /xx/xx/xx/xx/xx/mmproj-Qwen3.8-27B-Q80.gguf --host 0.0.0.0 --port 8080 --jinja --parallel 1 --split-mode layer --tensor-split 6,1 --fit on -fa on -c 98304 -ctk q80 -ctv q80 --image-min-tokens 1024 i load more on the 4090 because it's faster. [...] if i offload more to the 4090 the tk/s goes up, but i stress the card too much and that thing now is worth its weight in gold. [...]

[...] It runs great on a single RTX 4090 at the modest throughput I need. [...]

[...] (RTX4090; shipped to your door from the nvidia website). [...]

The only fine tune I've attempted is a Qwen3.5:0.8b trained on RL data generated using Qwen3.5:122b playing 50-thousand hands of Hearts (the card game) because I wanted a tiny model that was, idealy, an expert Hearts player so I could hopefully get a super-fast local model that I could play hearts against without any per-turn lag. I wanted more than 50k hands but even just 50k took almost 3 days of RL running on a RunPod RTX 4090. Used the RL data to build a LoRa adapter for Qwen3.5:0.8b -- ultimately a failure but mostly because the data we generated wasn't of sufficient quality. [...]

[...] I have an M3 Max MBP at 400 GB/s and also a desktop with a RTX 4090 with 1,008 GB/s While the M3 Max MBP can serve up MoE reasonably fast (60 token/sec)the RTX 4090 is an entirely different experience (170 token/sec). I also do a fair bit of experimentation and am currently running a custom decoder that requires expensive look-ahead, but I'm still able to get a usable 25 token/s on the RTX. [...] If you want to build software to run on H100s, you probably can't justify buying (and running) a single card. [...]

I run Qwen3.6 on RTX4090, and it does amazing job for the most parts. [...] The main idea is: - separate session that burns context on reading code and calling tools (context7, etc) - markdown report "here are relevant patrts of code, docs" "with evidence" to prevent hallucinations - separate session for planning (architect) - (critic <- architect) 1-3 times because small model skip over details - worker <- validator, again, the same reason Qwen3.6 can run for hours looking for a complex bugs in read-only mode, and usually it gets it. [...] Any bugs? [...]

I have 24GB of VRAM (via a RTX 4090) and run Qwen3.6-35b:iq4, so it's importance-aware quantization and isn't nearly as dumb as it sounds like, fitting the 35b into 18 GB so you have some left over. So far I've had no issues, other than it taking a while for things like image gen, which I found out if you're gonna do with any alacrity, just have a cloud model do it. For anything else local, including writing some automation scripts and such, it works great.

TLDR: Following my recent blog post and Hacker News post ( where I ran the desktop app on my M1 Max. Today, I used Edit Mind (my source-available, local-first video indexing tool) to process my YouTube library, 87 videos, 37h of footage in under 24h on a rented RTX 4090 via Docker + CUDA. Available on GitHub: Blog Post:

llama.cpp to get 115 tok/s on RTX 4090 with Qwen3.6-27B. For example in Windows the latest CUDA variant llama-b9678-bin-win-cuda-13.3-x64.zip and Unsloth UD-Q4KXL MTP gguf: llama-server.exe --host 0.0.0.0 --alias "Qwen3.6-27B-MTP" -m "F:\Qwen3.6-27B-UD-Q4KXL-MTP.gguf" -c 75000 -ngl 99 --metrics --temp 0.6 --top-p 0.95 --min-p 0.00 --top-k 20 --presence-penalty 0.0 --no-mmap -t 16 --spec-type draft-mtp --spec-draft-n-max 3 --reasoning on -fa on --parallel 1 -lv 4 Note that this does not use kv cache quants as in my case quants offload to CPU and tanks performance. [...]

I've been using unsloth/gemma-4-31B-it-qat-GGUF daily for various small parsing and programming tasks using opencode and llama-server's front end. The past couple of weeks have made a big difference after google released the QAT variant and llama.cpp got support for MTP which means it is possible to now get 60-80 Tok/s with RTX 4090. The model fits in VRAM comfortably enough to keep it loaded even while browsing and having multiple programs.

The same 24GB VRAM RTX 4090 I bought to play Cyberpunk 2077 with. Works perfectly fine in llama.cpp throwing 70+t/s at me with 128k q8 K/V context when using the IQ4NL quant + MTP at q4 MTP K/V. Also leaving this here because you might find it useful:

[...] So far it supports the following models: - LLMs (any OpenAI compatible API, vLLM, LM Studio, etc.) - image gen + image edit (flux klein) - text to speech (magpie, dia with voice cloning) - speech to text (OpenAI audio transcriptions + riva compatible) - image to textured 3d model (trellis2) - image+text to video (ltx2.3-gguf) - text to music (acestep) currently it is just me and Claude vibing. While using Fable 5 moved all of my local inference services to k3s across 3 RTX 4090 PCs and my DGX Spark, now I can just tell Claude/Hermes/etc. [...] For example here is 90 seconds disco funk track generated by acestep: I was inspired by AI Horde, and wanted to see if I could build something that could support all of the model modalities that I use for generating short-form AI slop content on local hardware. [...] I've been watching the quality of local AI inference making massive improvements in quality and performance, and I want to make it easier for people to try "local AI" even if they don't have a GPU.

[...] "$1600 is too much for a video card" - me a few years ago on not buying an RTX4090 from nvidia's website. [...]

[...] It mirrors pre-built wheels from every upstream source I can find, and for the gaps where nobody has built a wheel, I build them myself on GPU infrastructure. $ pip install easywheels $ easywheels install deepspeed Detected: Python 3.12, CUDA 12.8, RTX 4090 (sm89) Found: deepspeed-0.18.9+cu128-cp312-cp312-linuxx8664.whl Done in 9 seconds. [...] Linux builds are covered heavily. Windows build-out is in progress. [...] I wish I could give this away, but building wheels takes a ton of GPU time and that costs money. Source: Ask me anything about the build pipeline or why GPU packaging is this broken. [...]

wish I had a rtx5090 instead of the rtx4090 so I can run this, need 32GB VRAM I assume, no I use Linux and I do not enjoy MacOS, I wish x86 family has something like M5 Pro though.

[...] I just tried this GGUF with llama.cpp in its UD Q4KXL version on my custom agentic oritened task consisiting of wiki exploration and automatic database building ( ) I noted a nice improvement over QWen3.5 in its ability to discover new creatures in the open ended searching task, but I've not quantified it yet with numbers. [...] Some little difference with QWen3.5 : to avoid crashes due to lack of memory in multimodal I had to pass --no-mmproj-offload to disable the gpu offload to convert the images to tokens otherwise it would crash for high resolutions images. I also used quantized kv store by passing -ctk q80 -ctv q80 and with a ctx-size 150000 it only need 23099 MiB of device memory which means no partial RAM offloading when I use a RTX 4090.

I tried Gemma 4 A4B and was surprised how hart it is to use it for agentic stuff on a RTX 4090 with 24gb of ram. Balancing KV Cache and Context eating VRam super fast.

[...] Three layers: vector store (Qdrant, local), an observer that runs post-session and detects drift between baseline and reality, and a meta agent that assembles a compressed identity window before each conversation. [...] The interesting part: it has an observer that runs after sessions and detects drift. [...] I ran 44 hours of podcast transcripts through it on an RTX 4090 with faster-whisper. [...] It has an observer layer that detects patterns and drift over time, and a human-readable markdown mirror of everything in the vector database — inspect, edit, delete anything.

[...] - Version Comparisons: See exactly what changed (or what was added) between updates. - Auto-Rescans: We trigger a new scan whenever an extension updates. [...] Because I’m running this entire operation on a single RTX 4090 using a local Qwen3-Coder-30B model. I’m doing this locally because: - Privacy: I refuse to send extension code to some cheap no-privacy third-party AI APIs. - Cost: I can’t afford $50k/month in inference fees for 250k+ extensions. [...] - For Developers: A way to get "Verified Trust" signals to drive adoption. [...]

[...] On my RTX 4090, I'm getting results in under a second with 8-9 sampling steps. [...]

[...] Reading the issues on github[1], it does not seem many others have had success in using this effectively: - someone with a 96 Gb VRAM RTX 6000 Pro had cuda oom issues - someone somehow made it work on a RTX 4090 somehow, but RTF processing time was 12... [...]

[...] However, in the man pages of the facility it is clearly explained that the readings are estimated and not measured values and that they shouldn’t be used for making power consumption comparisons. [...] It is unfortunate that said reviewers do not bother to sanity check the numbers with a power meter and thereby perpetuate misleading information on Apple silicon performance/Watt. In a prior investigation, where I did extensive Linear Algebra performance investigations on the M4 Max, I found that it’s CPU performance/Watt in dense matrix multiplication is well below that of Ryzen 9950X, when the computations are carried out on its Neon vector units. Similarly for Apple’s M4 Max GPU vs nVidia last gen RTX 4090.

It's fast (3 seconds on my RTX 4090) It is amazing how far behind Apple Silicon is when it comes to use non- language models. Using the reference code from Z-image on my M1 ultra, it takes 8 seconds per step. Over a minute for the default of 9 steps.

I am using the Q6KL quant and it's running at about 40G of vram with the KV cache. Device 1 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||20.170Gi/23.988Gi] Device 2 [NVIDIA GeForce RTX 4090] MEM[||||||||||||||||||19.945Gi/23.988Gi]

[...] I tried to run it but of course it failed with NVIDIA GeForce RTX 4090 with CUDA capability sm89 is not compatible with the current PyTorch installation. [...] If you want to use the NVIDIA GeForce RTX 4090 GPU with PyTorch, please check the instructions at I was curious, but not curious enough to deal with this crap even if it's rather simple. [...]

Just tested cyberpunk 2077 on my rtx 4090 with the new DLSS 4 with the transformer model. I got a nearly 30fps boost using DLSS 4 over 3. [...]

I have a 9950X3D and an RTX 4090 in a 10L case (FormD T1). Water cooled but it does get a bit loud when I play games, silent for regular usage.

[...] They all want to try and suck us into the cloud and take away full control and local processing, but there's far more opportunity by building small AI systems and tools into the harness itself to make the models more intelligent. [...] Right now my rtx4090 sits there unused for most of the day while I'm paying for inference in the cloud... [...] Even better, building a framework around this concept and having the frontier model dynamically and adaptively generate the ideal execution system for any given t

[...] I have a dual rtx4090 machine, a 128gb m5 max mbp, and a dgx spark. RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic on the dual rtx4090 machine under vLLM absolutely slays at token speed. [...]

M3-Max laptop: 55 token/sec RTX 4090: 190 token/sec I don't have the number around but there is a notable latency for pre-fill on the M3, but once it's running the delay is negligible. The RTX, unsurprisingly, is all around superior performance wise, but: I use that computer for gaming and image gen work so I can't dedicate it as a server, and, especially when it's warmer, the heat generated under heavy loads is noticable.

[...] Suddenly the old version is worthless, how on earth did anyone get work done with that? Same reason people buy the RTX 4090 and 5090 cards - overpriced but they must have the "best". Never mind the diminishing returns trying to max out PC settings (3-4x performance hit for an almost imperceptible increase in graphics, ignoring DLSS) - it's the psychological cost of having to move a slider down a notch. I've been using Google and now DeepSeek v4 and I am having absolutely no problems and it's a fraction of the cost. [...]

Qwen3.6-27B-UD-Q4KXL can run at 45t/s with 131k q8 context on an RTX 4090. [...]

I have a second computer with an RTX 4090 for gaming (running Windows). I also used the new RTX 5090 running Linux to evaluate whether Proton/Wine allow me to run Windows games on linux (yes, it works, but the compatibility and frame rate issues make me stick to native Windows for now).

I've been using Claude Code regularly at work for several months, and I successfully used it for a small personal project (a website) not long ago. [...] I have 32GB of DDR5 (seems inadequate nowadays), an AMD 7800X3D, and an RTX 4090. [...] In terms of actual performance, it was painfully slow compared to the throughput I'm used to from CC, and the tooling didn't feel as good as the CC harness. [...]

I just tried with llama.cpp RTX4090 (24GB) GGUF unsloth quant UDQ4KXL You can probably run them all. [...] I tried G4 26B A4B as a drop-in replacement of Q3.5-35B-A3B for some custom agents and G4 doesn't respect the prompt rules at all. [...] I have not yet tried the smaller models with audio.

I get a slow-but-usable 10tk/s on kimi 2.5 2b-ish quant on a high end gaming slash low end workstation desktop (rtx 4090, 256 gb ram, ryzen 7950). Right now the price of RAM is silly but when I built it it was similar in price to a high end macbook - which is to say it isn’t cheap but it’s available to just about everybody in western countries. [...] My point is, I can get somewhat-useful ai model running at slow-but-usable speed on a random desktop I had lying around since 2024. [...]

You should use my model then, I spent about 30$ in electricity and used my existing RTX4090. It is not very good, but can you compare it with others really? You can use this service via a private API with a VPN, email me your credit card details for access.

[...] iCloud access, shared messages, Cut and paste between other Mac devices..I say this with a Ryzen 9 + rtx4090 super right next to the macs. (A studio and an M1 MBP with a new battery). [...]

[...] I am interesting if I can run it on a 24GB RTX 4090. [...]

Under the Planes section, look for planes that have non-zero "CRTCID". Those are the planes that actually get output to your monitor. Here's what I'm getting on an RTX 4090 / InnoCN 27M2V and Cooler Master Tempest GP27U.

I was just selling my RTX 4090 on Ebay recently and got a ton of bids from Chinese accounts. [...] As the parent comment's YT video describes, these are often just normal Chinese-Americans or international students who do this to make a bit of extra money.

[...] So unless you've got 4x GeForce RTX 4090's in your computer, CPU inference is going to be the best chance you've got at running 8x22b at top fidelity.