Ask HN: I have $5k to spend on a local AI machine, what should I get?
I would like to run local models as large and as fast as possible for around $5,000 USD. Is an Apple machine the best choice with their shared memory or is there a particular GPU that would be more cost effective? Thanks!
I guess I would buy nvidia digits https://www.nvidia.com/en-us/project-digits/
This is exactly what I'd recommend too, it's cheaper than buying GPUs individually and Digits has way more VRAM.
Thank you! This looks awesome!
Unpopular but highly promising way to go if training is on your mind- 4x 7900 xtx cards and some nuts and bolts to feed them could be a price per GPU memory high point. There are folks using ROCm with that to put up some interesting numbers in terms of wall clock and power required per training run.
This Reddit comment mentioned this site with used servers:
https://pcserverandparts.com/ https://www.reddit.com/r/LocalLLaMA/comments/1i8rujw/comment...
https://www.amazon.com/Yassk-Fortune-Telling-Floating-Answer...
AI in the palm of your hand! Best deal evarrr!
Be more specific - AI is a very broad field.
nVidia GPUs have the best inference speed (particularly around SDXL, Hunyuan, Flux, etc), but unless you're buying several used 3090s SLI style, you're going to have to split larger LLM GGUFs across main memory and GPU. I'm excluding the RTX 5090 since two of them (plus tax) would basically blow your budget out the water.
With Apple I think you can get up to 192GB of shared memory allowing for very large LLMs.
Another thing is your experience. Unless you want to shell out even more money, you'll likely have to build the PC. It's not hard but it's definitely more work than just grabbing a Mac Studio from the nearest Apple Store.
I just got a Mac Mini with maximum specs (can't believe how small the box it came in was!) and that's not a bad choice. As you say it has the advantage of handling large models. I think the 5090 will outperform it in terms of FLOPS but it only comes with 32MB compared to the 64MB you can get on an M4 mini. The 5090 itself will be $2000 (if you can get it at that price) compared to the $2500 max mini M4. You'll probably spend at least $1k for the rest of the PC worthy of the 5090 card.
[dead]