I wanted to run LLMs locally. Not through someone else's API, not renting GPU hours. Actually local, on hardware I own, with no per-token bill. The question was whether I could do it without spending more than $500.
Turns out the answer is: sort of. It depends heavily on what "run" means to you and how patient you are.
The hard constraint
$500 is the ceiling. No upgrades later, no "just a little more for better RAM." One purchase, done. That rules out anything from NVIDIA's consumer GPU lineup new. A 4060 alone costs $300, and you still need the rest of the machine.
Used hardware and bare-bones Mini PCs are where the action is.
Option 1: AMD Mini PC (GMKtec K6)
The GMKtec K6 runs about $456 new. It has a Ryzen processor with Radeon 780M integrated graphics, 16GB of RAM, and a small form factor. No discrete GPU, so you're running everything through the iGPU or CPU.
The 780M is surprisingly capable for its class, but "capable for an iGPU" and "good for LLM inference" are different conversations. ROCm support on consumer AMD hardware is spotty. Most people end up running llama.cpp in CPU-only mode, which works but is slow. We're talking 2-5 tokens per second for a 7B model quantized to Q4, depending on the context length.
For text-only inference on small models, it's functional. For anything involving image generation or larger models, it's a slog.
Option 2: Used OptiPlex + RTX 3060 12GB
This is the route most homelab people recommend, and for good reason. A refurbished Dell OptiPlex or similar business desktop runs $150-200 on eBay. Add a used RTX 3060 12GB for around $220-250 from a reputable reseller (Jawa, eBay IT recyclers), and you're in the $370-450 range before tax.
12GB of VRAM is the magic number for local inference. It fits a quantized 13B model comfortably, a 7B model with room for a large context window, and it can run Stable Diffusion without breaking a sweat.
The catch: you need to verify the OptiPlex has a PSU that can handle the 3060 (170W TDP) and a PCIe slot that isn't physically blocked. Some SFF models won't work. Tower models usually do.
Comparison
| GMKtec K6 | OptiPlex + RTX 3060 | |
|---|---|---|
| Approx. cost | ~$456 | ~$370-$450 |
| VRAM | Shared (0 dedicated) | 12 GB dedicated |
| 7B model @ Q4 | 2-5 t/s (CPU) | 20-40 t/s (GPU) |
| 13B model @ Q4 | Barely usable | 10-20 t/s |
| Image generation | Not practical | Works well |
| Form factor | Small (0.6L) | Tower (~20L) |
| Setup effort | Low (works out of box) | Medium (GPU install, PSU check) |
| Risk | Low (new, warrantied) | Medium (used components) |
What I'd pick
If you care about inference speed and model size, the OptiPlex + 3060 route wins. 12GB of VRAM changes what you can run. The Mini PC is tempting for its simplicity and size, but CPU-only inference on shared memory is rough. You'll spend more time watching tokens appear than thinking about what they say.
If you only need text inference on small models and value silence, simplicity, and a tiny desk footprint, the K6 works. Just don't expect magic.
Buying advice
For used GPUs, stick to resellers with return policies. Jawa is a solid marketplace for used hardware. They verify condition and handle disputes. eBay IT recyclers (the ones with thousands of reviews and 99%+ ratings) are also reliable. Random Craigslist deals carry more risk than I'd accept for a component I'm depending on daily.
For the OptiPlex, get a tower, not SFF. Verify the PSU wattage (ideally 300W+). Check that the PCIe x16 slot isn't covered by the heatsink shroud. These are common gotchas that aren't obvious from product photos.
And don't forget: you need 16GB of system RAM on the OptiPlex. Some refurbished units come with 8GB. Budget another $30-40 for an upgrade if needed.