Self-hosting your AIs: Hardware and Inference Optimization
French version available on LinuxFr.org Introduction Comrades, digital proletarians, penguins—unite! Against the rent-seeking of proprietary AI, let us arm our GPUs! Let’s seize the means of intelligence production! Alright, soapbox aside, let’s dive straight into the meat of the subject. This article is essentially a deep dive into my previous article on self-hosting AI. To get the most out of this, you should be familiar with: Linux in general Docker The PCIe bus Some basic concepts regarding self-hosted LLMs (otherwise, this is going to sting like hell). In my previous article, I suggested a few hardware solutions. Since then, I’ve gathered more data, played around much more with my toys, and—as with almost everything in the current AI world—things have moved surprisingly fast. I’ll also dive deeper into the two hardware setups I’ve tested. Each has its own pitfalls that are well worth mentioning. ...