I don’t have an answer for you, partly because there isn’t enough information about your aims. However, you can probably work this out yourself, compare prices for different hardware. You’d need some of that missing information to run the numbers, though.

I would imagine that an important input here is your expected usage.

If you just want to set up a box to run a chatbot occasionally and you get maybe 1% utilization of the thing, the costs are different from if you intend to have the thing doing batch-processing jobs 24/7. The GPU is probably the dominant energy consumer in the thing, so if it’s running 24/7, the compute efficiency of the GPU in terms of energy is going to be a lot more important.

If you have that usage figure, you can estimate the electricity consumption of your GPU.

A second factor here, especially if you want interactive use, is what level of performance is acceptable to you. That may, depending upon your budget and use, be the dominant concern. You’ve got a baseline to work with.

If you have those figures – how much performance you want, and what your usage rate is – you can probably estimate and compare various hardware possibilities.

I’d throw a couple of thoughts out there.

First, if what you want is sustained, 24/7 compute, you probably can look at what’s in existing, commercial data centers as a starting point, since people will have similar constraints. If what you care about is much less frequent, it may look different.

Second, if you intend to use this for intermittent LLM use and have the budget and interest in playing games, you may want to make a game-oriented machine. Having a beefy GPU is useful both for running LLMs and playing games. That may differ radically from a build intended just to run LLMs. If you already have a desktop, just sticking a more-powerful GPU in may be the “best” route.

Third, if performance is paramount, depending upon your application, it may be able to make use of multiple GPUs.

Fourth, what applications you want to run may (it sounds like you may have decided on Nvidia already) affect what hardware is acceptable. First, AMD/Nvidia, but also, many applications have minimum VRAM requirements – the size of the model imposes constraints. Have a GPU without enough VRAM to run what you want to run, and you can’t run the model at all.

Fifth, if you have not already, you may want to consider the possibility of not self-hosting at all, if you expect your use to be particularly intermittent and you have high hardware requirements. Something like vast.ai lets you rent hardware with beefy compute cards, which can be cheaper if your demands are intermittent, because the costs are spread across multiple users. If your use is to run a very occasional chatbot and you care a lot about performance and want to run very large models, for example, you could use a system with an H100, for example, for about $3/hour. An H100 costs about $30k and has 80GB of VRAM. If you want to run a chatbot a weekend a month for fun and you want to run a model that requires 80GB – an extreme case – that’s going to be a lot more economical than buying the same hardware yourself.

Sixth, electricity costs where you are are going to be a factor. And if this system is going to be indoors and you live somewhere warm, you can multiply the cost for increased air conditioning load.

You can use OpenCL instead of ROCm for GPU offloading. In my tests with llama.cpp that improved performance massively.

Definitely do benchmarks for how many layers you can offload to the GPU. You’ll see when it’s too many, as performance will crater.

By launching llama.cpp as a server you’ll actually be able to continue to use openwebui as you currently have.

Not sure about the setup, but I believe 7600XT would be the best buy, it has 16GB VRAM and it’s supported now

I have my gaming pc running as ollama host when i need it (RX 6700XT with rocm doing the heavy lifting). PC idles at ~50W and draws up to 200W when generating an answer. It is plenty fast though.

My mini pc home server is running openwebui with access to this “ollama instance” but also OpenAIs api when i just need a quick answer and therefor don’t turn on my pc.

You could try llamaCPP, I think its configured to run better on cpu’s

deleted by creator

Chinese mining motherboard with any xeon cpu (because of abundance of PCI lanes in cpu) and 12x16gb nvidia p100 and nvlink bridges (p40 gives 24gb of vram but doesn’t have nvlink capabilities and slower gddr5 memory, p100 have hbm memory on the contrary which is good for llm) and 8x64gb ddr4 ecc ram, if you buy gpus from ebay and other components such as motherboard, CPU, ram, and ssd from AliExpress it comes quite cheap