Show HN: Llama 3.1 70B on a single RTX 3090 via NVMe-to-GPU bypassing the CPU
via news.ycombinator.com
Short excerpt below. Read at the original source.
Hi everyone, I’m kinda involved in some retrogaming and with some experiments I ran into the following question: “It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?” This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the […]