The team behind continuous batching says your idle GPUs should be running inference, not sitting dark
via usenix.org
Short excerpt below. Read at the original source.
Every GPU cluster has dead time. Training jobs finish, workloads shift and hardware sits dark while power and cooling costs keep running. For neocloud operators, those empty cycles are lost margin. The obvious workaround is spot GPU markets — renting spare capacity to whoever needs it. But spot instances mean the cloud vendor is still […]