Mostly, most of the time, nvidia GPUs use CUDA to run LLMs. It’s insanely well supported on Linux (considering the general state of nvidia support on Linux) and insanely non-supported on Windows (tensoflow just refuses to use CUDA on Windows)
As for ROCM, it just doesn’t work. Almost nobody cares about ROCM, so most tools don’t support it and most OSes can’t run it