If you've worked with PyTorch, you've probably written something like model.cuda() and watched your model start running on the GPU. But what actually happened after that? How does a line of Python eventually turn into thousands of threads doing computation on a GPU? What exactly is CUDA? Where does ROCm fit in? And if you're someone who wants to go beyond simply using GPUs and actually understand them, where do you even start?I started asking myself these questions recently, and that slowly pulled me into the world of GPU programming. At first, I thought CUDA was basically just a way of writing code for NVIDIA GPUs. But it's much bigger than that. CUDA is NVIDIA's platform and programming model for general-purpose GPU computing. It gives you the tools and abstractions needed to write GPU kernels, manage memory, launch work, and interact with the hardware. Once you start writing CUDA, you stop thinking of a program as something that simply runs from top to bottom. Instead, you start thinking about how thousands of pieces of work can execute in parallel.
That leads to one of the first important things to understand about GPUs: a GPU isn't simply a faster CPU. CPUs are built around a relatively small number of powerful cores, while GPUs are designed to keep huge amounts of parallel work running at the same time. In NVIDIA's programming model, threads are organized into blocks, and threads execute in groups called warps. Those warps are scheduled onto the GPU's Streaming Multiprocessors, or SMs. You don't need to understand every hardware detail immediately, but once you start caring about performance, these concepts become difficult to ignore.
Then there's memory, which is probably where GPU programming really starts getting interesting. GPU memory isn't one giant pool where every access costs the same. You have registers, shared memory, caches, and global memory, all with different sizes and access characteristics. This means that an operation can be doing relatively little computation but still be slow because it keeps moving data back and forth between global memory and the compute units.
That changed the way I started looking at GPU optimization. Instead of only asking, "How can I make the GPU calculate this faster?", I started asking, "Why am I moving this data in the first place?" If you can load something once, reuse it several times, and avoid unnecessary trips to global memory, you can sometimes get a much larger performance improvement than simply trying to squeeze more computation out of the GPU.
So where does AMD fit into all of this? If CUDA is NVIDIA's GPU computing ecosystem, ROCm is AMD's corresponding open software stack for GPU computing. ROCm provides the compilers, runtimes, libraries, profilers, debugging tools, and other components needed to develop software for AMD GPUs. One of the main programming technologies within ROCm is HIP, which provides a C++ programming model that is deliberately similar to CUDA.
This is useful because the two ecosystems aren't completely unrelated worlds. CUDA code and HIP code share many of the same fundamental ideas: kernels, threads, blocks, memory management, synchronization, and parallel execution. AMD also provides tools such as HIPIFY that can help translate CUDA source code into HIP. That doesn't mean you can blindly convert any CUDA program and expect identical performance, because hardware-specific optimizations still matter, but it makes the ecosystems much closer than they might initially appear.And this is why I think it's useful to understand CUDA and ROCm as part of a larger GPU software stack, rather than treating them as isolated technologies. At the top you might have your ML application and PyTorch. Underneath that are GPU libraries and kernels. Those eventually interact with CUDA on NVIDIA hardware or ROCm/HIP on AMD hardware, which then interacts with the compiler and ultimately the GPU architecture itself.
For someone coming from ML, this is where things get interesting. Most of the time, you don't need to think about any of this. You write torch.matmul(), torch.softmax(), or torch.layer_norm(), and highly optimized implementations take care of the underlying work. That's exactly what frameworks are supposed to do. But eventually you might encounter an operation that's a bottleneck, a sequence of operations that could be fused, or a workload where the default implementation isn't ideal. That's when understanding what happens underneath the framework becomes extremely useful.
It also explains why technologies like Triton are becoming interesting. Triton gives you a higher-level way of writing custom GPU kernels without forcing you to manually deal with every individual thread in the same way CUDA does. You can think more in terms of blocks or program instances, while the compiler handles much of the mapping to the GPU. For me, this became a much more approachable way of experimenting with GPU kernels while still having to think about things like memory access, tiling, fusion, and data reuse.
And this is probably the biggest recommendation I'd give: don't just learn CUDA or ROCm by reading about them. Build something. I'm planning to approach my own learning through a GitHub repository where each kernel becomes an experiment. Start with a naive implementation, benchmark it, identify the bottleneck, optimize it, benchmark it again, and document what changed. Over time, that becomes much more valuable than simply having "CUDA" listed on a resume because you have actual evidence that you understand what the GPU is doing.
And I think that's ultimately why CUDA and ROCm are worth learning, especially if you're interested in ML systems. You start with something as simple as model.cuda(), but underneath that is an entire stack connecting Python frameworks, libraries, compilers, kernels, memory systems, and GPU hardware. Once you start peeling those layers back, things like FlashAttention, custom kernels, inference optimization, ML compilers, and GPU performance stop looking like completely separate topics.
You don't need to become a GPU expert before starting. Pick one kernel. Understand what it does. Write it. Break it. Benchmark it. Make it faster. Then move to the next one.