The Crowd Hides the Thread
The Crowd Hides the Thread
A GPU does not run one thread at a time, and it does not really run threads independently either. It runs a warp — thirty-two threads bound together, marching through the exact same instruction in lockstep, each one differing only in which data it touches. The illusion offered to the programmer is that of ordinary parallel threads, each with its own program counter, its own branches, its own private path through the code. The hardware underneath honors none of that separateness; it issues one instruction to the whole warp at once and lets an execution mask decide, silently, which lanes actually get to keep their result.
That mask is where a branch goes to hide. When threads within a warp disagree about which way an if-statement should go, the hardware cannot send half of them down one path and half down the other simultaneously — there is only one instruction stream per warp. So it takes both paths, one after another, masking off the threads that don't belong to the branch currently executing, and reconverges once both sides are exhausted. Divergence doesn't corrupt correctness; it just spends time serially on work that looked parallel from a distance.
What actually hides the cost of this — and of long-latency memory fetches — is not any one warp's cleverness but the sheer number of warps a streaming multiprocessor keeps in flight at once. The moment one warp stalls waiting on DRAM, the scheduler switches, at zero overhead, to another warp that's ready to issue. Latency isn't eliminated; it's buried under a crowd large enough that something is always ready to run while something else waits.
The thread was always a convenient fiction, offered so the programmer could write ordinary-looking code. What actually executes is a regiment moving as one body, and its speed comes from how well the crowd can absorb the moments any single member falls out of step.