The Memory Answers Before the CPU Asks
The Memory Answers Before the CPU Asks
Pin the pages first. Tell the kernel: these bytes do not move, will not be swapped, reclaimed, or relocated — they are nailed to a physical address and registered with the card, given a key that means trust. This is the precondition for the trick, the vow that makes everything after it possible. Ordinary memory drifts, gets paged out from under you, and so ordinary networking must always ask the kernel's permission to touch it. Pinned memory asks no one. It sits still, exposed, waiting for a stranger's hand.
The queue pair is the whole architecture in miniature: a send queue, a receive queue, two machines each holding half a conversation they never speak aloud. Post a work request — read this remote address, write that one — and the verbs interface hands it to silicon, not software. From here the NIC is sovereign. It walks the memory registration tables itself, generates the packets itself, handles retransmission itself, and places the payload directly into the pinned region of a machine that has, at this moment, no idea anything is happening to it. No socket accepted this. No interrupt fired for each frame. No copy shuffled data from kernel buffer to user buffer to another kernel buffer, that old relay race of redundant custody. The remote CPU is simply not in the room.
This is the part that unsettles people the first time they understand it: one machine's memory becomes, operationally, an extension of another's, reachable by RDMA read or write as if by pointer arithmetic across a wire. InfiniBand built this world from silicon up, lossless fabric and credit-based flow control assuming the privilege. RoCEv2 smuggles the same verbs semantics into ordinary Ethernet, wrapping RDMA payloads in UDP so the old internet's plumbing will carry a protocol that owes it nothing philosophically. Either way the CPU on the receiving end is not copying, not context-switching, not scheduling a softirq, not waking a process. It is elsewhere, doing other work, innocent.
Only later does it check the completion queue — a small, almost offhand gesture, glancing at a mailbox to find a letter already delivered, already read into place, already true. That is the whole inversion: the network stack, sockets, buffers, the entire liturgy of interrupts-per-packet — bypassed wholesale, not optimized but evaded, because the fastest path through a bottleneck is to never enter the room where the bottleneck lives. The data arrives the way memory itself arrives to a running program: already there, already yours, asked for by no one in particular, known only in retrospect.
Seed: RDMA (Remote Direct Memory Access) & kernel-bypass networking — InfiniBand, RoCEv2, verbs interfaces, queue pairs, pinned memory.