GPU Architecture Fundamentals

To write fast code, you must understand the hardware. A GPU is fundamentally different from a CPU, designed for massive parallelism.

Streaming Multiprocessors (SMs)

A modern GPU consists of dozens of Streaming Multiprocessors (SMs). Each SM contains multiple CUDA cores, Tensor Cores, and shared memory.

"A GPU hides memory latency through massive thread-level parallelism."

The Execution Model: SIMT

GPUs use the Single Instruction, Multiple Threads (SIMT) architecture. Threads are grouped into "warps" (typically 32 threads), which execute the same instruction simultaneously.

Let's represent the theoretical memory bandwidth $B$ as: $B = \text{Memory Clock Rate} \times \text{Bus Width} \times 2$ (Assuming Double Data Rate memory).

Memory Hierarchy

Understanding memory is the key to GPU performance.

1. Global Memory: Largest, but slowest (HBM).
2. Shared Memory: On-chip, very fast, shared within an SM.
3. Registers: Fastest, private to each thread.

Comparison of Memory Types

Memory Type	Latency	Scope	Size
Registers	~1 cycle	Thread	KB
Shared Memory	~30 cycles	Block	KB/MB
Global Memory	~400 cycles	Grid	GB

In the next chapter, we will write our first optimized CUDA kernel utilizing these memory structures.

GPU Architecture Fundamentals

To write fast code, you must understand the hardware. A GPU is fundamentally different from a CPU, designed for massive parallelism.

Streaming Multiprocessors (SMs)

A modern GPU consists of dozens of Streaming Multiprocessors (SMs). Each SM contains multiple CUDA cores, Tensor Cores, and shared memory.

"A GPU hides memory latency through massive thread-level parallelism."

The Execution Model: SIMT

GPUs use the Single Instruction, Multiple Threads (SIMT) architecture. Threads are grouped into "warps" (typically 32 threads), which execute the same instruction simultaneously.

Let's represent the theoretical memory bandwidth $B$ as: $B = \text{Memory Clock Rate} \times \text{Bus Width} \times 2$ (Assuming Double Data Rate memory).

Memory Hierarchy

Understanding memory is the key to GPU performance.

1. Global Memory: Largest, but slowest (HBM).
2. Shared Memory: On-chip, very fast, shared within an SM.
3. Registers: Fastest, private to each thread.

Comparison of Memory Types

Memory Type	Latency	Scope	Size
Registers	~1 cycle	Thread	KB
Shared Memory	~30 cycles	Block	KB/MB
Global Memory	~400 cycles	Grid	GB

In the next chapter, we will write our first optimized CUDA kernel utilizing these memory structures.

GPU Architecture Fundamentals

To write fast code, you must understand the hardware. A GPU is fundamentally different from a CPU, designed for massive parallelism.

Streaming Multiprocessors (SMs)

A modern GPU consists of dozens of Streaming Multiprocessors (SMs). Each SM contains multiple CUDA cores, Tensor Cores, and shared memory.

"A GPU hides memory latency through massive thread-level parallelism."

The Execution Model: SIMT

GPUs use the Single Instruction, Multiple Threads (SIMT) architecture. Threads are grouped into "warps" (typically 32 threads), which execute the same instruction simultaneously.

Let's represent the theoretical memory bandwidth $B$ as: $B = \text{Memory Clock Rate} \times \text{Bus Width} \times 2$ (Assuming Double Data Rate memory).

Memory Hierarchy

Understanding memory is the key to GPU performance.

1. Global Memory: Largest, but slowest (HBM).
2. Shared Memory: On-chip, very fast, shared within an SM.
3. Registers: Fastest, private to each thread.

Comparison of Memory Types

Memory Type	Latency	Scope	Size
Registers	~1 cycle	Thread	KB
Shared Memory	~30 cycles	Block	KB/MB
Global Memory	~400 cycles	Grid	GB

In the next chapter, we will write our first optimized CUDA kernel utilizing these memory structures.