The memory hierarchy
Registers, cache, RAM, and disk, from fastest to biggest
The memory hierarchy is the layered arrangement of a computer's storage, from a few fast registers at the top through cache and main memory down to disk, where each level down is larger and cheaper per bit but slower, so a small amount of fast memory sits in front of a large amount of slow memory.
You have built the two ends of a computer's memory. A register file is a handful of registers the CPU reads and writes in a single clock cycle: tiny, but instant. RAM is a large bank of addressed cells that holds the whole program and its data: roomy, but comparatively slow, because the address has to be decoded and the cell driven back across a long bus. Real machines do not choose one or the other. They stack several kinds of memory, each a different tradeoff between speed, size, and cost, and let the fast ones stand in front of the slow ones. That stack is the memory hierarchy.
Here is the honest map to what you have built: our register file is the top tier, the registers; our RAM is the main-memory tier below it. A real CPU inserts one more kind of memory between them, called a cache, and hangs a huge, slow disk underneath. This lesson is about why that layering exists and why it works so well. There is no new circuit to build here; the idea is an arrangement of the parts you already have.
The tradeoff: you cannot have fast, big, and cheap at once
Memory technologies force a choice. Fast memory is built from more transistors per bit, so each bit takes more silicon area and costs more, which means you can only afford a little of it. Cheap, dense memory packs bits tightly, so you get a lot of it, but reaching a bit takes longer. You can pick fast or big or cheap, but not all three in one technology, so a computer uses each technology for the tier it is best at.
The clearest example is the split between cache and main memory. A cache is made of SRAM, where each bit is a little latch of about six transistors: fast and stable, but bulky and expensive, so a cache is small. Main memory is made of DRAM, where each bit is just one transistor and a tiny capacitor: dense and cheap per bit, so you get gigabytes, but slower to read and it must be periodically refreshed because the capacitor leaks. Same job, storing bits, opposite tradeoffs, so both are used, at different tiers.
The levels, top to bottom
Ordered from fastest and smallest down to slowest and largest, a typical desktop or phone has these tiers. The exact numbers vary by machine, but the orders of magnitude are real, and the gap between the top and the bottom is enormous:
| Level | Built from | Typical size | Typical access time |
|---|---|---|---|
| Registers | flip-flops in the datapath | under a kilobyte | about 1 cycle |
| Cache (L1 to L3) | SRAM | tens of KB to a few MB | a few to a few tens of cycles |
| Main memory (RAM) | DRAM | gigabytes | tens to hundreds of cycles |
| Disk or SSD | flash or magnetic | terabytes | tens of thousands to millions of cycles |
The times are in cycles, not seconds, on purpose. What matters is the *ratio*, not the absolute value: a register access is effectively free within the datapath, while a single trip to DRAM costs the processor hundreds of cycles it could have spent computing, and a trip to disk costs so many that the operating system will run other work while it waits.
The desk, the shelf, the library, the warehouse
Picture yourself working at a desk. The few papers on the desk are your registers: right in front of you, grabbed instantly, but there is only room for a handful. A small bookshelf beside the desk is the cache: a few seconds to reach, holding the books you are actively using. The campus library is main memory: everything you might need is there, but each trip is a walk across campus. And a distant warehouse is the disk: it stores far more than the library, but fetching something means a slow drive across town.
You do not move the whole library onto your desk, because it does not fit and you would spend all day carrying. Instead you keep on the desk and shelf only what you are using right now, and walk to the library or drive to the warehouse only when you need something new. A computer runs its memory exactly this way: keep the hot data in the small fast tiers, and reach down to the big slow tiers only on the rare occasions you must.
Why a tiny fast tier works: locality
A small fast memory in front of a large slow one would be useless if the program's next access were random, because the fast tier could hold only a sliver of everything and would almost always miss. The reason it works is that real programs are not random. Their accesses cluster, a property called locality of reference, and it comes in two everyday flavors:
- Temporal locality: if you used an address recently, you are likely to use it again soon. A loop counter, a running total, the top of a stack: the same few locations get touched over and over.
- Spatial locality: if you used an address, you are likely to use one near it soon. Stepping through an array, or executing instructions in order, marches through neighbouring addresses.
Because of locality, at any moment a program is really working on a small working set of nearby, recently-used data, far smaller than all of memory. A cache is simply an automatic copy of that hot subset kept in fast SRAM: hardware watches what you touch, keeps a copy of it (and of its neighbours, for spatial locality), and serves the next access from the fast copy instead of the slow trip to DRAM. The tiny tier wins not by being big, but by holding the right small slice.
Locality is the single assumption the whole memory hierarchy rests on. It is why a cache of a few kilobytes can serve the overwhelming majority of accesses to gigabytes of RAM, and why the hierarchy gives you the illusion of a memory that is as large as disk yet nearly as fast as a register. Take locality away (truly random access across all of memory) and every tier below the top misses, and the machine crawls at the speed of DRAM or disk.
A worked example: what the fast tier is worth
Suppose a loop makes a million accesses into the same
512 eight-byte locations (a 4 KB working set), and put rough numbers on it: a DRAM access costs 200 cycles, while an access served from the fast tier costs 1 cycle. The first time each of the 512 locations is touched it is not in the fast tier yet, so it is fetched from DRAM. Every later touch finds it already sitting in the fast copy.- Cold accesses (first touch of each location):
512 x 200 = 102,400cycles. - All the rest, served fast:
(1,000,000 - 512) x 1 = 999,488cycles. - Total with the fast tier: about
1,101,888cycles, call it1.1million. - Total if every access went to DRAM:
1,000,000 x 200 = 200,000,000cycles,200million.
The fast tier cut the work from
200 million cycles to about 1.1 million, roughly 180 times fewer, and it did so while holding just 4 KB. That is the payoff of the hierarchy in one number: a small fast memory, backed by locality, turns a program that would spend nearly all its time waiting on memory into one that mostly runs at full speed.Common mistakes. First, the fast tiers do not hold a full copy of RAM; they are far too small. They hold only the current hot subset, and they only help because of locality, so a program with poor locality (chasing random pointers across gigabytes) defeats the cache and runs at DRAM speed no matter how clever the hardware. Second, do not confuse a register with a cache: a register is named explicitly by an instruction (the program decides what goes there), while a cache is automatic and invisible, managed by hardware that copies whatever you happen to touch. Third, bigger is not automatically faster: a larger memory array is generally slower to access, which is exactly why the tiers exist instead of one huge fast memory.
Try it
A program repeatedly sums the same 100-element array (each element 4 bytes) inside a tight loop, millions of times. Which tier of the hierarchy ends up serving almost all of those accesses, and which kinds of locality are at work?
Answer
The array is only
400 bytes, so it easily fits in the small fast tiers (an L1 cache, and the running total can even live in a register). After the very first pass loads it from DRAM, nearly every later access is served fast. Both kinds of locality apply: temporal, because each iteration re-reads the same elements, and spatial, because the elements are contiguous, so fetching one pulls its neighbours into the fast copy too. Only the first pass pays the slow DRAM cost; the millions of later accesses hit the fast tier.Every layer of a modern machine, from this 8-bit teaching CPU up to a phone or a server, is organised around this hierarchy. It is the reason the fast processor you built is not starved waiting on slow memory. Our register file is the register tier and our RAM is main memory; a real CPU slots a cache between them, and the same idea of keeping the hot subset close is what makes the whole system fast.
That covers why the layering exists and why it pays off. The mechanics of the cache tier itself, how it slices an address to find data, what a hit and a miss are, and how it decides what to keep, are the subject of the cache lesson, and the performance lesson turns these speed-versus-cost tradeoffs into an equation.
Frequently asked
What is the memory hierarchy?
The memory hierarchy is the layered arrangement of a computer's storage: a few fast registers on top, then cache, then main memory (RAM), then disk. Each level down is larger and cheaper per bit but slower, so a small amount of fast memory sits in front of a large amount of slow memory and serves most accesses.
Why do computers have different levels of memory?
Because no single memory technology is fast, big, and cheap at once. Fast memory (SRAM) uses more transistors per bit, so it is small and costly; dense memory (DRAM) is cheap and large but slower; disk is huge but far slower still. A computer uses each technology for the tier it suits, giving the illusion of storage that is nearly as fast as a register yet as large as a disk.
What is locality of reference?
Locality is the tendency of programs to access memory in clustered, predictable patterns. Temporal locality means a recently used address is likely to be used again soon (a loop counter). Spatial locality means addresses near a recently used one are likely to be used soon (walking an array). Locality is why a small fast cache can serve the vast majority of accesses to a much larger memory.
What is the difference between a register and a cache?
A register is named explicitly by an instruction, so the program controls exactly what value sits there, and it is read in about one cycle. A cache is automatic and invisible: hardware keeps a copy of whatever data the program happens to be touching, so a later access can be served from the fast copy instead of a slow trip to main memory. Registers are the top tier; cache is the tier just below them.
Every lesson here builds toward one thing: a working CPU, from the transistor up.
Open the free lab →