CSEC3616Cybersecurity Engineering

    Hardware protection and privilege rings

    How memory protection, privilege rings and isolated cryptographic hardware enforce access control beneath every operating-system mechanism this module has covered so far.

    • Explain segment addressing and why it makes a segmentation fault a security mechanism, not just an error.
    • Trace x86's ring model from the 80286 through to the nine-ring picture modern Intel CPUs actually implement.
    • Distinguish ARM's MPU and MMU tiers, and explain what CHERI adds on top of both.
    • Explain how a TPM protects a key without ever exposing the key itself.

    16 min read

    Intuition

    Every access-control rule this module has covered, DAC, MAC, Unix permissions, the same-origin policy, is enforced in software, and software runs on hardware that has to agree to enforce it. If a process could simply read another process’s memory directly, or run kernel code from user space by choice, none of those rules would matter. Hardware protection is what makes them unbreakable from underneath, rather than merely unbroken by convention.

    Mechanism

    Memory protection keeps one process’s memory apart from another’s. The CPU addresses memory with two registers together: a segment register, which identifies which segment of memory is meant, and an address register, which points to a location inside that segment. A process that tries to reach outside its own segment triggers a segmentation fault, SIGSEGV, on both reads and writes, and the operating system, not the process, decides what happens next.

    Threat

    A process reads or writes memory belonging to another process, or to the kernel, by addressing outside its own allocated segment.

    Control

    Segment addressing plus the SIGSEGV check on every access is the hardware’s own enforcement of the boundary. It runs beneath the operating system’s own access control, so it holds even against a process that has already found a way around software-level restrictions.

    Mechanism

    Privilege rings layer the CPU’s own permissions the same way. Code runs in one of several rings, each a distinct privilege class, and the current privilege level can only be changed by code already running in ring 0. Ring 3 code, the outermost and least privileged, cannot reach ring 0 objects directly; it has to go through a gate, a controlled entry point that lets code run at a different privilege level under supervision, and the same mechanism manages supporting infrastructure such as multiple stack segments.

    Two Intel chips set this up historically. The 80286 introduced segment addressing and protection rings in the first place. The 80386 added built-in virtual memory and expanded memory segments to 4GB.

    Mechanism

    Modern Intel CPUs carry this further than the classic four rings. There are actually nine rings: ring 0 to 3 for normal code, a second ring 0 to 3 underneath that for the hypervisor’s VMX root mode, and system management mode (SMM) at the bottom for the BIOS. In practice only four of the nine are used: SMM, ring 0 of VMX root mode, the ordinary ring 0 for the operating system kernel, and ring 3 above that for applications. Rings 1 and 2, in both sets, sit unused.

    The lecture’s own diagram of this, Figure 2, credited to Wikipedia, did not survive PDF extraction. The diagram below is built from the same four levels the notes describe directly.

    x86 privilege rings — ring 3 to SMMRing 3 — applicationsRing 0 — OS kernelVMX root ring 0 — hypervisorSMM
    outermost — least privilegedinnermost — most privileged

    The current privilege level can only be changed by ring 0 code; lower rings reach higher ones only through gates. Modern Intel CPUs actually define nine rings — 0–3 for normal code, a second 0–3 set for the hypervisor's VMX root mode, and SMM beneath that for the BIOS — but rings 1 and 2 of each set are rarely used. The four squares above are the levels the lecture says are used in practice: applications in ring 3, the OS kernel in ring 0, the hypervisor in VMX root ring 0, and firmware in SMM, each level able to see and control everything outside it.

    Exam detail

    Nine rings exist; four are used. That gap is worth stating explicitly, because “how many privilege rings does x86 have” and “which privilege levels are actually used” are different questions with different correct answers. Applications run in ring 3, the operating system kernel in ring 0, the hypervisor in VMX root ring 0, and firmware in SMM, each level able to see and control everything less privileged than it.

    Mechanism

    This is the same principle of least privilege the rest of the module keeps returning to: all code should run with the minimum privilege it needs, never more, and the number of paths into the kernel should be kept to the smallest set possible and tightly controlled. Rings are the hardware’s version of that rule. Less-privileged code, further from ring 0, cannot reach more-privileged code directly; every path in goes through a controlled gate instead.

    Mechanism

    ARM is the processor family in most phones, tablets and IoT devices. Unlike Intel, ARM licenses processor cores to other chip designers rather than building complete chips itself, so its hardware protection is far more customisable. Early ARM cores had no memory management built in at all. Current variants split into two tiers: a memory protection unit (MPU), which enforces fixed memory regions without virtual addressing, and a memory management unit (MMU), which adds full virtual memory on top.

    ARM’s latest addition is CHERI (Capability Hardware Enhanced RISC Instructions), which brings fine-grained capability support to the architecture. A process spawning a subthread can hand it read and write access to a specific memory range, rather than the whole address space, so several sandboxes can run inside a single process. The lecture frames the payoff as a long-term promise rather than a shipped result: used thoroughly across an operating system like Windows, Android or iOS, CHERI has the potential to prevent most of the zero-day exploits seen in recent years, but that depends on adoption that has not happened yet.

    Compare

    Enforces a fixed set of memory regions and their permissions. No virtual addressing, lighter weight, common on smaller or real-time ARM cores.

    Adds full virtual memory management on top of protection, translating virtual addresses to physical ones the way a desktop or phone CPU does.

    Mechanism

    Isolated cryptographic components take a different approach: instead of isolating code, they isolate the key itself. A cryptographic key stored in dedicated hardware never has to leave that hardware to be used; the surrounding system reaches it only through a restricted interface that performs authorised operations without ever exposing the raw key. The lecture’s example is the TPM (Trusted Platform Module), originally built for Digital Rights Management, which now provides that kind of isolated storage for cryptographic operations more broadly.

    Threat

    Malware running with full access to the operating system reads a cryptographic key directly out of memory or disk, because the key was stored the same way as any other piece of data.

    Control

    A TPM never releases the key itself. Software asks the TPM to perform an operation, such as signing or decrypting, through its dedicated interface, and only the result crosses back out. Compromising the operating system does not hand an attacker the key, because the key was never reachable from there in the first place.

    Pitfall

    Do not describe hardware isolation as something that replaces the access control discussed earlier in this module. Segmentation, rings and isolated key storage sit underneath DAC, MAC, ACLs and Unix permissions; they are what makes those enforceable at all, not an alternative to them.

    Recall

    Modern Intel CPUs have nine privilege rings. How many are actually used, and which ones?

    Four: system management mode (SMM) for firmware, ring 0 of VMX root mode for the hypervisor, the ordinary ring 0 for the operating system kernel, and ring 3 for applications. Rings 1 and 2, in both the normal and VMX root sets, sit unused.