ASLR, DEP, and Modern Exploit Mitigations

ASLR, DEP, and related exploit mitigations are defensive features built into operating systems, compilers, and hardware that make memory-corruption vulnerabilities far harder to turn into working exploits. They do not fix the underlying bugs; instead, they raise the cost and complexity of exploitation so that many flaws become impractical to weaponize. Understanding these mitigations is essential for both defenders hardening systems and researchers studying how attacks and defenses co-evolve.

The Problem Mitigations Address

Most memory-corruption exploits share a two-part strategy. First the attacker corrupts memory using a bug such as a buffer overflow — see memory safety for how these bugs arise. Then they leverage that corruption to redirect execution to code of their choosing, typically to achieve arbitrary code execution.

Historically both parts were easy. Program memory layouts were predictable, so an attacker knew exactly where to jump, and any memory region could hold executable code, so injected data could simply be run. Exploit mitigations attack each of these assumptions in turn, forming layers of defense that an attacker must defeat simultaneously.

Data Execution Prevention (DEP)

Data Execution Prevention (DEP), also known as NX (No-eXecute) or W^X (write XOR execute), enforces a simple rule: a page of memory may be writable or executable, but not both. Memory holding data — the stack, the heap, input buffers — is marked non-executable, while memory holding code is executable but read-only.

This defeats the classic code injection attack, in which an attacker writes machine code (shellcode) into a data buffer and jumps to it. With DEP enabled, the processor refuses to execute instructions from a data page and faults instead. DEP is enforced in hardware through a dedicated bit in the page tables, making it cheap and effective.

Return-Oriented Programming

DEP forced attackers to stop injecting new code and instead reuse code already present in the program. Return-Oriented Programming (ROP) chains together short instruction sequences called gadgets — snippets that already exist in the binary and its libraries, each ending in a return instruction. By carefully arranging return addresses on the stack, an attacker strings gadgets together to perform arbitrary computation without injecting any code, sidestepping DEP entirely. ROP is a major reason no single mitigation is sufficient.

Address Space Layout Randomization (ASLR)

Address Space Layout Randomization (ASLR) attacks the attacker's other assumption: predictability. Each time a program runs, ASLR randomizes the base addresses of key memory regions — the stack, the heap, shared libraries, and, when the binary is a position-independent executable (PIE), the program code itself.

Because ROP and similar techniques require knowing the exact addresses of gadgets and functions, randomizing the layout means the attacker no longer knows where anything is. A hardcoded address that worked once will point somewhere harmless — often an unmapped page — on the next run, crashing the program instead of exploiting it.

ASLR's strength depends on entropy: how many possible layouts exist. On 64-bit systems the large address space provides substantial entropy, making blind guessing impractical. This is one reason 64-bit ASLR is considerably stronger than its 32-bit predecessor.

Defeating ASLR

ASLR's weak point is the information leak. If a separate vulnerability lets an attacker read a single pointer from memory, they can compute the randomized base address of a region and recalculate every gadget's location. This is why real-world exploits frequently chain an information-disclosure bug with a memory-corruption bug: the leak defeats ASLR, and the corruption then leverages DEP-bypassing ROP.

Complementary Mitigations

DEP and ASLR are the foundation, but modern systems layer several more defenses.

• Stack canaries (stack-smashing protection) place a random guard value between local variables and a function's saved return address. Before returning, the function checks the canary; a linear stack overflow that overwrote the return address will also have corrupted the canary, so the program aborts.
• Control-Flow Integrity (CFI) restricts indirect jumps and calls to a precomputed set of legitimate targets, directly disrupting ROP by rejecting returns and calls that land in the middle of functions or on unintended gadgets.
• Shadow stacks, increasingly available in hardware, keep a protected second copy of return addresses and detect any mismatch, defeating return-address overwrites.
• Pointer authentication, available on some architectures, embeds a cryptographic signature in unused pointer bits so that forged or corrupted pointers are detected on use.

Because each mitigation targets a specific step of exploitation, their real power is cumulative: an attacker must chain bypasses for all of them at once.

Enabling Mitigations in Practice

Mitigations only protect software that is actually built and run with them. Developers should verify they are enabled rather than assume defaults:

1. Compile with hardening flags so stack protectors, non-executable stacks, and position-independent code are emitted.
2. Build position-independent executables (PIE) so ASLR can randomize the main program image, not just libraries.
3. Enable relocation read-only (RELRO) to protect internal linking structures from being overwritten.
4. Keep the operating system and compiler updated so hardware-backed mitigations like shadow stacks and CFI are available.

A representative hardened compiler invocation looks like this:

gcc -fstack-protector-strong -D_FORTIFY_SOURCE=2 -O2 \
    -fPIE -pie -Wl,-z,relro app.c -o app

These mitigations are powerful, but they are a second line of defense. The most reliable protection remains preventing the memory-corruption bugs in the first place, for instance by adopting memory-safe languages for new and high-risk code.

Key Takeaways

• Exploit mitigations do not remove memory-corruption bugs; they make those bugs much harder and costlier to exploit.
• DEP (NX / W^X) blocks code injection by making data memory non-executable, which pushed attackers toward Return-Oriented Programming.
• ASLR randomizes memory layout so attackers cannot predict addresses, but it can be defeated by an information leak.
• Stack canaries, Control-Flow Integrity, shadow stacks, and pointer authentication add complementary layers that must all be bypassed at once.
• Developers must explicitly enable hardening flags and PIE, and treat mitigations as defense in depth behind fundamentally safer code.