x86-64 Kernel Internals: Stacks, Interrupts, and Context Switches

1. Memory & Pointers Fundamentals

Before we can understand stacks or interrupts, we need a rock-solid model of what memory actually is at the hardware level. Everything else builds on this.

Memory as an Array of Bytes

Physical memory (RAM) is conceptually a giant array of bytes. Each byte has an address — a number that identifies its position in this array. On x86-64, addresses are 64-bit numbers, though typically only the lower 48 bits are used.

Bytes vs Bits, and Multi-Byte Values

A byte is 8 bits. One byte can represent values 0–255 (unsigned) or -128 to +127 (signed). But most useful values need more than one byte:

Little-Endian Byte Order

x86-64 is little-endian: the least significant byte is stored at the lowest address.

Registers vs Memory

Registers are tiny storage locations inside the CPU itself. They are not part of RAM. The CPU can access registers in a single clock cycle — hundreds of times faster than RAM.

2. Code Execution Model

Now that we understand memory and registers, let's see how the CPU actually executes your code.

Where Machine Code Lives: The Text Segment

The Instruction Pointer: RIP

The Fetch-Decode-Execute Cycle

1. Fetch: Read the bytes at address RIP from memory.
2. Decode: Figure out what instruction this is and what operands it uses.
3. Execute: Perform the operation.
4. Advance RIP: Move RIP forward by the instruction's length.
5. Repeat.

CALL vs JMP: What CALL Does Extra

3. Stack Basics

The stack is just a region of memory. There's nothing magical about it — it's bytes like everything else. What makes it special is how we use it.

RSP: The Stack Pointer

Why the Stack "Grows Down"

On x86-64, the stack grows toward lower addresses. When you push, RSP decreases. When you pop, RSP increases.

PUSH and POP: The Exact Operations

4. Stack Frames

Function Prologue and Epilogue

System V AMD64 ABI: Argument Passing

Return values go in RAX.

Stack Frame Layout Diagram

5. Calling Conventions & Register Saving

Caller-Saved vs Callee-Saved Registers

Stack Alignment: The 16-Byte Rule

The System V ABI requires RSP to be 16-byte aligned at the point of a call instruction.

6. Interrupts & CPU Exceptions

The Interrupt Descriptor Table (IDT)

What the CPU Pushes on Entry

IRETQ vs RET

7. Stack Switching on Interrupts

Privilege Rings: Ring 3 vs Ring 0

The TSS: Task State Segment

IST: Interrupt Stack Table

The IST provides up to 7 dedicated stacks for critical exceptions. Each IDT entry can specify an IST index (1-7). If non-zero, the CPU loads RSP from that IST entry, regardless of current privilege level.

8. Context Switches

Thread Switch vs Process Switch

CR3 and Page Tables

Why Context Switches Are "Slow"

• TLB flush: After switching CR3, all cached address translations are invalid.
• Cache pollution: The new thread's working set isn't in L1/L2/L3 cache.
• Branch predictor pollution: The CPU's branch predictor has learned the old thread's patterns.

9. Kernel-in-Rust Practical Guidance

The x86-interrupt Calling Convention

Rust's extern "x86-interrupt" is special. Unlike extern "C", it:

• Expects the interrupt stack frame to already be on the stack
• Automatically saves/restores all scratch registers
• Uses iretq to return, not ret
• Handles the error code for you (if present)

use x86_64::structures::idt::InterruptStackFrame;

extern "x86-interrupt" fn page_fault_handler(
    stack_frame: InterruptStackFrame,
    error_code: u64,
) {
    // Handle page fault...
    // Compiler generates iretq, not ret
}

Common Pitfalls

Setting Up TSS and IST in Rust

use x86_64::structures::tss::TaskStateSegment;
use x86_64::VirtAddr;

pub const DOUBLE_FAULT_IST_INDEX: u16 = 0;

lazy_static! {
    static ref TSS: TaskStateSegment = {
        let mut tss = TaskStateSegment::new();
        
        tss.interrupt_stack_table[DOUBLE_FAULT_IST_INDEX as usize] = {
            const STACK_SIZE: usize = 4096 * 5;
            static mut STACK: [u8; STACK_SIZE] = [0; STACK_SIZE];
            let stack_start = VirtAddr::from_ptr(unsafe { &STACK });
            stack_start + STACK_SIZE  // Stack end (grows down!)
        };
        
        tss
    };
}

Summary

You now have a complete mental model. Go write that kernel! 🦀