Interrupts

Two interrupt controllers behind one registration API. On x86-64 it is an IDT with IOAPIC routing, the local APIC for IPIs and timers, and MSI-X for PCIe devices; on arm64 it is GICv3, with PCIe MSI translated into LPIs by the ITS. Drivers see the same two functions on both.

The registration API#

kernel/interrupt.h is the whole surface:

Interrupt::Register(handler, irq, vector);       // edge, pinned to the BSP
Interrupt::RegisterLevel(handler, irq, vector);  // level, shared, balanced

Register is for the edge-triggered system lines — the PIT or HPET tick, the 8042 keyboard, the serial port. They stay on the CPU that registered them (the BSP) because timekeeping and the tick depend on it, and they own their GSI and their vector exclusively: a second registration on either is refused loudly rather than silently reprogramming the pin.

RegisterLevel is for PCI INTx lines. They may be shared — several devices on one GSI chain onto one vector, up to eight — and their destination CPU is chosen by IrqBalance. Chaining a level handler onto a GSI already owned by an edge handler is refused, because it would reprogram the pin and feed the edge handler phantom interrupts.

A handler is an object with a GetHandlerFn() (the assembly stub to put in the IDT) and an OnInterrupt(ctx) (used by the shared and arm64 dispatch paths). Handlers must not signal EOI when they are dispatched through the shared path — the dispatcher does it.

The x86 backend is kernel/interrupt.cpp (IDT + IOAPIC); the arm64 backend is arch/arm64/interrupt_arm64.cpp (GIC INTIDs). IRQ routing is deliberately not abstracted in the HAL yet — only the CPU-side operations are (hal/irqchip.h: IrqEoi, SendIpi, GetCurrentCpuHwId, IrqIsInService, SendNmiIpi).

x86-64: the IDT#

256 entries. The vector space is carved up like this:

Every slot starts out pointing at a per-vector DummyInterruptStub, 256 of them generated by a NASM %rep, each with its own vector number baked in. That matters: firmware that leaves an LVT armed, or a masked 8259 answering an ExtINT INTA cycle, otherwise lands in an anonymous panic with nothing to chase. A stray interrupt is reported a few times and then only counted.

Entry into C++ goes through a stub macro that pushes a full Context — every GPR plus the flags — and passes it as the first argument:

%macro InterruptStub 1
%1InterruptStub:
    PushAll
    mov rdi, rsp
    cld
    call %1Interrupt
    PopAll
    iretq
%endmacro

That Context is what makes bt, the panic report and the profiler possible: it carries the interrupted RIP, the original RSP and the frame pointer, so a stack walk can start exactly where the interrupt landed rather than guessing how many of its own frames to skip.

Shared vectors switch to SharedInterruptStub, whose dispatcher asks the local APIC which vector is actually in service (IrqIsInService) before calling anything, so handlers of unrelated shared vectors do not see phantom interrupts.

x86-64: IOAPIC routing#

Interrupt::Register programs the redirection entry directly. The IOAPIC driven is one: the MADT's whose pins start at GSI 0, the legacy IRQs' (or, on a machine with none such, the first listed) -- a machine may have several, as AMD's does, the FCH's at GSI 0 and the GNB's past it, and the driver takes a GSI for the index of its pin. An override naming a GSI past 255, or an IRQ overridden once already, is left out and said. RegisterLevel first asks ACPI for the GSI the ISA IRQ maps to and for the MADT polarity flags (bits [1:0]: 00 = bus default, 01 = active high, 11 = active low; PCI's bus default is active low), then asks IrqBalance for a destination CPU.

The PIC is remapped out of the exception vectors and then masked entirely, early in Main2.

MSI-X#

drivers/msix.h is one class with two backends: drivers/msix.cpp for x86 (a LAPIC-addressed message) and arch/arm64/msix_its.cpp for arm64 (a message the ITS translates). The table setup and the BAR mapping are identical; only the message composition and the interrupt install differ.

Setup() finds capability ID 0x11 in PCI config space, reads the table size from the message-control word ((ctrl & 0x7FF) + 1 entries), and reads the BIR and offset from the next dword. The table itself lives in one of the device's BARs, so it size-probes that BAR (both halves, if it is 64-bit — probing only the low half computes a bogus size for BARs above 4 GB), maps it with MapMmioRegion, and masks every entry.

EnableVector(index, handler) then, in this order:

1. Allocates a CPU vector from 0x40 upward, atomically (Cmpxchg on a shared counter, so concurrent device init cannot double-allocate).
2. Asks IrqBalance for the destination CPU. This happens before the entry lock is taken, because Balance() nests the two locks the other way round (IrqBalance → Retarget → entry lock).
3. Writes the entry: address 0xFEE00000 | (apicId << 12), upper address 0, data = the vector, control = masked.
4. Installs the IDT descriptor and calls back into the handler.
5. Unmasks the entry.
6. On the first vector only: sets INTx Disable in the PCI command register, sets MSI-X Enable and clears Function Mask — firmware may have left it set, which would keep every vector masked after the enable.

Retarget(index, apicId) masks the entry, rewrites the address and restores the previous control word: the address must not change while the entry is unmasked. EntryLock serialises all of that against a concurrent Mask.

Rust drivers reach the same table through the FFI (kernel_msix_create / kernel_msix_register_handler / …); their vectors land in 32 pre-generated stub slots that dispatch to the registered Rust closure under a read-write lock, with an in-flight counter so unregistering can wait out an ISR that is mid-call.

arm64: GICv3, and the ITS#

INTIDs are the identity here — Interrupt::Register's irq argument is the GIC INTID, and the vector argument is ignored. The ranges in use are SGI 1 (the IPI), PPI 27 (the generic timer), SPI 33/34 (PL011, RTC), SPI 48–79 (virtio-mmio), and LPIs from 8192 up (MSI).

All IRQs arrive at ArmIrqEntry, which loops:

• Read ICC_IAR1_EL1. INTIDs 1020–1023 mean "nothing (more) to acknowledge"; an empty first read is a genuine spurious interrupt, an empty re-read is just the loop draining. The check is bounded to that range on purpose — LPIs are numerically above it and must fall through, or they would be acknowledged (raising the running priority) and never EOI'd, wedging the CPU interface against every interrupt of equal or lower priority.
• The IPI and the timer tick EOI themselves, because their handlers may Schedule() away and never come back to the loop.
• LPIs go to Its::HandleLpi, everything else to the registered handlers, and ArmIrqEntry writes EOIR afterwards.

The ITS (arch/arm64/its.cpp) is what makes PCIe MSI work. At boot it maps its registers, provisions the device and collection tables and a command queue, enables LPIs on the boot CPU's redistributor, and maps one collection to it — all LPIs are delivered there, since there is no MSI balancing on arm64 yet. Per device it issues MAPD (with an interrupt translation table sized for the device's event count), and per vector MAPTI + INV + SYNC to bind (DeviceID, EventID) to an LPI. The DeviceID on QEMU virt is just the PCI requester ID. The MSI-X entry then gets GITS_TRANSLATER as its address and the EventID as its data.

its=off disables all of this; virtio-mmio devices do not need it, so the machine still boots and runs.

IRQ balancing#

IrqBalance records every device IRQ at registration time and spreads them round-robin across the running CPUs once SMP bring-up completes (Balance(), called from BpStartup right after StartAll). Devices that register later join the rotation immediately. System IRQs are not recorded and stay on the BSP.

IOAPIC lines and MSI-X vectors get separate cursors, because they cannot address the same set of CPUs: an MSI-X entry carries a full 8-bit destination, while an IOAPIC redirection entry in physical mode has four bits and reaches APIC IDs 0–15 only. Handing an IOAPIC line a higher ID does not spread it — it aliases it onto another CPU, or on a hybrid part onto an ID no CPU has, which loses the interrupt for good. The round-robin starts after the BSP so device IRQs prefer the other cores.

IPIs, and the NMI#

Hal::SendIpi(cpu, vector) is a LAPIC ICR write on x86 and an SGI on arm64. The IPI handler runs any queued IPITasks for this CPU and ends in Schedule(), which is what makes an IPI a reschedule request — and also why a softirq raise on the CPU that is already running the softirq task must not send one.

IPI tasks carry TLB shootdowns (see Paging), remote stack capture for bt, and CPU shutdown. The shootdown path panics after ten seconds naming the CPUs that never acknowledged.

The NMI is used for two things, and the handler asks about them in this order: a panic on another CPU asking for this one's stack (record it and halt — the panic is stopping everything anyway), then a performance-counter overflow, which is a profiler sample and the one kind of NMI this kernel must return from. Anything else panics. On AMD parts a counter interrupt can arrive just after the overflow that caused it was already handled, so one such NMI per handled sample is swallowed and counted (lscpu reports the total). arm64 has no NMI — Hal::NmiIpiSupported() answers false and the panic path skips remote stack capture.

Deferred work: SoftIrq#

Interrupt handlers do the minimum and raise a soft IRQ; the work runs in task context on the CPU that raised it, in one of the per-CPU softirq/N tasks. Four types today: NetRx, NetTx, BlkIo, TcpTimer.

A type runs on at most one CPU at a time — handlers are not reentrant — so a CPU that loses the race keeps its pending bit set and retries, and the CPU that finishes kicks the losers rather than leaving them until the next tick. The wake protocol (pending bit, then unblock; block, then re-check) is the one described in Scheduler.

SoftIrq::Init() runs late in boot, and several things depend on it: until it exists a block read returns without having read anything, which is why partition probing for the disks that came up under rust_init and the disk-log setup happen after it rather than next to the driver init.

Counters#

irqstat prints one counter per source: pit, hpet, 8042, serial, ipi, lapic-timer, shared, msix, dummy, spurious, posted, timer-wake. lapic-timer counts x86's ticks, and timer-wake the local timer's interrupts on either architecture that came for a sleeper's deadline and not the tick. The msix and shared rows are what answer "has this card stopped interrupting, or has it stopped receiving?" — a question that was unanswerable on a running machine until they were counted, and most of why the EX44 receive stall took as long as it did to corner.