From a CPU flag to Debian guests: a month of the nos hypervisor

On 2026-09-22 the nos hypervisor was a loadable module that could turn the CPU's virtualization extension on and off. On 2026-10-21 the same module, hv.ko, boots Alpine and Debian as they ship and gives Linux guests several CPUs, virtio disks, a switched network with NAT, ACPI and an IO-APIC — under AMD-V on a Ryzen 5 3600 and under Intel VT-x on an i5-13500, both on bare metal. This is stage 3 of the roadmap, told from git log: what each step took, and what each new machine showed that the one before it could not.

The shape, decided on day one#

The first commit (eec0894) set out three pieces: hvarch, the CPU's extension — VMCB, VMCS, run stubs, control MSRs — and all of the hypervisor's unsafe (48 sites then, 195 at the Intel merge a month later); hv, everything above it, safe — guest memory and its nested page table, the devices, the Linux loader, the run loop (two unsafe sites then, four now); and modules/hv, the shell command and the tasks a guest's CPUs run on. hv and hvarch are not default workspace members, so a kernel built without the module has no hypervisor in it at all.

A module, because the state it takes belongs to the CPU — EFER.SVME and VM_HSAVE_PA on AMD, CR4.VMXE and VMX root operation on Intel — and a machine that runs no guests should never touch it. And because unloading is a gate: hv-test.py turns the extension on for every CPU, unloads without turning it off, loads again and asks every CPU what it has. A control run with the off switch sabotaged failed six checks, which is how we knew the gate could fail at all.

AMD-V came first because development happens on a Mac, where the x86 kernel runs under QEMU's TCG, and TCG emulates SVM with nested paging and no VMX at all. It has holes of its own — before 9.2 it does not put a guest with paging off through the nested table — so the first built-in guests all started in long mode (more on TCG).

A month in one table#

A day to a shell#

The first VM (78fd761) ran guests of a handful of instructions: port I/O, a hypercall with all fifteen registers checked both ways, a write the nested table refuses, a triple fault, cli; jmp $. What keeps the host the host's is not left to policy: hvarch sets it on every entry, whatever the VMCB says — the host's interrupts, every port and MSR, #DB and #AC (a guest can make their delivery loop inside the CPU forever, CVE-2015-8104 and CVE-2015-5307), and #MC, raised again into the host's own handler.

Then Linux. The loader writes the zero page, the e820 map, page tables and a GDT through copying accessors — there is no Rust reference into guest memory anywhere, since the guest changes it under the compiler. The first boots hung in loops: 100,004 reads of the PIT's port 0x42 and 20,000 of the RTC's 0x71, waiting for a counter that never counted and an update bit that never cleared (6688f58). With a PIT, an RTC, an 8259 pair and a timer tick, a tinyconfig 6.18 reached BusyBox — and its timer died a second later: an idle Linux runs sti; hlt, the HLT sits in the STI's interrupt shadow, and injection was refused 3.7 million times in a diagnostic (ccbad38). Then BusyBox's line editor asked where the cursor was (ESC[6n) and spun polling the port for an answer that was only fed at an idle HLT (e0a2305). On 2026-09-23, the day after the first commit:

$ hv boot /bzImage initrd=/initrd cmdline=console=ttyS0 nolapic rdinit=/init
  --- ttyS0 ---
[    0.000000] Linux version 6.18.53 ...
...
[    0.540000] Run /init as init process
nos-guest: init is up, / on ramfs, busybox v1.36.1
...
BusyBox v1.36.1 built-in shell (ash)
~ # id
uid=0 gid=0

All four demos of the stage 3 plan were done. The next commit had an idle guest's HLT halt its vCPU instead of exiting some 46,000 times a second: over 120 s of the same boot, 5,633,131 exits became 76,334 (aa8e263).

What real hardware said#

The same day the guest ran on the AX41's Zen 2 — Run /init 18 ms into the guest's clock, against half a second under TCG — and stopped twice where TCG had never gone (532219c). CPUID leaf 0x80000001 had passed through whole but for SVM, and a Zen 2 has MWAITX there: the guest's udelay became MONITORX, an intercept with no answer. And Linux on any Zen reads AMD's FCH reset-status register at 0xFED803C0, whether or not there is one. The leaf became an allow-list, an instruction CPUID did not offer became the #UD a CPU without it gives, and an unanswered read of the MMIO window now finds a read-only page of all ones — kept honest under TCG by a built-in guest, absent, that reads it.

The AX41 kept doing that. A guest that rebooted spun at 100% of its CPU: 1.3 million reads in 48 seconds of the status of an 8042 that is not there (f58af03). Debian's clock counted 25 seconds out of 62: at 250 Hz, woken at the host's 100 Hz tick, its vCPU was handed one timer edge for every two or three periods. Every elapsed period is now owed and handed over one at a time, as KVM's PIT does, and hv-distro-test.py holds each guest's idle clock to nos's — 1.000 with the fix, 0.514 without (768a56b).

Guests that stay up#

hv start leaves a guest running; hv list, exec, send, wait and stop reach it, and hv attach gives its console live over ssh -t (98100a6, 3aab12c). Disks are legacy virtio-blk over PCI, reached by port I/O — still no instruction decoder — and the rings are the guest's: every index, address and length is checked before use (cf32d7c). The network is virtio-net on a switch inside the module, nos's end a virtual NIC, hv0, in its own stack (1f67325); next came NAT, a DHCP server on the switch and nos's DNS server for the guests (45df223).

Alpine's virt ISO then booted as it ships (47568b5), needing what the purpose-built kernel never did: the PIT's one-shot mode for high-resolution timers; an RTC with the host's time — it had answered a fixed 2026-01-01, and OpenRC took every file on the ISO for one from the future; and a CPUID topology of one CPU, since the guest had taken the host CPU's APIC ID for its own. Debian 13's cloud image followed the same night, provisioned by systemd.set_credential= on its command line (8928493). On the AX41 a cold apt-get update took 39 s and dropped some 1,400 frames, because a frame for a busy guest waited for the host's next tick; the switch now kicks a running vCPU out of its guest, as KVM does (3c8bd36).

VT-x, and what nested KVM hid#

The Intel backend arrived on 09-24 (232a1d8), brought up under nested KVM on an Intel machine. One bit outside the hypervisor stood in its way: VMX requires CR0.NE, nos never set it, and vmxon would have faulted into a panic; boot64.asm now sets it on every CPU, and hv on checks the fixed bits first and names a CPU that refuses (dc059db). After that, each fix answered a real kernel crashing:

• Linux booted whole, then oopsed at RIP: 0x0 the moment it ran /init: userspace's first SYSCALL jumped to the host's LSTAR. VMX has no vmsave, so the syscall MSRs now go through the MSR-load lists (0eb1531).
• Alpine double-faulted the first time an interrupt returned to user mode: swapgs changes KERNEL_GS_BASE with no wrmsr to intercept, so the guest's value is now stored back at every exit (1d293b7).
• Alpine took tens of minutes to its login prompt: sixty-odd vmread/vmwrite an exit, each a trap to the outer KVM. Synced lazily it took about 19 s (495b58b), and measured and thinned, an exit went from 17.9 µs to 5.8 (0478d8b).

A review against the manual then found what nested KVM covers up — every item passing the gates as they were (e433276). The worst: CR8 was not intercepted, so a guest's mov cr8, 15 wrote the host CPU's own task priority and kept every interrupt but NMIs off it, the TLB shootdown IPI included — a panic ten seconds later. A built-in guest, tpr, now checks that the host's CR8 is still 0 after a guest has written 15.

The machine Linux expects#

cpus=N (14a03e0) gives a guest up to 16 CPUs, each with an x2APIC, the others started as a PC starts them: INIT, then a start-up IPI into real mode. Linux promptly measured its APIC bus at 10,005 MHz and refused the timer — spinning in calibrate_APIC_clock it made no exits, then took the PIT's owed ticks in one burst; PAUSE exits fixed it. On the AX41 a guest's two CPUs on two CCXs saw their TSCs 2,052 cycles apart: on those AMD families LFENCE serializes only with a DE_CFG bit the firmware had left clear, which nos now sets at boot (c94e584). MSI-X took a receive interrupt from six exits through the 8259 to one: a two-CPU Debian fetched a gigabyte three times at 117 MB/s without dropping a frame (0f9480b).

On 09-29 came ACPI, whose DSDT must contain the PCI host bridge, because a Linux with ACPI never probes the bus without it (bc9e15c) — and hv stop began by pressing the power button:

$ hv stop 0 secs=300
hv: vm 0 stopped -- the guest powered itself off, S5 by its ACPI PM1 control register, at 0xffffffff8f1d5ae3

MMIO arrived the same day (57c9826): the faulting instruction fetched through the guest's own page tables, decoded — only the mov forms Linux's MMIO accessors use, checked against clang's encodings of 1,054 instructions and two million random byte strings — and performed. The xAPIC page was its first device, an IO-APIC its second (86a1721).

Fuzzed, reviewed, isolated#

That evening hv-fuzz compiled the hypervisor's own sources for the host and drove them — each device, the MMIO path, the loader, and the whole machine under the real run loop with a script in place of the CPU — and reported five findings, among them a disk status write whose address wrapped past 2^64 (4ce4e77). A day later its guestnet target found, on its first input, that a guest could send as another and take its NAT flows; the switch now checks every frame where it enters (d1aeecf; more in the fuzzing post).

On 10-10 a review of the whole hypervisor by five readers at once left a list, fixed a commit an item (57347b1), and then its isolation section (19d0d14): guests had been able to reach nos's own sshd and hold its login slots, NAT let them reach a cloud's metadata address, and the CPUID policy told them they had speculation controls they did not have. The speculation part has a post of its own.

Intel silicon, and the Intel round#

On 2026-10-20 nos booted on the EX44 — an i5-13500, six P-cores of two threads and eight E-cores — from its system image, beside Ubuntu. Every built-in guest passed on a P-core and on an E-core; an exit cost 1.67 µs and 2.07 µs, against 5.8 under nested KVM. Linux ran with one, two and four CPUs, a guest fetched a gigabyte through NAT in 9.1 s, and rmmod with five VMs running stopped them all and left VMX off on all twenty CPUs. The silicon also confirmed the VT-x gaps the review had listed; their fixes were merged the next day (48bfaf5).

The Intel round merged the same day (3912397). Every core had been running at its 2.5 GHz base clock; with hardware P-states on, a guest's busy loop took 3.06 s where it had taken 5.55 (Linux on the box: 2.81). The first HWP version hung the EX44's boot with no console to show why, so panic=N now resets a machine N seconds after a panic's report. Guests are shown leaf 7 and ARCH_CAPABILITIES, so Alpine stopped believing its CPU open to Meltdown, and pause-loop exiting cut a two-CPU guest's 20-second boot from 299,508 exits to 99,297. The virtual APIC is in too, but opt-in (apicv): in a guest-to-guest TCP test it cut exits by 25 to 50% and still averaged 5.88 s against 5.70 without, because an interrupt for another vCPU still needs a kick that posted interrupts would remove.

What is next#

What is missing is listed in the hypervisor doc: device work off the first vCPU's task, posted interrupts, XSAVE and AVX for guests (a distribution built for x86-64-v3 does not start), modern virtio, the HPET and PCIe's ECAM; an arm64 backend that today reports and refuses, since nos drops EL2 at boot; and stage 4's control plane, which will put hv start behind an API. To try what exists, see running a Linux guest, or the hypervisor page for the whole picture.

Update, 22 October 2026: posted interrupts were merged the day after this post, and with them the CPU's own delivery of a guest's interrupts became the default wherever the CPU has posted interrupts. With a one-shot host timer, moderated receive interrupts, a NIC's offloads and mapped virtqueue rings after them, the guest-to-guest TCP test above takes 2.72–2.83 s; a NIC's work moved to a task of its own was measured no faster for a guest of one CPU and left out. The story is in Guest networking from 5.7 seconds to 2.8.