Filesystems

How files work in nos: what gets mounted at boot and how the root is chosen, the file API kernel code uses, what the ext2 driver does and does not do, and how a root filesystem image is made and checked. The shell side is in Shell commands.

What is mounted at boot#

rust_mount_root_fs() (src/rust/fs/src/rootfs.rs) runs late in the boot, after the disks that appear during rust_init (NVMe) have been probed for partitions, and does what root= on the command line asks:

ro mounts the root read-only. A procfs goes on /proc in every case (the directory is created on the root if it is missing). If the named root is not found the fallback layout is mounted and the boot continues, so a wrong root= costs a shell without persistence rather than a box without a shell.

Both shipped configs say root=auto: the ISO (build/grub.cfg) so that any disk labelled nos attached to QEMU becomes the root, and the disk image (build/grub-disk.cfg), whose one partition carries that label. The label is what makes the same command line work whatever bus the disk is on — virtio on QEMU and the clouds, NVMe on the Hetzner boxes.

fstest=on runs the filesystem self-test (below) on / right after the mount; the smoke tests boot with it and assert fstest: passed.

The file API#

Rust reaches files through crate::vfs_instance() (src/rust/fs/src/vfs.rs), and C++ through the kernel_vfs_* calls it exports -- there is no C++ view of the layer, so a file that needs one declares the calls it makes where it makes them (kernel/module.cpp reading a .ko, kernel/cmd.cpp for crc32, sha256, wget and /etc/rc). Paths are absolute; the longest mount prefix wins; . and .. resolve in the tree, and .. stops at a mount root.

let file = Open::new(vfs, b"/lib/modules/foo.ko", OPEN_READ)?;   // closed when it goes
let got = file.read(&mut buf)?;                                  // 0 at end of file

An open file is a Handle: a slot in the VFS's table of open files and the generation of what is in it. That is what crosses the C ABI as well -- the RustFile* the C++ shell holds is that word and not an address -- so a handle from anywhere is looked up rather than followed, and one that was closed, or never opened, or whose filesystem has been unmounted at shutdown, reads as no file instead of as whatever memory is there now.

• open(path, flags) with OPEN_READ, OPEN_WRITE, OPEN_CREATE, OPEN_TRUNCATE, OPEN_APPEND (which puts every write at the end and implies OPEN_WRITE); close, read, write, seek, tell, size. A write past the end extends the file; the gap reads as zeros.
• stat (type, size, inode), read_dir(path, index) to walk a directory by index, rename (a move within one filesystem; a directory cannot move under itself), truncate (either way), sync, remove (a directory goes with everything under it), create(path, directory).
• files.rs for the whole-file cases: list_dir and read_file for what the shell prints, replace_file/locate for a configuration file that a full disk or a crash midway must not leave empty, and the kernel_file_* ABI over them that a module -- and the shell's /etc/rc -- reads and writes by.

An open file pins its vnode: remove and rename refuse a file (or a directory containing one) that is open, and unmount refuses a filesystem with open files. unmount_all at shutdown does not refuse — it is shutdown — and the handles it abandons stop naming anything.

A mounted filesystem claims its block device (kernel_blockdev_claim_as) until it is unmounted, and so do the disk log and a module writing to a device direct (blkload's write tests) -- and the shell's format and diskwrite, for as long as they write. A claim is refused while another overlaps it — a stretch of the same disk: the same device, the disk a partition is on, a partition of that disk, or another partition of it that the disk's table has overlap (the table is whoever wrote the disk's to choose) — so a mount fails (vfs: the device is in use by a mounted filesystem in dmesg) on the disk under a mounted partition, on the disk log's area, or on a device a write test is running on; and those, in turn, keep off a mounted one. Reads need no claim.

One mutex serialises every VFS call, so a multi-megabyte read holds up an ls from the UDP shell for its duration. Everything it guards is inside it: the mounts, each owning its filesystem, and the open files.

What a filesystem is#

A filesystem is a type that implements vfs::FileSystem, mounted as a Box<dyn FileSystem> the mount owns; unmounting drops it. The trait is the whole seam: mount/unmount, root, lookup, load_dir, create_file, create_dir, read, write, truncate, rename, remove, sync, and name/info/device for what mounts prints and the device it claims. Every call arrives with the VFS lock held, so a filesystem sees one call at a time and does no locking of its own.

What a filesystem holds is a vnode::Tree: an arena of nodes -- a name, a kind, a size, an inode number, the children of a directory in order -- named by NodeIds, a slot and the generation of what is in it. Parents, children and open files all refer to a node by that number, never by an address, so the tree is ordinary owned data: a filesystem hands the VFS an id and the VFS asks the filesystem's tree what it is. An id of a node that has been freed finds nothing. The slots are taken from the allocator in chunks and fallibly: an image says how many nodes there will be, and an image is not to be trusted with that.

There is no unsafe in the VFS, the tree or any of the four filesystems. ext2's on-disk structures are read and written whole through kcore::pod, which is where the two pointer casts that takes are; what is left in the crate is the C ABI at its edge, where a path or a buffer arrives as a pointer and a length.

ext2#

src/rust/fs/src/ext2.rs reads and writes ext2 rev 1 with 1, 2 or 4 KiB blocks: direct, indirect and doubly-indirect blocks (a file can reach 4 GiB at 4 KiB blocks; the triple-indirect block is not implemented and a file needing it is refused), sparse files, directories of any size. Directories are read from disk the first time a path walk enters them, not at mount, so the memory cost is what is used, not what is on the disk -- all of a directory or none of it: one whose read fails part way keeps nothing it read, and is read again whole at the next look.

What it refuses, and why:

• a geometry that does not hold together: more blocks than the device has, an inode count that is not its groups' inodes, a group whose bitmaps or inode table lie outside the filesystem. Every number the driver works out from the superblock is then one that cannot overflow or lead off the device.
• any incompat feature other than filetype — ext3 with journal recovery pending, ext4 extents, 64bit, meta_bg: same magic, different format.
• an image without filetype (rev 0): directory entries do not say what they name.
• any ro_compat feature other than sparse_super and large_file is mounted read-only: gdt_csum above all, whose group checksums a write would silently break.
• symlinks and device nodes are left out of the tree; a file over 4 GiB is skipped with a message.
• a write or a truncate past what the blocks map -- 64 MiB at 1 KiB blocks, 512 MiB at 2 KiB, 4 GiB at 4 KiB -- is refused whole, before anything is allocated. A write that fails part way (the disk full, a block that would not read) gives back what it allocated past the end of the file, and the file keeps the size it had; what it wrote inside that size stays.
• a name with a NUL in it: the VFS refuses one for every filesystem.

Writing keeps the on-disk structures consistent for e2fsck and for Linux, which can mount the same partition (that is how modules get onto a Hetzner box: copy them in under Ubuntu). The rules the driver follows, in place of the journal ext2 does not have:

• Order. Allocation bitmaps go down first, then the data and indirect blocks (plain writes) followed by a device flush, then the inode, then the free counts in the group descriptors and superblock. A crash at any point leaves at worst blocks marked used that nothing references — e2fsck reclaims them — and never a block both referenced and free. A create is the same: the bits, then the inode, then the name; and one undone after an error writes its inode deleted before freeing the bits, or, when that write fails, leaves them taken.
• A file's data pays for what it changes. A write that allocated nothing — a disk image having its blocks written over, a guest's — moved no pointer and changed no bit, so nothing is ordered against it: the blocks go down plainly, and so does the inode, and only when the size or the second of the mtime changed; sync puts them on the medium, and a guest's flush is a sync. A write that allocated flushes the device once — the bitmap and the data it wrote then on the medium — and writes what points at them after that, plainly: each indirect block it filled in once, however many pointers of it the write set (they wait in its buffer for the flush; one written as it was set would have reached the disk before the bitmap), the inode, the counts. So what a write leaves in the device's cache reaches the medium at the next flush -- another write's that allocates, a sync, the unmount -- and a caller that needs it there sooner syncs, as the two-step replace, grubenv and a guest's flush do. Creates, renames and removes keep the FUA writes of their inodes and bitmaps.
• Frees run the other way. A truncate cuts the tail off the block tree, commits the inode without it, and only then clears the bits, in batches of 512 blocks so a large file does not hold a huge list in memory. A remove takes the name out of its directory first, so a crash leaves an orphan for lost+found, never a name leading nowhere; a directory goes deepest first, by a walk and not a recursion, as deep as the paths reach.
• Renames add the new name before dropping the old one, so a crash leaves the file reachable twice, which e2fsck reduces to once.
• The valid state bit is cleared at a read-write mount and set again at a clean unmount, as Linux does. A boot after a crash prints was not cleanly unmounted and mounts anyway; run e2fsck under Linux when convenient. The unmount happens on poweroff and reboot, before the soft IRQs stop (a block request completes through one).
• Directories are modified linearly and the htree index flag is dropped from any directory the driver touches; images for nos are made with -O ^dir_index so Linux never builds an index in the first place.
• Only the primary superblock and group descriptors are updated; the backups drift, which e2fsck tolerates (-b uses them for recovery only).

There is no cache of data: a file's blocks go to and from the device as the call asks for them, a batch at a time -- up to 64 blocks, a page of a 256 KiB buffer each (Disk::write_pieces, read_pieces), in flight together on NVMe and virtio-blk, where they used to go one synchronous request after another. A batch of a write stays under one pointer block -- the inode's, the indirect block's, one of the doubly-indirect block's -- because moving on to the next puts the last one down after a flush, and the blocks it points at have to be on the disk by then, not waiting in the batch. A partial block is read in first, and a block the batch allocated is written even when the batch stops short, so it never holds what the disk had there before. Metadata -- inodes, bitmaps, directories, indirect blocks -- still goes a block at a time. The inode table block, the indirect and the doubly-indirect block last used are kept in memory, so a file read or written a piece at a time does not read its inode back for every piece, and a sequential pass does not re-read its indirect blocks per data block.

nanofs, ramfs, procfs#

nanofs (src/rust/fs/src/nanofs.rs) is the kernel's own small checksummed filesystem: 1024 inodes, 64 MiB of data, files up to 1 MiB, every write copy-on-write with the inode committed last. It predates ext2 write support and remains for what it is good at — a small, self-verifying store (format nanofs, mount nanofs, scripts/mkfs_nanofs.py) — and takes the whole file API, including writes at an offset and truncates, by rewriting the file. The mount reads the whole tree, by a walk with a stack of its own, so a tree as deep as the inodes allow reads back whole; a read checks the file's data against its checksum the first time after a mount or a write, not on every piece. Mounted read-only it writes nothing, not even the bitmap repair the mount makes of a reachable inode or block marked free. format and mount refuse a device smaller than the layout (69 MB) before touching it.

ramfs (src/rust/fs/src/ramfs.rs) is the in-memory filesystem the fallback layout puts on /, and what TestVfs exercises at every boot: a file is one buffer that doubles as it grows. procfs (procfs.rs) is a ramfs whose /proc/interrupts is regenerated on each lookup -- the VFS looks a file up before it reads it or reports its size, so refreshing there keeps the two consistent; /proc/version and /proc/cmdline are written once at mount.

Making a root filesystem#

scripts/mkrootfs.sh root.img 256 rootdir        # 256 MiB, populated from rootdir/
scripts/mkrootfs.sh root.img 64 "" 1024         # empty, 1 KiB blocks (what the smoke tests use)

This is mke2fs -L nos -O ^dir_index -d rootdir: the label root=auto looks for, no directory index, populated from a directory without loop devices or root privileges (inside the nos-builder Docker image on macOS, where the copy goes through a directory in the container because a macOS bind mount serves no extended attributes). Attach it to QEMU as a raw disk and the ISO boot mounts it:

qemu-system-x86_64 ... -cdrom nos.iso -drive file=root.img,format=raw,if=virtio

scripts/build-disk.sh makes nos.qcow2 the same way — one MBR partition, ext2 labelled nos, /boot/kernel64.elf and /boot/grub inside it, GRUB's boot code in the boot sector, ROOTFS=dir to add content (modules under lib/modules) — and runs e2fsck over the result. Nothing in it needs --privileged.

On a machine that dual-boots Ubuntu (the Hetzner boxes) the root is an ext2 partition made under Ubuntu, mkfs.ext2 -L nos -O ^dir_index /dev/nvme0n1pN, filled by mounting it there, and named on the nos command line by label or UUID. After a nos session, e2fsck -f under Ubuntu is the check that the driver left it as it should.

Checking it#

fstest [dir] [size] from the shell (size at least 8 KiB), or fstest=on at boot, runs The self-test (src/rust/fs/src/selftest.rs) runs in a directory it makes and removes: write and read back, append, write at an offset, truncate both ways with the gap checked for zeros, rename, move into a subdirectory, readdir, the operations that must fail, a file of size bytes (300 KiB by default) written in 64 KiB pieces and read back in 12345-byte ones, patched in the middle, cut short, grown back, and a recursive remove. fstest / 5M at 4 KiB blocks reaches the doubly-indirect blocks; the smoke tests get there at 300 KiB by using 1 KiB blocks. crc32 <path> checks a file against its copy on the host.

TestVfs in kernel/test.cpp runs the same test on a ramfs at every boot, before any disk is up.