Connect to a running nos

You can reach a running nos in four ways, and on a machine with no serial port the three network ways are all you have. By the end of this guide you will have used each one with nos running in QEMU: typed at the console, run commands over the UDP shell, logged in with SSH, and received the kernel log with netconsole.

You need:

• nos built and booting in QEMU;
• Python 3 for the two clients, scripts/udpsh.py and scripts/netconsole.py;
• for SSH, an OpenSSH client and an Ed25519 key (ssh-keygen -t ed25519 creates one). nos's sshd accepts no other key type.

The serial console#

Where the console is depends on the script:

The serial port carries the boot log from its first line. Once the shell starts, trace output stops going to the console and only the shell's own output appears there; dmesg still has the rest. console=serial or console=vga puts the shell on just one of the two consoles; the disk image boots with console=serial.

The UDP shell#

The UDP shell runs any command for anyone who can send the machine a datagram, unencrypted and without authentication. That's why it's off by default in every configuration. Use it on a VM's private network, never on a port the internet can reach.

The udpshell=PORT kernel parameter starts it. The arm64 script has a variable for it and forwards host UDP port 9000 to the guest:

NOS_UDPSHELL=9000 ./scripts/qemu-arm64.sh

Once the log shows boot: complete, connect from another terminal:

python3 scripts/udpsh.py 127.0.0.1

nos udp shell -> 127.0.0.1:9000
$ uptime

On x86-64, the disk boot's script is the one that forwards UDP port 9000. Add the parameter to the disk's GRUB entry in build/grub-disk.cfg:

-	multiboot2 /boot/kernel64.elf console=serial dhcp=auto dns=on root=auto
+	multiboot2 /boot/kernel64.elf console=serial dhcp=auto dns=on root=auto udpshell=9000

Then rebuild the disk, boot it and connect:

./scripts/build-disk.sh
./scripts/qemu-disk.sh
python3 scripts/udpsh.py 127.0.0.1

The edits on this page change tracked files (build/grub-disk.cfg, build/grub.cfg, scripts/qemu-arm64.sh), so revert them when you're done.

The client's usage is udpsh.py <host> [port] [timeout]; the port defaults to 9000 and the timeout to 30 seconds. Press Ctrl-D to quit.

• A reply holds at most 32 KB. Anything beyond that is cut off and marked [output truncated]. The kernel log holds 2048 messages, so ask for the end of it: dmesg 40.
• A command that runs longer than the timeout, such as a blkload run or a long ping, needs a longer one: python3 scripts/udpsh.py 127.0.0.1 9000 90.
• On a timeout the client prints [timeout, reconnecting] and starts over. If some datagrams of a reply were lost, it prints what arrived and names the missing chunks.
• loglevel N raises the trace level of the running kernel, which lets you see a subsystem that traces above the default level without rebuilding.

On a server set up from the system image, nos-install --udpshell[=PORT] turns the UDP shell on (see Install nos on a dedicated server).

SSH#

sshd is a loadable module that serves the same shell over SSH. It uses curve25519-sha256 key exchange, ssh-ed25519 host and user keys, and the chacha20-poly1305@openssh.com cipher. Each connection gets one session: a shell, with or without a terminal, or a single command. Everything else is refused: sftp and therefore modern scp, port forwarding, and agent and X11 forwarding. Any user name logs in with an allowed key. qemu.sh, qemu-disk.sh and qemu-arm64.sh forward host port 2222 to the guest's port 22.

Start sshd at boot#

Nothing runs insmod at boot unless /etc/rc tells it to. Each line of that file is a shell command, and the shell's task runs them once, after DHCP has set up the network. A root filesystem that holds the module, your key and that file therefore comes up with SSH running. For x86-64:

mkdir -p rootdir/etc/ssh rootdir/lib/modules
cp out/x86_64/modules/sshd.ko rootdir/lib/modules/
cp ~/.ssh/id_ed25519.pub rootdir/etc/ssh/authorized_keys
printf 'insmod /lib/modules/sshd.ko\nsshd start\n' > rootdir/etc/rc
./scripts/mkrootfs.sh root.img 64 rootdir

Boot the ISO with that image as its root disk. This is the machine the sshd test boots:

qemu-system-x86_64 -display none -m 1G -smp 4 -cdrom nos.iso -serial file:nos.log \
    -drive file=root.img,format=raw,id=drive0,if=none \
    -device virtio-blk-pci,drive=drive0,disable-legacy=on,disable-modern=off \
    -device virtio-net-pci,netdev=net0,disable-legacy=on,disable-modern=off \
    -netdev user,id=net0,hostfwd=tcp::2222-:22 -device virtio-rng-pci

When nos.log shows sshd: listening on port 22, log in:

ssh -p 2222 root@127.0.0.1

nos -- `help` lists the commands, `exit` leaves

To run a single command instead: ssh -p 2222 root@127.0.0.1 uptime.

The same directory works for the disk image: run ROOTFS=rootdir ./scripts/build-disk.sh, then ./scripts/qemu-disk.sh. For arm64, build the directory with the arm64 module and turn it into the disk the arm64 script boots:

mkdir -p rootdir-arm64/etc/ssh rootdir-arm64/lib/modules
cp out/aarch64/modules/sshd.ko rootdir-arm64/lib/modules/
cp ~/.ssh/id_ed25519.pub rootdir-arm64/etc/ssh/authorized_keys
printf 'insmod /lib/modules/sshd.ko\nsshd start\n' > rootdir-arm64/etc/rc
./scripts/mkrootfs.sh root-arm64.img 64 rootdir-arm64
qemu-img convert -f raw -O qcow2 root-arm64.img nos-arm64.qcow2
./scripts/qemu-arm64.sh

A server set up from the system image starts the same way: its /etc/rc is these two lines, and nos-install --ssh-key adds the key.

Load sshd by hand#

On a running nos, insmod loads a .ko from any mounted filesystem, and wget can download it there from a web server on your machine. QEMU's user network reaches the host at 10.0.2.2. With the UDP shell on (arm64 in this example), serve the modules from the host:

python3 -m http.server 8000 --bind 127.0.0.1 --directory out/aarch64/modules

Then, in udpsh.py:

$ wget http://10.0.2.2:8000/sshd.ko /sshd.ko
$ insmod /sshd.ko
$ sshd allow ssh-ed25519 AAAA... you@laptop
$ sshd start

insmod replies module: sshd loaded at 0xFFFF8000.... sshd allow takes the whole line of a .pub file and appends the key to /etc/ssh/authorized_keys. The first sshd start creates a host key at /etc/ssh/ssh_host_ed25519_key, and then prints what it listens on:

sshd: made a host key, /etc/ssh/ssh_host_ed25519_key
sshd: listening on port 22 -- eth0 10.0.2.15 -- host key SHA256:... (ED25519), 1 authorized keys

Check that this fingerprint matches the one ssh shows on the first connection. To start sshd again after a reboot, run rc add insmod /sshd.ko and rc add sshd start. This only helps on a root filesystem that survives the reboot; see Starting it at boot.

The sshd command#

The host key and the allowed keys last only as long as the root filesystem. On a ramfs root they're gone at the next boot, along with /etc/rc, so the host key changes and ssh refuses to connect because the key changed. For a throwaway VM you can skip the check, as the sshd test does:

ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -p 2222 root@127.0.0.1

The module must come from the same tree, or the same release, as the kernel. The loader refuses a module built against a different kernel interface (see What a module may call).

Netconsole#

The netconsole=ip:port kernel parameter streams the kernel log to a UDP collector as each line is produced. The stream starts with the first line of the boot, because lines are buffered until the network has an address. Since it needs that address, pair the parameter with dhcp=auto. Start the collector:

python3 scripts/netconsole.py

It reports netconsole: listening on 0.0.0.0:6666. Under QEMU's user network the host is 10.0.2.2, and outbound UDP needs no port forwarding, so the parameter is netconsole=10.0.2.2:6666. For arm64, add it in scripts/qemu-arm64.sh:

-APPEND="dhcp=auto dns=on root=auto"
+APPEND="dhcp=auto dns=on root=auto netconsole=10.0.2.2:6666"

For the x86-64 ISO, add it in build/grub.cfg and rebuild nos.iso:

-	multiboot2 /boot/kernel64.elf dhcp=auto dns=on root=auto fstest=on
+	multiboot2 /boot/kernel64.elf dhcp=auto dns=on root=auto fstest=on netconsole=10.0.2.2:6666

The collector prefixes each line with the time it arrived (seconds since the collector started). It reports a gap in the datagrams' sequence numbers as --- netconsole: N datagram(s) lost, seq A..B ---, and a sender that restarted as well. That way you can tell a log the network cut short from a machine that stopped. Its options:

• nctail=N sends only the newest N KiB of the lines buffered before the link came up. If a machine dies soon after its network appears, its little remaining time goes to the lines closest to the failure.
• The stream is the kernel log, the same lines dmesg holds, not the shell's echo. The netconsole shell command shows the target, how many bytes are buffered, and the drop and send counters.
• A panic report is flushed to the collector too, on a best-effort basis: only if the collector's MAC address is already in the ARP cache.
• On a server set up from the system image, use nos-install --netconsole IP:PORT.

The log goes out unencrypted, to whoever holds that address. Use netconsole only on a network you trust.

Troubleshooting#

• udpsh.py keeps printing [timeout, reconnecting]. The kernel was booted without udpshell=, the port isn't forwarded (qemu.sh forwards no UDP port), or the command takes longer than the client's timeout. For a slow command, pass a longer timeout as the third argument.
• ssh connects, but you never get a session. Check from the console or the UDP shell: lsmod shows whether sshd is loaded, sshd shows whether it listens and with how many keys, and dmesg 60 rc: shows what /etc/rc did. Without an allowed key, sshd start says sshd: nobody can log in yet, and a key that isn't ssh-ed25519 is never accepted.
• insmod refuses the module as built against another kernel interface. The .ko and the kernel come from different trees. Rebuild both, or take both from the same release.
• The collector receives nothing. The kernel has no address because it was booted without dhcp=auto, or the collector listens on a different port. Nothing is sent before DHCP finishes.
• The collector prints --- netconsole: ... lost --- lines. UDP dropped datagrams on the way. The machine didn't stop at that point.

Next steps#

• Debug nos with GDB: panics, trace levels, a machine with no serial port
• Install nos on a dedicated server, where the network is the only console
• Write a loadable kernel module in Rust
• Boot a Linux guest under the nos hypervisor: hv attach drives a guest's console through ssh -t
• The full references: UDP remote shell, sshd, Netconsole