All documentation
How it works
vnet: networks across hosts
docs/vnet.mdView on GitHubvnet.ko (src/crates/modules/net/vnet) keeps the networks a cluster's VMs
share across its hosts -- each a private subnet of its own, its ports, the
public addresses routed to the hosts, and the cluster's firewalls -- on
every host of the cluster, through Raft. hv (hv.ko) carries a VM's
packets on one: through the switch on its own host, through a sealed tunnel
to a VM of the same network on another host, out to the world masqueraded
from its own host or from a public address one to one. The design, the
decisions behind it and what each stage built are in
plans/vnet.md; this page is how it is used and what it
does.
guest - virtio-net - port -+ +- port - guest
the switch (host A) the switch (host B)
vnet.ko == the bus ==> hv: where each port is, ^
(Raft; the registry) the public addresses |
hv0 tunnel === UDP 39241, sealed === tunnel
| (keys over ipc: hv.tunnel@B)
NAT: masquerade | one to one -- eth0 -- the world
Making them
vnet stands on the cluster's registry (cluster.md): its hosts are the registry's, by node. On one host of it, the group and its members, as NBS's are made:
$ insmod /lib/modules/vnet.ko # on every host, after cluster.ko
$ vnet init lab
vnet: lab made
$ vnet host add b
vnet: b is member 2
$ vnet host add c
A network -- a private subnet of /16 to /28, apart from every other network of the cluster's and holding no host's own address, its first address the gateway on every host:
$ vnet create lan 10.10.0.0/24 mtu=1400 # 1440 at most: the tunnel adds 60 bytes to a packet
$ vnet create dev 10.20.0.0/24 out=no # no way out to the world
$ vnet list
A VM on one
A VM is on one network, chosen as it is made, written network = "vnet:<name>" in its vm.toml -- and nothing else about it: its port, its
address and its firewall are vnet's.
$ hv create web /bzImage net network=vnet:lan address=10.10.0.50 firewall=web
hv: web given port 4 on vnet:lan at 10.10.0.50, judged by its firewall web
$ hv start web
$ vnet ports # every port, its network, address and host
$ hv delete web # its port deleted with its folder
- The port is made at
hv create, or at the VM's first start, and attached at each start with a read the Raft leader vouches for -- up to five seconds for one; with none, a port the host's own state binds here starts unconfirmed, said inhv list, and is asked again until it is confirmed. - The guest is told its address, its gateway and the MTU on its command
line (
ip=), by DHCP (option 26 for the MTU) and by its NIC (VIRTIO_NET_F_MTU). Its MAC is02:00and its address's four bytes. - On a network the switch carries IPv4 and ARP alone: ARP and DHCP
answered by the switch, nothing broadcast, another network never reached,
a packet past the MTU answered "fragmentation needed" from the gateway.
hv listcounts what each port's VM sent that went nowhere. hv probe add <name> network=vnet:lanmakes a port with no VM behind it, which answers ARP, echo and UDP to port 7 and pings from where a guest would be: what a network is checked with before any guest boots.
Firewalls
Every VM's NIC is judged at its port by one firewall: what comes in denied
unless a rule lets it, what goes out allowed, the answers to a VM's own
flows let back. A VM on a cluster network is judged by a firewall of the
cluster's, kept in vnet's state -- default unless it names another --
written and read as hv's own are (The firewall):
$ vnet fw default add allow in from net # a network's VMs reach each other
$ vnet fw create web
$ vnet fw web add allow in tcp 80,443
$ vnet port firewall 4 web # port 4 judged by web from the next table on
A change of a firewall's rules, and a port moved to another firewall, reach every host within a second of being made -- hv's keeper asks vnet's table every second -- and the flows the new rules deny are cut.
A firewall's policies are what a packet no rule matches meets, one each
way -- in denied, out allowed, unless changed; its rules are the
exceptions, the first that matches deciding. In the panel, a host's page of
VM networks has the cluster's firewalls -- a rule added, the policies set,
from a firewall's row -- and their rules in a table of their own, each row
taken out or moved from there (the vnet provider's rules).
Across hosts
A VM reaches a VM of its network on another host as it reaches one on its own: the switch carries the packet into the tunnel, sealed into a UDP datagram to the other host's address, opened there into the port's inbox (Across hosts: the tunnel).
-
Where a host is: the registry's address for it -- its
underlayone, or itsipcone where it has none -- at that address's port where it has one of its own, else UDP 39241. A host behind a NAT is reached at the NAT's address and the port the NAT forwards to it; two behind one address have ports of their own, the ones their ipcnet has already, and the NAT forwards each in UDP as it does in TCP:$ cluster invite b 203.0.113.7:39241 <key> # b and c behind 203.0.113.7: 39241 -> b, 39242 -> c $ cluster invite c 203.0.113.7:39242 <key> # TCP for ipcnet, UDP for the tunnel $ cluster address b underlay 198.51.100.4:40000 # or the tunnel elsewhere than ipcnetA datagram is known by its key, never by where it came from: whatever address and port the NAT gives it, it is its sealer's. A host's address changed in the registry reaches every tunnel with vnet's next table.
-
Sealed with ChaCha20-Poly1305, each host giving every other the key to seal to it with, over the message channel (
hv.tunnel), made anew for every message, given anew everyrekey=seconds (3600) and the one before keptkeylife=(120):insmod /hv.ko rekey=S keylife=S. A counter seen before, a key not given, a tag that does not hold, a packet from a port the receiver's table has elsewhere: dropped, counted. -
An echo every five seconds to each host, every third one as large as the networks' largest MTU: a host that stops answering is said in the log and its key asked for anew; a way that loses the large echoes and not the small ones is said too -- an underlay short of the networks' MTU.
-
hv tunnelsays each host -- its next hop, its keys, when it last answered -- what this host serves, and what was sent, taken and dropped, and why;trace on tunnelrecords each drop and each key (tracing.md).
Public addresses
A public address is routed by the provider to one host of the cluster's, and may be put on the port of a VM on any host: the world reaches the VM at it, and the VM goes out from it, one to one -- the guest sees its private address alone.
$ vnet ip add 203.0.113.9 a # routed to a by the provider
vnet: public address 203.0.113.9, routed to a
$ vnet ip set 203.0.113.9 4 # onto port 4, wherever its VM runs
vnet: 203.0.113.9 on port 4
$ vnet ip move 203.0.113.9 b # the provider routes it to b now
$ vnet ip set 203.0.113.9 none # off the port: it goes out from its own host again
The host it is routed to serves it: NAT maps it one to one to the port's address, ARP on its uplink answers for it, and the world's packets go through the tunnel to the VM's host; the VM's packets for the world come back the same way and leave from the address (A public address across hosts). A VM with no public address goes out masqueraded from its own host. A public address on a network that does not go out is not served.
When things fail
| What happened | The packets | What is seen |
|---|---|---|
| vnet has no leader | Go on by the last table; a port bound here attaches unconfirmed | "unconfirmed" in hv list, asked again until vouched for |
| vnet.ko not loaded, or unloaded | VMs running go on by the last table; a new one on a cluster network refused | "vnet is gone from this host" once, in the log |
| A port moved, deleted, or of another generation | The VM keeps running, its port shut: nothing to it or from it | hv list says why |
| Another host rebooted, or its hv loaded anew | New keys both ways; seconds of loss | hv tunnel: the keys, the echoes |
| A host silent | Nothing to its VMs | "answered no echo", in the log, and its key asked for anew |
| The underlay does not carry the networks' MTU | The large packets lost | "does not carry a packet of N bytes", in the log |
| Two hosts at one address and port -- behind one NAT, with no port of their own | What is sent to one reaches whichever the NAT forwards it to | "both at ...: two hosts behind one address need ports of their own", in the log |
| A public address's host down | The world does not reach the VM at it | vnet ip set ... none, or vnet ip move once the provider routes it elsewhere |
Limits
IPv4 and ARP alone on the cluster's networks; no broadcast between ports; one network and one NIC a VM; a packet goes to the port it is addressed to and no other, so a VM that routes for its neighbours does not work. TCP crosses the tunnel a packet of the MTU at a time. What is left -- a move refused while its VM runs, the alerts of a public address's host -- is in plans/vnet.md.
The gates
tests/hv-fuzz.py(host, in CI):vnetswitch(the switch's rules for the cluster's networks, held to a model of its own),vnetwire(the datagram -- every one held to the wire built without vnetwire -- its window, the keys' messages),tunnel(three hosts' real switches and tunnels over a wire that loses, repeats, damages, reorders and forges, public addresses through the real NAT);--mutants: ninety-nine rules taken out, each to be found.tests/raft-test.py(host, in CI): vnet's state (vnetmeta).tests/vnet-test.py(by hand, ontests/hub.py):--phase v1bthe group and its state on three hosts,v2one host's networks with probes,v3the tunnel between three,v4public addresses across them (testing.md).