All documentation
How it works
NBS: replicated block storage
docs/nbs.mdView on GitHubNBS is the replicated block storage for VM disks, images and snapshots planned in plans/nbs.md. As built (steps N5 to N8), it is two modules on every host, and a client:
-
nbs-mds.ko(src/crates/modules/storage/nbs-mds), its metadata service: the state every host of an NBS cluster holds alike through Raft --- the NBS cluster's identity and its members;
- each host's chunk server and its devices;
- the volumes, and on which three hosts each of their chunks is;
-
nbs-cs.ko(src/crates/modules/storage/nbs-cs), its chunk server (cs): the host's devices, formatted as NBS's own, and the chunks on them -- written, read, fenced and freed as the metadata says ("The chunk server", below). -
the client (
src/crates/common/storage/nbsclient, no kernel in it): a volume's session -- attached, its cs's fenced, its open chunks reconciled, read and written through the cs's, flushed, closed, detached -- driven over ipc bynbsdrive(src/crates/modules/storage/nbsdrive) in the modules that use a volume: hv.ko'snbs:disks,nbs io, nbsload.ko ("The client", below).
The state is nbsmeta::State (src/crates/common/storage/nbsmeta, no kernel
in it), the state machine of the raft crate's rsm (raft.md). The
member is the one the cluster registry runs too, raftmember
(src/crates/modules/cluster/raftmember), with NBS's hooks. Its messages go over
the message channel between hosts, ipcnet (ipc.md), to nbs.mds on
every member.
Using it
The hosts first trust each other: ipcnet and the cluster registry (cluster.md), which pins every host's key on every other. Then on each host:
$ insmod /lib/modules/nbs-mds.ko # dir=/nbs/mds unless dir= names another
$ insmod /lib/modules/nbs-cs.ko # after it: the cs asks this host's mds what it is to take in
compact=<bytes> is how much log the member keeps past its snapshot before
it takes a new one. It is the state's size if that is more, and raft's
1 MiB unless given ("The snapshot", below). out_after=<seconds> is how
long a cs is silent before the leader's controller has it out and its
places taken by others: ten minutes unless given (Repair).
On the first host:
a$ nbs init store
nbs: NBS cluster store made (53f0...), a its first member (1730...) -- `nbs host add` the next
On any member, a host of the registry made a member:
a$ nbs host add b
nbs: b is member 9512... -- a learner until it holds the log, then a voter while fewer than five vote
And then:
nbs this host's member: incarnation, role, term, leader, what it applied, members and voters
nbs hosts every member: its incarnation, voter or learner, the leader, a member that lost its state
nbs host remove <host> <host>'s member out, for good: its incarnation never taken again
nbs host replace <host> <host>, which lost its state, back as a new member
What is stored, on any member:
nbs devices add <id|name> [wipe] a device of this host's for its chunk server, by its id or its name
nbs devices drain <id> the device takes no new chunks
nbs replica bad <set> <host> <host>'s cs taken down in <set>, to join its place again and copy it anew --
the member the scrub found unlike the two others
nbs cs every host's chunk server: its devices, its room, what of it is reserved
nbs volume create <name> <size> [host=<host>] size in MiB, or with M, G or T after it; its own host this one
nbs volume resize <name> <size> grown, never made smaller
nbs volume delete <name>
nbs volume snapshot <volume> <name> attached: on its writer's host, which pauses it
nbs volume freeze <name> an image: read-only for good, cloned from
nbs volume clone <image|snapshot> <name> [host=<host>] a volume of its, thin
nbs volumes every volume, and what each is: a volume, a clone of, an image, a snapshot of
nbs volume <name> one, and the hosts its replicas are on
For example:
a$ nbs devices add part:4f7661ab-0525-4f56-9d6b-9283d322e47c
nbs: device part:4f7661ab-... of this host's chunk server, 292 MiB -- device 3 in the metadata; its cs takes it in
a$ nbs volume create disk1 64
nbs: volume disk1 (1), 64 MiB
a$ nbs volume disk1
nbs: volume disk1 (1), 64 MiB, 16 chunks of 4 MiB, its own host a, detached
replicas, chunks a host: a 16, b 16, c 16
nbs devices add finds the device on this host by its id or by its name. It
records the device's id, never its name: names change between boots, ids do
not. Refused:
- a device with no id;
- one that something holds at the time: a mounted filesystem, the disk log, a guest of hv's;
- one with something on it that is not this NBS cluster's -- anything at all
in its first two pages but zeros or a cs's superblock of this cluster: a
filesystem, a partition table, a boot sector, another cluster's cs.
wipetakes it all the same: its first 8 KiB are zeroed once the metadata has it, and its cs formats it.
A device added again as it is -- the one its cs holds -- is answered with its number, nothing read or written.
The same is on the management bus as the nbs provider (provider.rs,
nbs.descriptor.json), which the panel shows:
- to read:
status,hosts,cs,devices(every host's chunk server's devices: their ids, sizes and states),volumes,volume-- each volume with itskind(volume,imageorsnapshot) and itsorigin, the image or snapshot a clone was made of and the volume a snapshot was taken of, by name while that is there; and itshealth, the worst of what its chunks say (nbsmeta::repair::volume_health, asnbs statuscounts the cluster's):failedwhere a chunk has no active replica, or the scrub found its replicas unlike in a set as it still is -- known on the leader's host alone, its newest findings alone --criticalwhere one has a replica alone,degradedwhere one has two or is being copied onto a joiner, elseok; with the counts behind it (chunks_at_two,chunks_at_one,chunks_unreachable,chunks_copying,chunks_unlike). The panel colours it by the tones the descriptor gives it -- yellow, orange, red, green -- and a VM's disk on the volume by the worse of it and the guest's I/O: red while its reads and writes fail, yellow while they wait past 50 ms on the average; - what a person should look at,
alerts, the worst first -- or one row saying all is well: this host's member stopped, no leader (no quorum: nothing changed or repaired, the volumes attached going on), chunks with no replica active, replicas the scrub found unlike, a device failed, fewer than three hosts with room (no new chunk, no repair), chunks with one replica, chunk servers down or out, chunks at two or being copied, a chunk server under a tenth free, a host the registry has in maintenance holding replicas -- asked of theclusterprovider through the bus, as NBS's controller does not read the flag yet -- and a device draining; each with its host, its detail and its count where it has one, its tone the descriptor's. The panel shows it first on NBS's page, and a VM's disk on a volume of NBS's yellow while there is no leader; - to change:
volume_create,volume_delete,volume_resize, andvolume_snapshot,volume_freeze,volume_clone(N10d) -- a snapshot of an attached volume asked on its writer's host, as the command is; - the cluster itself, as
nbs initandnbs hostmake it:init, an NBS cluster of this host alone;host_addandhost_replace, jobs -- an offer is waited on for up to 30 seconds, and given up when the job is cancelled;host_remove, the host's name typed again in the panel; - the devices:
device_add, a device of this host's to its chunk server, withwipeas the command's;device_drain, of any host's (host, this one where none is given) -- a change to the metadata alone, its device taking no new chunks until it is added again.
Reads are this member's view, at the entry it has applied. Every change is a command through the leader, answered once this member has applied it too: what is read next on the same host holds it.
What it keeps
-
The NBS cluster: its id, 128 random bits made at
nbs init, and a name. Every message of the member's carries the id before its bytes, and a member takes none of another cluster's: dropped, counted, said in the log once in ten seconds at most. -
Its members, in the order they joined: a host's ipc name and the incarnation its mds made when it was offered the cluster -- 64 random bits, never 0, the member's raft node. The first five to join vote; the rest are learners, which hold everything and do not vote.
-
The members removed: an incarnation removed is never taken again, whatever an old copy of the list says.
-
Each host's chunk server (cs), kept by the host's name. A member replaced keeps its host's cs, and a member removed leaves its cs taking no new chunks. Its devices:
- each by its id -- a GPT partition's GUID, an MBR partition's signature and number, an NVMe's serial;
- with its size in 4 MiB slots, after what the cs keeps of its own at the device's start: a slot for its superblock, then its table of slots, twice;
- active, or draining: taking no new chunks. What a drained device holds is to move with N9; added again, it takes new chunks again;
- the format its cs found it at -- the 64 random bits of its superblock --
once its cs has proposed it (
DeviceFormatted). A device its cs reads blank where a format is recorded is lost, not formatted again: wiped or rotten, what it held gone.nbs devices add <id> wipezeroes its first pages and drops the record, and its cs formats it anew (the review of 2026-11: formatted, it served its replicas as zeros where they held data, and nothing said a thing).
A cs's room is the slots of its active devices. Reserved of it is a slot for every chunk on it, active or draining, and for every chunk of a set it is joining in another's place (N9). A cs is up, down or out as the leader has committed it (Repair): only an up one takes new chunks.
-
The volumes:
- a number never taken again;
- a name -- lowercase letters, digits,
.,_and-, up to 63; - a size in MiB, up to 16 TiB;
- its own host: where a replica of every chunk goes, and where its VM is to run;
- its attach: a writer and its generation, or readers. A generation is the cluster's next, never given twice, whatever the volume: a volume made again under a name refuses what a writer of the one before asks (the review of 2026-11: its generations began again at 1, and a late snapshot, detach or attach anew of an old writer's was taken as the new one's);
- the tokens of the last four sessions that let it go: an attach of one of those, applied late, is refused as stale rather than attach the volume to nobody;
- its map: a chunk for every 4 MiB;
- its kind (N10): a volume, an image -- frozen read-only, to be cloned -- or a snapshot, a volume's map at a moment; and what it was made of: the volume a snapshot was taken of, the image or snapshot a clone was made of.
-
The replica sets: three cs's on three hosts, shared by every chunk placed on the same three, so that a host lost changes the few sets it is in and not every chunk it holds. Each member is active or down -- a down one with the last version it was active at -- and a set has at most one joiner, a cs being copied onto (Repair). New chunks go only into a clean set, every member active and no joiner; an empty set is made again for three new cs's before a new one is made -- never the clean set of three cs's the same placement fills, so that the sets a placement takes are counted before it takes any.
-
The chunks: the set each is in and how many maps name it -- 12 bytes of memory a chunk, and 4 more a place of a map; whether a chunk was written is a flag the snapshot's format keeps and no command sets yet. A chunk freed goes onto a free list and is taken again first.
-
The copy-ups pending (N10): by a volume and a place in its map, the new chunk a write to a shared chunk is to go to, and the old one the map still names.
Snapshots, images and clones (N10)
A snapshot or a clone shares the chunks of what it was made of: its map a copy of the other's, every chunk counted once more, and no room reserved -- thin (decision 5). The metadata's commands, as the shell and the client are to drive them (N10b-N10d):
Snapshot { volume, name, generation }-- of a volume only, at the generation of the writer that paused for it, or 0 for one with none: a writer not paused could have a write on its way into a chunk the snapshot shares from then on.nbs volume snapshot <volume> <name>asks it (N10c2): of a volume with no writer, as it is; of one attached, only on its writer's host -- elsewhere refused, naming that host -- whose mds pauses the writer's session first (Said::Pause): the session sends nothing new, closes every chunk it wrote -- each once the writes out to it are done -- and answers (Ask::Paused); the snapshot is proposed at its generation, and once it is applied the session is told to go on (Said::GoOn) and reads its map anew -- every chunk shared -- before it writes. A pause not answered in 5 s is given up, the session told to go on, and the snapshot refused. A session that answered and hears no word in 10 s attaches anew (Reattach): a snapshot of its pause applied late is refused then. The mds tells a session to go on only where it never proposed, or once the proposal was applied -- one that may still commit leaves the session to its own wait. A stop asked of a session while it is paused is begun once the pause is over: nothing is let go of while its snapshot may still be applied. A sessionnbs iokeeps on the host is driven by the snapshot's command while it is asked there, as a guest's disk is by a task of its own: left alone between its commands, it would not pause.Reattach { name, host, generation, token }-- the writer's own attach anew (N10c2), at a new generation: refused where the volume's writer is no longerhost's atgeneration-- taken over since, on that host or another -- and asked again by its token, answered as it was.Freeze { name }-- a volume made an image, refused while attached; its copy-ups pending given up.nbs volume freeze <name>asks it (N10d).- A snapshot or a clone asked again under a name taken is answered as done
by the metadata where it is of the same origin -- a proposal applied
twice is one ask -- and refused by
nbs volume snapshotandnbs volume clone, and their providers' calls, before it is proposed: a person or a tool asking again would take it for a snapshot of what the volume holds now, which it is not (the review of 2026-11, its question 3). Clone { from, name, host }-- of an image or a snapshot only (elseNotFrozen).nbs volume clone <from> <name> [host=<host>]asks it (N10d), this host its own where none is given. A clone's writer reads and writes chunks recorded as another volume's on their cs's, each asked of the mds first; a write it makes in place -- its image deleted, the chunk its alone -- makes the chunk its own. And a chunk a cs found open is refused, as found open, to its writer alone: the volume's as well as the generation's, as generations are counted by volume, and a clone's writer may be at the very one another volume's left a chunk open at.CopyUp { name, place, generation }-- the writer's, before it writes a chunk others share: a new chunk in the same set, so that each of its cs's copies it locally; its room taken on each, refused asNoRoomwhere one has none left -- the room kept to rebuild a host may go to it. Pending, the old chunk named by the map still, so that nothing can free it while the cs's copy it (Op::CopyUp, below). Asked again, it is answered the pending chunk; of a chunk shared no longer, the chunk itself. Refused asJoiningin a set being joined (N10b): a joiner lists what its source holds once, and a new chunk behind its listing would never reach it -- so a join gives up its set's copy-ups pending as it begins, their writers ask again once it is over, and what the cs's copied for them is freed as any chunk gone. A join still needs room for every chunk of the set, those among it: the controller plans none with less. Refused asDrainingin a set with a member drained (N10c): its devices take no new chunk, and its place is to be joined by another cs first. A copy-up pending of a chunk named by one map now -- a snapshot dropped meanwhile -- is given up (N10c): one left would hand its new chunk, copied as the old one was then, to a copy-up asked after the place was written directly.CopyDone { name, place, generation, chunk, incarnation }-- the map names the new chunk, the old one counted down, freed at none. Named by its number and its incarnation both: a number is taken again once its chunk is freed, and a copy-up given up -- a join of its set begun -- and asked anew by a late copy of the writer's ask can be given the number it had. Named by its number alone, the end of the old one's, late too, took the new one's: the map named a chunk nobody had copied into, zeros where the volume had data (found by the cluster target, N13c).
A volume deleted, or an image, counts its chunks down: those a snapshot or a clone still names live on. A chunk's count is a u16; a map names a chunk once at most, and there are fewer volumes than it holds.
On the host, in /nbs/mds:
member-- which member of which NBS cluster this host is: its incarnation, the cluster's id, its own name and the cluster's on the first line; for a host that took an offer, a line for each member it was offered by, which it knows the members by until its log has caught up. Written once and never written over: the mark that this host is a member.raft/-- the member's log (raft::log::DiskLog): its segments, its hard state, and its newest snapshot,snap.<index>.begun-- the mark that the member began its log, written and synced once its first segment is on the medium, before the member runs: amemberandbegunbeside araft/that holds no log is a log gone, and the member is not started as the one it was ("A host that lost its state", below).
Where replicas go
A volume's chunks are placed in the apply of the command that makes or grows
it, from the committed state alone, so every member places them alike
(plans/nbs.md, "Where replicas go"; nbsmeta's place.rs):
- three replicas on three hosts;
- only on the cs's of members that are up, on their active devices, where there is room;
- one on the volume's own host, where it has room;
- the other two on the hosts with the most room left, ties broken by a hash of the volume, the chunk and the cs;
- room reserved at create: a volume takes its whole size, written or not, and one that does not fit takes nothing;
- with four hosts or more, a volume refused where it would leave the others less free room than the biggest host's replicas take, so that a host lost can be rebuilt on the rest.
A volume that cannot be placed is refused, and why:
- fewer than three hosts take new chunks at all:
fewer than three hosts take new chunks; - three or more do, but are too full:
not room for its chunks on three hosts; - rule 6:
it would leave too little room to rebuild the biggest host.
Repair
How three are kept (plans/nbs.md, "How three are kept"). What N9a has of it is the metadata and the controller; the copies are N9b's, the leader's turns N9c's.
- A cs's state.
CsDown-- silent 5 s to the leader -- has it down in every set it is an active member of, the last active member of one apart, and drops every join it is the joiner of: writes go on to the others. Its version at the time is kept with each place, as the last it was active at.CsOut-- silent pastout_after, 10 minutes held off by the registry's maintenance flag, or its host removed -- is down for good: its places are to be taken by others.CsUpis a cs that answers again; its places in its sets stay down until it joins them again. - A join (
Join) puts a cs into a set as its joiner, to take one place once its copy is done: its own place, the member down and back, which copies only what it missed; or another's -- a member down and out, or one whose cs is drained, which stays active meanwhile -- with room for every chunk of the set. It names the set's version it was planned at, and is refused once the set has moved on.Joinedmakes the joiner the member in that place, active, the room of the one it replaces given back;Unjoingives the join up. Each raises the set's version: a chunk's epoch moves on with it, so the writes of the epoch before are refused at the cs's that know the new one (rule 3 of plans/nbs.md's "Replication: the rules"). - The joiner is written, not read: a client writes it with every member
active, and reads only from active members; the host's mds vouches it for
writes alone. Silent to the writer, it is asked out as a member is
(
ReplicaDown), after the same look by the leader, and its join dropped. - The controller (
nbsmeta::repair::plan) is a pure function of the committed state and what the leader sees of each cs -- answering, for how long not, in maintenance -- the same in the member as in the fuzzer. Its turn proposes the cs's to go down, up or out; where none is to, joins: the sets with one active member first; a member down whose cs is up again joins its own place; one gone, or drained, has another join in its place -- the most room first, ties by a hash of the set and the cs; two joins a joiner and sixteen a turn at most. A set is rebuilt whole. A join the leader has seen stand past its time -- half an hour, and a second more for each chunk of the set, 4 MiB at 4 MiB/s -- it gives up (Unjoin), and says so: a joiner that answers the leader's look but never ends its copy -- its host's mds gone, which a copy asks its end of -- held its set from any other join and every copy-up of its chunks for good. The time counts from when this leader first saw the join, so that a new leader gives each its whole time again (the review of 2026-11, its question 2). A copy that cannot begin -- its volumes' fences never laid on the joiner -- its cs gives up itself after five minutes, as a copy with no answer it can use. - Seen:
nbsmeta::repair::healthcounts the chunks by how many of their set's members are active, and those being copied onto a joiner;nbs statussays them, with the joins under way and the cs's down and out. - In nbs-mds (N9c) the leader looks at every cs once a second
(
Op::Ping) and plans every ten seconds -- sooner, a second after its last plan at the soonest, where the state moved on (a cs changed, a join done or dropped) or a cs went silent or came back -- proposing what the plan gives once what it proposed last is applied. After a proposal of its is refused it keeps to the ten seconds, saying each refusal. A cs's silence counts from when this leader began to watch it: a new leader takes none for down at once. The registry's maintenance flag is not read yet.
The copy
A joiner's cs copies the set (N9b): nbswire's copy, driven by nbs-cs's
task, whose mds tells it its joins (Question::Joins) with its devices.
- From a source, an active member: the set's chunks it holds a page at
a time (
Op::Holdings, 4,096 a page), each pulled (Op::Pull) and summed (Op::Sums). Each names the joiner and the set's version, which the source's mds vouches for (Question::VerifyJoin). A pull and a sum move the chunk's epoch on at the source, durable before they read: a write of the epoch before is refused there from then on, and its writer, its map read again, writes the joiner too. An epoch older than the one a chunk is known at is refused even where the cs's cache once vouched for it. - The listing waits, and is whole. A source lists the set's chunks only
once every request it took before the listing has ended
(
Requests::mark,ended_before): a chunk whose first write was still on its way into the source was left out, and the joiner made a member without it, reading zeros where the set had data. And only while it holds all it was given (Store::whole): a source with a device failed, or a record lost, lists, pulls and sums nothing -- its list left out what it had lost, and the copy was done at once (both found by the review of 2026-11). - A chunk found open is no source: restarted with it open at a writer's
generation, the cs may have lost what that writer wrote since its last
flush, which a joiner would take as the set's. Its pulls and sums are
refused as failed -- the copy takes another source, or waits for the
writer to take this one out, or for a writer's reconcile to make the
chunk the same, which ends the mark -- where the cs is its set's last
active member, its writer's next request there fails as the last replica
refused (
LOST_LAST), and the next writer's start reconciles it. Refused, they keep the mark too: a pull's moving the epoch on writes the chunk's record, which would take it off. (Both found by tests/fuzz/nbs's cluster target, N9d: a joiner copied from such a replica, and a pull from one had taken the mark off.) - The marks: what clients write to the set from the join on is marked,
a bit a sector, in memory alone, from the first write the mds vouches for
as the joiner's (
Told::VerifiedJoining). The copy writes only where no sector is marked: the client's data is newer, and the source has it or will. A joiner takes no write of an epoch before its join: one from a view that old -- a writer taken over since among them, which the members fenced against it refuse -- would be marked, and kept over the set's (found by the cluster target, N9d). Its writer reads its map again. - The joiner fenced first (N13): its mds gives each join with the
newest generation of every volume a writer has had with a chunk in the
set (
JoinView::fences), and the joiner's cs fences itself so, durable, before its join's marks begin and again as the view is refreshed (copy::fence_for) -- a fence it cannot have keeps the join from beginning. Down when the volume's writer of now fenced the set's members, it holds the fence of the one before; and that one, taken over, its map read anew -- a view as new as the join -- would write it there alone, unasked, at the epoch the copy moved its chunks on to, the members refusing it (found by the cluster target, N13). - A member back pulls whole what it may have missed -- a chunk last written at the version it went down at or after, or one it does not hold -- and holds the rest to the source's by sums, copying only what differs: one it found open when its devices were taken in among them, a sum finding what a power cut lost as it finds a write missed. It trusts no chunk it holds: taken out, it kept each at the epoch it last knew, and a writer since taken over, its view as old, could write it there unfenced, which the members fenced against that writer refused (found by the cluster target, N9d; a touch, which moved the epoch on and kept the data, did before). And as the copy closes a chunk, the joiner knows it at the copy's epoch from then on, as the source does once pulled or summed -- those the sums found the same among them: kept at the epoch it last knew, such a chunk took a write of that writer's once the join was done, the member active and its new writer's fence never come to it (found by the cluster target, N13: a chain passed the write on; sent directly, as before N13, it would have been taken the same way).
- The check: a chunk pulled is summed against the source's. A granule that differs is pulled again and held to the source's a sector at a time, the marked sectors left out; a sector that does not read back as copied is a round, and past three the join is given up and said. The chunk is then closed, its data flushed first -- flushed too where a client's write left it open at the writer's generation, which leaves it open: the join's end makes this cs a member, and a power cut after must not take the copy's sectors (found by the cluster target, N9d). A chunk the joiner holds open from before its join -- a member then, taken out with the chunk open, its writer closing it on the members it had left -- is nobody's but the copy's: the copy closes it at its end, whether it wrote the chunk or found it the source's already, and leaves a mark open only where a client has written the chunk since the join began (found by the cluster target, N9d: kept, it made the chunk found open once the joiner crashed, and as its set's last member it was no source for the joiner after it).
- The source's open mark kept. A chunk the source holds open as it sums
it -- its writer's writes out, or the writer lost with some of them on
some replicas alone -- the copy leaves open on the joiner at the same
generation (
Op::Sumsanswers it with the sums), unless that writer, or one after it, has closed the chunk on the joiner since the join began, which the joiner keeps in mind as it keeps the marks. An open chunk's replicas may differ, and the mark is what has the next writer's reconcile make them the same: closed by the copy, the source taken out later took the last mark with it, and two active replicas stayed apart for good (found by tests/fuzz/nbs's cluster target at seed 14649, past the gate's range, after the review of 2026-11 -- its question 5). So that the writer's own close comes to the joiner too, a close is held to its epoch as a write is: one by a view older than the chunk's -- the copy has moved it on -- is refused, and its writer, its map read again, sends its closes out to every replica the view has now (closes_to_view); one by an epoch the cs does not know, or of a chunk it holds none of in a set it joins no join of, is done only once its mds vouches for it -- a joiner learns its join so before the close counts. The cluster target holds every chunk whose active replicas are apart to a mark where the next reconcile asks (marked_where_apart), each time a set changes. - What the source has none of is dropped as the copy ends: a chunk of the set the joiner holds at an epoch before its join that no client has written since -- a member back's, written by a writer cut off with it, which never reached the others. Kept, it came up as the set's once the member was active again, and its next writer's reconcile could copy it onto every member (the review of 2026-11).
- Its end is asked of the mds (
Question::JoinDone, proposed asJoined); a copy that fails gives the join up (Question::GiveUp, asUnjoin). A join found as nbs-cs starts is given up -- asked again until the giving up is applied, and never copied meanwhile: its marks are gone, and a copy without them could write over a client's write. The controller joins it again under a new version. - A chunk found open is written by its set's joiner, as a joiner is
written and not read, and its copy pulls the chunk; refused to its writer
as
Failedonly where its cs is an active member.
The scrub
The leader looks at every chunk in turn, a chunk every scrub= seconds (ten
unless nbs-mds is given another, 0 for none), and holds its set's active
members alike (nbswire::scrub, the one nbs-mds runs and the cluster target
does):
- The look: each active member asked for the chunk's record and the
CRC-32C of each of its 64 KiB granules (
Op::Scrub) -- read as a joiner's sums are, but moving no epoch on, so that a writer is refused nothing. A chunk open on any member is in flight and is left for a later round, as is a set that moved on between the asking and the answers. - A member that holds none of it says so, and is compared as what such
a chunk reads as: zeros (
nbscs::zero_sums, a slot being zeros before a chunk takes it). A member that lost a chunk its set's others hold differs from them as a rotted one does. A chunk no member holds was never written: nothing is read, and the next chunk is looked at a hundredth of the pace on -- the pace is the reads', so that a volume mostly unwritten takes a round little longer than what is written of it, its empty chunks asked about no faster than that. - Two looks: two members' answers are not taken at one instant, and a write and a close between them would show a difference that is none. A difference is a finding only when a look two seconds on finds every member's answer as the first did.
- What it finds it says, in the log and in
nbs status-- the chunk, its set and version, and the member unlike the two others where two agree, with the command that takes it down; a finding of a set that has moved on since is shown as history -- once for the set's version, and takes nothing down. A cs keeps no checksum of its own data, and two members alike are no proof which one holds it: a joiner that copied a source that had rotted makes two alike in rot, and a scrub that took the odd one down would take the one good copy down, its join back copying the rot over it (found by the cluster target, N9e). A person decides:nbs replica bad <set> <host>takes that member down (ReplicaBad, at the set's version now; refused for a member down already, a set with a joiner and the set's last active member), and the controller has it join its own place again, a member back whose copy holds every chunk to a source's by sums and pulls what differs.
nbs status on the leader counts what its scrub has done since it began
to lead -- chunks found alike, held by none, let be (open, moved on, an
answer late or refused, written between two looks) and unlike, those with
a member unlike the two others among them -- and lists its sixteen newest
findings.
The digest
A member that applies an entry differently from the others is a fork that nothing else would notice. So:
- every 64 entries -- or, for a state past 16 MiB, every as many entries as it has 256 KiB: a state of 1 GiB once in 4,096 -- each member takes the SHA-256 of its snapshot's bytes: the state's tables, in a fixed order;
- the leader proposes its own, as an entry like any other, and every member holds its own to it;
- a member whose own differs stops, and says at which entry and what the two digests were.
The leader's digest is what every member is held to: where the leader's own
state is the one gone wrong, every right member stops, and the cluster with
them -- the leader alone is no majority, and commits nothing more. That is
the design: a fork is stopped loud, and nothing a wrong state answers goes
on into the log; which member is wrong is left to whoever reads the two
digests, and a member started again from another's snapshot (nbs host replace) is the way back. A vote of every member's digest, which would stop
the leader alone, is not made (the review of 2026-11, its question 1).
nbs shows the newest:
nbs: the state's digest at entry 64: 694d3a0bd0f289ec... -- 1 of the leader's held to it, 0 too old to
A digest is the member's own, never in a snapshot. A member started again from a snapshot has none until the next 64th entry.
The snapshot
- When: once the log past the last snapshot is as large as the state,
1 MiB at least -- raft's floor, which
compact=changes. - Written as a stream. The tables are kept in pieces of 1024 records, each shared copy-on-write. A snapshot takes the pieces' handles at its index and writes them out a piece at a time; the apply goes on meanwhile. The state is never in one buffer, which the kernel's heap would refuse past 512 KiB.
- Read back as a stream too, and held to every rule applying keeps: every count, the free list walked, each cs's reserved room. A snapshot that breaks one is refused.
- Sent whole to a member added after the log it needs was compacted, as raft sends any snapshot.
How a host becomes a member
The registry says which hosts may talk at all. ipcnet delivers nbs.mds --
a Cluster name -- only between hosts it has pinned, so a host the registry
has is all nbs host add needs:
- The member the command is typed at offers the cluster to
nbs.mds@<host>. The offer carries the cluster's id and name and its members. - On that host the mds, in no cluster, takes it. It makes an incarnation no member of the offer has, clears its log folder, writes its member file and answers with the incarnation. Then it starts its member over the empty log.
- The command commits
Add { host, incarnation }. The leader puts the incarnation into raft's membership as a learner, raft brings it the log, and once it holds the log the leader makes it a voter, if fewer than five vote. Until its log has the members, the new member checks senders by the members the offer named.
An add asked again is harmless. A host that is a member already answers the
offer with the incarnation it is, and Add of the same incarnation is done
again. A host that is a member of another NBS cluster declines.
A host that lost its state
A host whose /nbs/mds was lost -- a disk replaced, an install wiped --
comes back with an mds in no cluster, and must never be counted as the member
it was: Raft would count the vote of a member that forgot what it voted for.
So:
- Its mds, in no cluster, answers every raft message for a member it does not
have with "no such member" (at most every 5 s a host). The leader takes
that from the host the member is on alone. It says so in its log once a
minute and in
nbsuntil the member is replaced:a says its mds has no member 1730...: its state is gone -- nbs host replace a takes it back as a new member. nbs host addof it is declined by the host itself, which sees in the offer that the cluster has it as a member still.nbs host replace <host>removes the old member -- for good, its incarnation never taken again -- and offers the host the cluster as a new one.
A member removed that did not lose its state is still the incarnation it
was. Added again, it is refused (that incarnation was removed: never taken again). To take it back, rmmod nbs-mds there, rm /nbs/mds, insmod,
and nbs host add it.
A host whose log alone was lost -- raft/ holding none, its member and
begun there -- is the same loss, and is never started as the member it
was, nor recovered from: the module says so at its load, in nbs and in
the provider's status (stopped), with what to do --
rmmod nbs-mds, rm /nbs/mds and insmod there, then nbs host replace
on a member (plans/vnet.md, V1a: before it, such a member started as the
incarnation it was, its votes forgotten). A host that took an offer and
never wrote a word -- a member, no begun, an empty raft/ -- starts, as
it always did.
When the majority is gone
nbs recover on the last member standing makes it the NBS cluster's one
voter, as cluster recover does the registry's (cluster.md,
"When the majority is gone") -- the same code, raftmember's recover:
refused while the member follows a leader or any other voter's mds answers
a probe within 3 s (TAG_PROBE); without the cluster's name it says what it
is about to do, and with it, its log is made the log of a group of itself
alone, the member started again over it, and every other member removed
for good -- each back only as a new member, by nbs host add once its
folder is gone. What the lost members had committed and this one had not is
gone: a volume made, an attach, a snapshot. A writer on a lost host must
never come back -- its host's folder goes, as any removed member's -- for an
attach generation it was given may be given again. Typed at a host's own
shell alone, as cluster recover is: no provider's call makes it. Every
member is to run a build with it first: a member of an older build does not
answer the probe, and would be taken for gone.
How a member runs
raftmember's, as the registry's member runs (cluster.md, "How a member runs"):
- one task,
nbs/mds, over the log, ticking every 100 ms; - a message taken only from the host its node is on;
- a shell's command asked once whoever leads, under a session of its own
and sent again to whichever member leads for up to 30 s (
raft::asks); answered once this host's member has applied it too, so that what is read next on the host holds the change; - raft's membership kept to the state's voters and learners, a change at a time, a removed leader handing the lead on first.
What is NBS's own:
- the cluster's id on every message;
- its state's digest, held to the leader's; its cs's questions, its clients' asks, the controller and the scrub.
What every group whose hosts are offered it has, NBS's and vnet's alike, is
raftmember's (offered.rs, plans/vnet.md, V1a), and its members' rules
members' (src/crates/common/cluster/members):
- the offer, and its answer from a member already;
- the listener that holds
nbs.mdswhile this host is in no cluster, taking an offer and answering raft messages for members it does not have; - the lost members kept for
nbsto say; init,host add,remove,replaceandrecover; the member's start over its log, with thebegunmark, and its stop;Init,AddandRemoveas applied, who votes -- the first five -- and the rules a snapshot read back is held to.
nbs init starts the member over a new log with this host its one voter,
commits Init, and only then writes member. Should any of it fail, the
member is stopped, its log and member removed, and the listener started
again.
The chunk server
nbs-cs.ko is a host's cs: one task, nbs/cs, that holds the host's devices
and serves nbs.cs, a Cluster name, to NBS's clients on any host of the
registry. What it does to its devices is nbscs
(src/crates/common/storage/nbscs, no kernel in it), which tests/fuzz/nbs runs
over disks that lose power. What a client says to a cs, and a cs to its mds,
is nbswire (src/crates/common/storage/nbswire).
nbscs the cs: what it is doing, its devices, the requests it served
nbscs chunks the chunks it holds: each one's state, volume and generation
nbscs fences the volumes it fences, each at its generation
For example:
a$ nbscs
nbscs: cs 0 -- serving: 2 chunks held, 2 open, 1 volumes fenced, 9 requests served, 0 refused, 0 writes passed on along chains, 0 not, 0 answers lost
device 3 part:4f7661ab-0525-4f56-9d6b-9283d322e47c 292 MiB, 284 MiB free, 0 slots lost, active
What nbscs shows is what the cs's task last made of itself: at the end of
a turn that fenced, closed, freed or took a device in, and once a second
otherwise -- the shell never waits on the cs's devices.
The cs is one task asleep on one event, which its mailbox and its
devices' queues signal (N12c). Awake, it takes what has come: the leader's
look (Op::Ping) answered there and then, whatever its devices are doing;
an answer of its host's mds handed to the request that asked; a request kept,
in order, for a future of its own -- up to 64 in flight, the rest waiting
their turn to begin, refused for now past the mailbox's own bytes. Then it
polls every future again while anything moved: a device's request reaped,
a turn let go, a commit round or a flush ended. Nothing it does waits on a
device or on the mds: a request waits where it is, pending, and the device
sees as many requests at once as its queue has buffers (8 of 256 KiB). A
step not done in 30 s fails its device. A look says whether the cs is there,
not how long its queue is: before N12 a cs served one request at a time,
and a look behind a writer's stop -- 140 closes, three flushes each, where a
flush took 10 ms -- missed its 250 ms, and the leader took a cs that was
serving all along out of its sets (plans/nbs.md, N11).
Its devices
Every 5 s -- every second while it does not serve -- the cs asks this host's mds which cs it is and which devices it has: a device just added is taken in before the leader's controller, two seconds on, takes a cs that does not serve for down and has its places joined again for nothing. The mds answers from its applied state; no other host's is asked. The cs takes each device in:
- held with a queue of its own (
DiskQueue) and claimed asnbs-cs, so that nothing else writes it while the cs has it; - opened where it is this cs's: its superblock, records and fences read back, and a torn or stale copy of any written again from the other;
- formatted where it is blank and the metadata has no format of it; its
format then proposed for the metadata to record (
Question::Formatted), every 5 s until it has it; - left as it is where it is anything else -- NBS's of another cluster, cs or
device number (a disk moved here), not NBS's at all, another size than
the metadata's, another format than the one it recorded (formatted again,
or an old copy of it), or blank where it recorded one: wiped or rotten,
what it held lost to this cs, which serves nothing without it until
nbs devices add <id> wipe. Said in the log once and innbscs, and tried again every 5 s.
It serves only once every device the metadata gives it is taken in. With one
missing, a chunk on that device would read as nothing, and its first write
would put it in a second slot, which the device coming back would contend
with. Until then every request is refused NotReady, and the client asks
again.
A cs's device
No filesystem: a format of NBS's own (nbscs's format.rs).
slot 0 page 0, 1 the superblock, twice: the cluster, cs and device it is, its size, its format's id
page 2..34 the volumes' fences, copy A
page 34..66 the volumes' fences, copy B
slot 1.. the table of slots, copy A, then copy B: a record a slot
then the slots, 4 MiB each: the chunks' data
- Every record, a fence's or a slot's, has a sequence number and a CRC32C seeded with the format's id -- 64 random bits made at the format -- so that a record of an earlier format, or of another device, never passes.
- A record is changed in copy A, which is flushed, then in copy B, flushed too. One copy is ever being written, so a power cut leaves the other good. Of two good copies the newer is the record; zeros in both, it was never written.
- A record neither of whose copies is good is lost: its slot is never used again, and that is said.
- A device is blank where its first two pages are zeros. Anything past a superblock's 128 bytes in them is not a cs's, since a format never writes there: a filesystem whose first bytes are zeros is never taken for blank. A copy of the superblock that is not the good one -- a format cut after copy A, a copy rotted -- is written again as the device is opened: left so, one rotten bit of the other read as blank (the review of 2026-11).
What a request leans on
- A fence is durable before it is answered, on every device of the cs. A
volume's fence is the newest attach generation the cs has been told. A
request of an older one is refused (
fenced), after a crash too. - An open mark is durable before the first write of a generation to a chunk. After a crash, every chunk whose replicas may differ says it is open, at the generation that wrote it last: what N8's reconcile goes by.
- A chunk is held by its number and its incarnation. The metadata gives a freed chunk's number to the next chunk made, at the next incarnation. A slot of the chunk a number was is never read or written for the one it is now, and is freed at the new one's first write; a request of the older incarnation is refused.
- A slot is zeros before a chunk takes it, the zeros flushed before the record that gives it the slot: what was freed is never read through another chunk.
- A close is durable only after the chunk's data is: a flush begun after the chunk's last write was done comes first.
- Requests in flight together (N12b: the engine's operations are
async fns, many of them out at once on the cs's one task): a chunk's requests are done one at a time, in the order they came; a volume's together, and its fence alone -- after every request of the volume asked before it, before every one asked after -- so that no request of an older generation is done once a newer one's fence is answered. A record or a fence changed marks its page, and a commit round writes copy A of every page marked on every device, flushes, then copy B, flushed: never both copies of a page in flight, and the opens, closes and frees that came together made durable by one write of each copy and two flushes. A flush covers the writes done before it began -- each device counts its data writes done, a flush the count it began at -- so a close or a client's flush waits for one begun after the writes it needs, joining one in flight that did. - Another volume's chunk is read or written only once the host's mds vouches that this volume's map names it (N10b): a snapshot's or a clone's map shares the chunks of what it was made of, and a clone whose image is deleted writes the image's chunks it alone names. Written, the chunk is the writer's volume's from then on -- what its reconcile asks the cs for, and what its flush is held to.
- A copy-up's chunk is durable before it is answered, its data before its record, and never written over once held (below).
- A chunk the cs does not hold is read as zeros only once the mds vouches that the volume's map names it (N10c): one freed since -- a copy-up done, the old chunk named by no map -- is not a chunk never written, and a read of it still on its way from before is refused.
- A write of a chain is applied once (N13): the cs keeps, for each
volume's writer, its mark and the ids above it it has applied; a write
that comes again -- along its chain and directly -- is answered as done,
not written again, and past 65,536 kept, refused for now until the mark
moves on. Once, that is, for as long as nothing else can have written its
sectors: an id applied before a join of its set began at this cs was not
marked by that join, and its copy may have put the source's data over it
-- the write that comes again then, sent again because it is not done,
is written again and marked (
Store::joins_begun, counted as the write is marked: found by tests/fuzz/nbs's cluster target, N13, where a member taken out and back answered as done a write its copy had undone). A write that comes late from before the join is refused by the epoch, as any is. - A chunk's requests end in the order they came (N12c), each from its arrival to its answer, its questions to the mds among it -- the engine's own turns order them inside the store only. A write sent twice by its client, the first copy answered while the second waited for the mds, would otherwise end after the client's close of the chunk, or its next write there, and undo it: the chunk open again, older data over newer. The garbage pass frees a chunk in the same order, so that no free comes between a request's look at the chunk and what it does: a read the mds vouched for, its answer on its way as the metadata dropped the chunk, would read zeros where the chunk held data.
A copy-up's copy (N10b)
Before a writer writes a chunk others share, it asks the metadata
CopyUp, then has each active member of the chunk's set copy it into the
new chunk, locally (Op::CopyUp { from, to, position, index }), then asks
CopyDone (N10c). On a cs (Store::copy_up):
- a new chunk the cs does not hold is asked of its mds first
(
Question::VerifyCopy), every time -- once a copy-up; - the copy is of the whole chunk, 256 KiB at a time, into a slot the cs takes off its free list only once the chunk's record is written: a copy that fails part way leaves the slot free;
- its data is flushed before its record is written -- closed, the copy-up's volume's, written at the set's version now, so that a member back copies it as any chunk written while it was out;
- asked again, with the new chunk held, it does nothing: what the writer wrote to it since is kept;
- the old chunk held by none here -- never written -- the new one is not taken either: both read as zeros;
- refused where the old chunk was found open at the cs's start: it may lack what its writer wrote since its last flush, which the copy would keep for good. The writer takes the member out, and it copies the new chunk when it comes back.
A copy-up holds the cs's task for its copy -- 4 MiB read and written, and three flushes.
Checked with the host's mds
A request names the volume, the attach's generation and the chunk: its
number, its incarnation, its epoch -- its replica set and the set's version
-- its place in the volume's map, and the entry of the metadata the client
read the map at. What its engine has checked already, the cs does. A chunk
it does not hold yet, or at an epoch other than the one it knows, it asks its
own host's mds about first -- the mds off the data path, and a cs that trusts
no other host's (decision 3 of plans/nbs.md). The mds answers from its applied
state once it has applied as far as the request's entry, Behind until then,
which the client asks again. It vouches that:
- the chunk is in that place of that volume's map, at that incarnation;
- this host's cs is an active member of the chunk's set -- or, for a write, its joiner -- at the set's version;
- a write is the volume's writer's, at its attach's generation; a read of generation 0 is a read-only one's.
A fence past the one the cs has (Op::Fence) is laid only once the mds
vouches that it is the volume's writer's now (Question::VerifyFence): any
host of the registry could fence any volume at any generation, up to the
last, and its real writer was refused (the review of 2026-11). What names no
entry to be applied as far as -- whose cs this is, which of its chunks and
volumes are gone, which joins are its -- the mds answers only once a read the
leader confirms is served (raft's ReadIndex, raftmember's Core::read): a
member replaced, brought its log from the start, or started again behind
what it had applied, told its cs that the chunks and volumes made since were
gone, and the cs freed them (the same review).
What the mds vouched for is kept, up to 65536 of them, and taken only for a chunk the cs holds (N10c), whose record refuses an epoch older than its own even so. One the cs does not hold -- never written, or freed since -- is asked about every time: nothing here refuses its epoch but the mds (found by tests/fuzz/nbs's cluster target: a read of a chunk never written, vouched for at a set's version, kept; the set joined; the writer's first write of it, by its view of before, taken as vouched for, and the joiner never written). A copy-up's copy it vouches for (N10b) where the copy-up is pending at that place of that volume, from that chunk to that one at their incarnations, both at the set's version now, this host's cs an active member of the set, and the volume's writer of now asking.
What is freed
Every 30 s the cs asks its mds which of the chunks it holds are gone --
freed, their number another chunk's now, or of a set the cs is neither a
member nor the joiner of any more, a member replaced by another's join --
and gives their slots back, and which of the volumes it fences are gone,
and takes their fences off (nbswire's collect, which tests/fuzz/nbs's cluster
target's cs's run too). A page of 4,096 chunks at a time; a chunk freed only
at the epoch it was listed at -- one a copy or a client has taken at a newer
epoch since, the cs made the set's joiner meanwhile, is kept. And sooner
than 30 s once a write was refused for want of room: the room of a replica
taken out came back to it only that way.
A client's requests and their answers go between hosts by ipcnet's bulk connection, all of them, so that they keep their order and their data do not hold up raft's messages. A request carries 256 KiB at most. One to a host with no cs -- nbs-cs not loaded there -- comes back from the channel as a bounce, and the client takes the cs for one that does not answer.
The client
A volume is used through a session of it (nbsclient::Session): a state
machine that is handed what comes -- its host's mds's answers, the cs's --
and the time, and says what to send and which of its driver's operations
are done. No kernel is in it; nbsdrive::Drive drives it over a mailbox of
its own in the kernel, and tests/fuzz/nbs's cluster target over a network of
its own.
A session's life:
- Attach at the host's mds (
nbs.mds,TAG_ASK): for writing at a new generation --forcetakes the volume from a writer on another host -- or read-only, which a writer elsewhere refuses. A reader's attach is answered with the generation a writer would be given next, which its reads carry (Op::Read'sas_of): a writer attached after it -- byforce, which lets the readers go -- fences the volume there or past it, and a cs refuses the reader's reads as fenced from then on, its session failed and said; read at no generation, it read on under the writer, what it read torn between the two (the review of 2026-11, its question 4). The attach carries the session's token: asked again, its answer late or lost, it is answered with the generation it gave, not another. One not answered in 10 s fails the session. Its answer says whether this is the volume's first attach for writing, made empty and no clone: all of it reads as zeros (fresh, which hv's pour of an image leans on to skip zeros -- a clone's first attach is not fresh, its origin's data under it). - The map, read from the mds at the attach's entry or later: each place's chunk and incarnation, its set, the set's cs's and their states, each cs's host.
- Fences: every cs a set of the volume writes to fenced at the generation, from which on an older writer's requests -- writes, reads, closes, flushes -- are refused.
- Reconcile: each chunk any replica holds open -- left so by a writer
that stopped uncleanly -- read from one replica, this host's if it has
one, and written to every one, 256 KiB at a time, then closed. A cs says
what it holds open a page of 4,096 at a time (
Op::Held), every page asked: a session took its first alone, and served the rest unmended (the review of 2026-11). A volume's last chunk is mended whole too, past the volume's end where its size is no multiple of 4 MiB: a chunk is whole on its replicas whatever the volume's size (the review of 2026-11: the piece past the end failed, told to nobody, and every start stopped there). The start goes on for as long as it mends: its driver gives up only when nothing has moved for two minutes. After an attach anew, also every place a write out had reached a replica with, open anywhere or not: a join's copy of a write out closes what it copies, and its source taken out and back left the chunk open nowhere (N13c). A chunk others share is written in place -- what the reconcile writes, one of its replicas holds already, the same for every map that names it -- not held for a copy-up, which waits for any join of its set (N13c: a join stuck under its one source's failed device failed the session at the mend's deadline). - Serve, until stop: what was asked before it done, a flush, every
chunk written closed, detached. A stop asked while the session is still
starting, or paused for a snapshot, is begun once it serves; one asked
again is refused, the first standing. A flush or a close asked while the
session reconciles after an attach anew waits until it serves: answered
done before, it said durable what the mend had not yet written (the
review of 2026-11). An operation not done in a minute fails, whether it
was sent or still waited its turn. Every five seconds the writer tells
its host's mds it is there (
Ask::Here): an mds started again knows the session to pause for a snapshot without waiting for an attach.
What it holds to (plans/nbs.md, "Replication: the rules"):
- A write is done once every replica in has it. One that has not
answered in 0.5 s is asked out at the mds (decision 6): the leader looks at
the cs (
Op::Ping, 250 ms) and takes it out of its set -- the set's version up, and with it every chunk's epoch -- only if the cs does not answer it either; otherwise the ask is refused for now, and asked again only while something still waits for that replica: one that answered since is let be (the review of 2026-11: asked again five times a second for the session's life, the first look of the leader's a moment late took out a replica nothing waited for). The ask names the set's version as the writer's view has it, and the metadata refuses it as stale once the set has moved on: a member asked out late is taken out of nothing it has joined since. A read-only session asks nobody out: a reader is no writer of now, and a replica silent to it is read from another. A cs that refuses as failed is taken out as it is, with no look. Only the volume's writer of now takes a replica out, and only of a set of its volume's: a writer fenced since is refused (NotTheWriter), its session failed -- else a write of its would be done by taking out the replicas that refuse it. The set's last active replica is never taken out: writes wait for it. - A write to a place whose close is out goes to each replica directly, behind the close on the way, the close sent no more, until the place is closed again: along a chain it could pass the close, and a close sent again could come after it -- either closed the chunk over the write, and a writer taken over before its next close left a chunk open nowhere whose replicas were apart (found by the cluster target after the review of 2026-11).
- A write's piece leaves the writer once (N13): along its chain -- its
replicas as the view has them, this host's first where it has one, the
others in the set's order. Each replica passes it on to the next as it
takes it, before its own write is done, and answers the writer's session
itself, at its mailbox by name (
<module>.nbsw.<token>@<host>--hv.nbsw.…,nbsload.nbsw.…-- registered for the cluster by nbsdrive under the namespace of the module it drives the session for, the only one a module may name a mailbox under: an ipc address means nothing to a third host; a name not had is said, and that session writes directly; the writer's own host's cs, whichname@hostwith its own name does not reach -- the channel routes to peers alone -- answers by the name alone). Each hop is a request of its own, its id and its sub, so that the session's bookkeeping is the one above. A replica is waited for from the answer of the one before it in the chain -- which passed the piece on as it took it -- and sent the piece directly where it has not answered 0.25 s after; one whose replica before it is taken out of the piece unanswered is sent it at once; the 0.5 s counts from the direct send: a forwarder that died, or a link down between two replicas, never has a live replica asked out. Counted from the piece's leaving, as first written, a chain slow all along -- its devices, its links -- had every hop after the first sent twice, the writer's line carrying more than before N13 just when it was busiest (found by the cluster target, whose network carried the copies until answers went late and live cs's were asked out). Every request of the chain goes with the session's mark for its replica -- every request of its there below it answered -- and a replica applies a write of a chain once: the copy that comes again, along the chain after the direct one or the other way, is answered as done, not written again -- it could come after the session's next write of the same blocks and put the older data back (found by tests/fuzz/nbs's cluster target at once) -- and so does every other send of a write of a session that chains, to one replica again after a refusal or to each where its chain could not be had: sent without its mark, a late copy by another way than a newer write of its blocks put older data over that one (found by the cluster target). A session with no name --nbs io's -- sends each piece to each replica directly, as before N13. - A write alone goes to each replica directly (N13c): one whose session has no other write out -- none of another's pieces sent and not done -- is sent to each replica at once, its marks with it as above, and answered by each by name. Its time is the slowest replica's, not the chain's hops one after another, each taking the whole piece before it passes it on: at depth 1 the chain cost a 256 KiB write half its speed (N13b). With another write out, a chain again -- one copy leaving the writer's host, which is what depth is for. On the AX41's three VMs, 16 MiB to 64 MiB written at depth 1, its chunks open, three rounds each: 4 KiB 1350-1377 IOPS along chains, 1954-2002 alone (0.73 ms to 0.51); 256 KiB 24.6-25.0 MiB/s, 46.7-47.9; at depth 2 and 8, and 4 KiB at 32, the same both ways, the writes going along chains then.
- Two writes that overlap are never out at once. A read goes to one active replica, this host's where it has one, and to another when it does not answer.
- Reads and writes out at once are held to a window. Their bytes are
WINDOW_BYTESat most -- 2 MiB, half what ipcnet holds waiting for a peer (4 MiB a class) -- or one alone, whatever its size; the rest wait their turn in the session, nothing of them sent, and go as those out are done. A guest keeps up to 64 requests of 248 KiB in flight, and before the window all of it went at once: past the link's queue the rest bounced, went again 200 ms on (AGAIN_NS), bounced again, the link idle between -- on the demo stand, whose links carry 69 MB/s and answer in a millisecond, writes took 1.5 s at 10 MiB/s, whileiopingat depth 1 had 1.4 ms. A request waiting its turn has not gone, so no replica is silent for it. The window is a session's: volumes written at once from one host share the link's queue, two of them fitting it. - Its view of a set is whole, or held as behind. Each change raises a
set's version by one, so a take-out's answer one past the view is that
take-out alone; one further is changes the view lacks -- a join begun --
and the set is read again, its writes held until then. A map read that
finds a set changed sends its writes out again by the new view, so that a
write is done only by every replica the view has in, a joiner among them.
A read that finds no active replica in the view waits for the map: the
set's last active member is never taken out. A cs that comes into the view
while the session's start fences is fenced too, and the start waits for
it. A session detached or failed takes no answer of before -- the mds's
to an ask, nor a cs's to a request: a copy-up's member refusing its copy
as
NotAttachedonce the stop had let the volume go failed a session detached (found by the cluster target, N13c). An ask to take a cs out refused as no replica of its set -- its place another's since, a joiner's of the controller's -- has the map read again, which drops the ask; what waits for the cs is done as it comes due, by the view read -- a request to a cs no set writes as a silent one's, a close the cs refuses as not in the chunk's set (NotHere) at once. (Each found by the cluster target with the controller in it, N9d.) A place whose close is settled so -- refused as no replica's, or waiting on a cs taken out of every set -- is not taken for closed until the next map read (close_moved), which sends the close to the cs that took that one's place: a joiner made a member may hold the chunk open as the source it copied it from did, and the close of the view before never reached it (found by the cluster target, seeds 18153 and 25047, once a copy kept its source's mark). A close refused for its view, to be sent again after the map read, is sent no more once its place is written again: sent after the write, it closed the chunk over it (seed 20346). - A flush waits for every write before it, then asks every cs that has
acknowledged a write since the last flush began, and that some set of the
volume still writes, to make its writes durable. It is done once they have
and once every flush before it is, since each takes the cs's dirty when it
begins; one that fails fails those out after it, and the cs's it waited
for are dirty again, for the next flush to ask: a flush done says durable
what was written before it, the failed one's too -- on its set's last
active replica, which no ask takes out, it would say so of a write the
replica lost (#95). A close is a flush, then
every chunk written closed -- 128 places at a time, the next as those are
done, as a stop and a pause close too: every replica's answer comes to the
session's mailbox, and a random writer's stop over 8 GiB closed some 2,000
chunks on three replicas at once, its mailbox's 1,024 places overflowed,
the answers past them lost, and the replicas that sent them asked out as
silent -- the head of every chain among them (N13b, the EX44; the mailbox
has 4,096 places now, and a cs counts the answers the channel would not
take:
nbscs's "answers lost"). By bytes, nbsdrive keeps the reads it has out to 3 MiB -- half the session's 8 MiB mailbox, less a MiB for the rest -- so that their answers fit come each twice, a request unanswered a second being sent again; a read past it waits in the driver, in the order it came. hv's disk kept 64 reads of 256 KiB out, 16 MiB of answers for that mailbox, and an answer with no room was lost, its request sent again a second on (the review of 2026-11, m26). The mailbox is not made bigger: a module's mailboxes hold 64 MiB together, and hv's are a volume's each -- eight volumes at 8 MiB. A cs taken out of every set holds nothing the volume counts on: a flush, a close or a fence asked of it is done, not waited on -- nbs-test's repair phase found a close waiting for good on a cs the controller had taken out (CsDown) while it still owed a flush. - A replica that restarted with a chunk open at the writer's generation
may have lost what was written since the last flush, which the others
have: it refuses that writer's requests of the chunk -- and its flushes --
as failed, which takes it out. Where it is the chunk's last active replica
the session fails (
LOST_LAST): the volume is to be attached again, whose reconcile takes what the replica holds. - A close of the driver's that waits on a place a write took at the very moment it began -- so taken as written before it -- has it closed once the write is done: nothing else closed it, and every flush after waited for the close to its deadline (found by the cluster target, seed 17081, once its end held what was asked before a stop to be done).
- A chunk written, flushed and idle for 3 s is closed on its replicas, so that a crash leaves few to reconcile. An operation not done in a minute fails, said. A write that so fails, a piece of it not done sent to a replica, may be on some of its replicas and not on others -- and a close of its chunk later would keep them apart for good, a read answered by whichever it went to -- so the session attaches anew then (decided 2026-11-12). One whose pieces not done reached no replica -- held for a copy-up a stuck join holds back, its others on every replica -- fails as any other: attached anew for it, the reconcile could wait on the same join and fail the session. Attached anew, every operation out fails with the old generation, whose requests the new one's fences refuse, and the new one's reconcile makes every chunk the session left open the same on its replicas, as it does a crashed writer's, before it serves again; a stop asked meanwhile is begun then. A session taken over meanwhile is refused it, and fails.
- A writer that follows another -- by force, or after a crash -- holds what that one had done and not flushed as a crash would leave it: there or not, as the replica its reconcile read had it.
- A write to a chunk others share -- a snapshot's or a clone's map names
it too, as the map says (
Place.shared) -- is held while the chunk is copied up (N10c): a new chunk asked of the mds (CopyUp), each active member of the set, as the view has it, asked to copy the old one into it (Op::CopyUp), and the mds told (CopyDone); the map names the new chunk then, and the writes held go to it. A member that refuses as failed, or is silent 0.5 s, is asked out as for a write, and copies the new chunk when it comes back, as any chunk written while it was out. A member's refusal as of another epoch or map, a take-out's answer past the view -- the members asked may not be the set's: done without the one asked, a copy-up was done by none, the new chunk zeros under the writes after (found by the cluster target) -- and a copy-up's end refused, the copy-up given up meanwhile, have it asked again from its start once the map is read again; each step of it does nothing done already. Its end is asked only once every member the view has active then has copied: a member made active since the members were asked -- a join done, the map read again for another reason -- has copied nothing, and the copy-up is asked again from its start (found by the cluster target once its cs's answered later, N12c: the one member asked crashed, the two joined since never asked, the one asked taken out one version on). A chunk shared no longer is answered as itself, and written as it is. Refused for now -- the set being joined, or a member of it drained -- it is asked again, the write waiting; refused for room, its writes fail, said. - What a request out costs. A write's piece is up to 256 KiB, and its
request -- the data and a head of 57 bytes -- a block of 512 KiB of a
kernel's heap, whose page-sized blocks come from a window of addresses
of their own for each size: on a host of 1 GiB that window holds about
170 such blocks, however much is free. So a request out keeps no bytes
of it: what made it -- the session's generation, the map's index and the
chunks it names, its stamp -- is kept, and the piece's data is the
driver's until the operation is done; a resend makes it again as it first
went, and the bytes are the send's alone, handed to the mailbox, which
keeps them in pages. A request the heap has no room for now is sent again
in a moment, said once, its cs not counted silent for it -- never the
host's panic (found by hv-nbs-test, the guest of a host whose replica was
killed under its write: the session kept a copy a replica of every piece
out and cloned one for every send, 192 blocks for a guest's 64 requests
of 256 KiB). A cs whose answer -- a read's is a piece's data -- finds no
room refuses for now (
NotReady), asked again in a moment.
nbs io
nbs-mds's nbs io (io.rs) is a session of the volume's for the shell and
the tests:
nbs io attach <volume> [force] attached for writing here: a new generation; force takes it from another host
nbs io write <volume> <offset> <bytes> <byte> <bytes> of <byte> at <offset>, to every replica in
nbs io read <volume> <offset> <bytes> from one replica: this host's where it has one
nbs io flush <volume> what was written durable on every replica in
nbs io close <volume> that, and every chunk written closed on its replicas
nbs io held <volume> the volume's chunks each cs holds open: asked of each, not of a session
nbs io detach <volume> flushed, closed, let go of
The first write attaches the volume for writing on this host unless
attach did; the session is kept for the commands after, until detach or
another attach, and moves only while a command drives it -- one of its
own, or nbs volume snapshot asked on this host, which pauses it. A read
on a host with no session of the volume's attaches read-only for itself and
lets go after -- refused while another host writes. Offsets and sizes are
whole 4 KiB. For example:
a$ nbs io write disk1 0 65536 0x5a
nbs: 65536 bytes of 0x5a written at 0 of disk1, generation 1: 1 chunks, to every replica in
a$ nbs io read disk1 0 65536
nbs: 65536 bytes at 0 of disk1, from a: 0x5ax65536
nbsload
nbsload.ko is blkload's for a volume: a load test through a session,
IOPS, bandwidth and latency down to the percentiles (common/latency, the
histogram they share).
nbsload <volume> [randread|randwrite|read|write|verify] [bs=4k] [qd=8] [secs=5] [span=<MiB>] [force]
The command's own task keeps qd operations out at once; a write test
attaches for writing -- force takes the volume -- and destroys what is
on it. verify writes blocks each of which says which block and which of
its writes it is, holds every read as it goes to the block's last write
done, and after a flush reads every block it wrote back: one that does not
hold its last write is said, with where.
A guest's disk: nbs:<volume>
hv's guests take a volume as a disk: disk=nbs:<volume> on hv start, or
in a vm.toml's disk. The disk is a virtio block device of 512-byte
sectors over the volume's 4 KiB blocks -- its physical block, so that a
distribution's image boots off it as off any disk -- the volume's size, its serial nbs-<number>; its task drives the session --
attached as the guest starts, every cs fenced, the open chunks reconciled --
and the guest stopped lets the volume go: flushed, closed, detached. hv start <name> force takes the VM's volumes from a writer elsewhere: a VM
moved off a host that is gone. No vm.toml says force, and it is the first
boot's alone: a reboot of the VM -- its own reset under restart, an hv restart -- attaches plainly, and fails, saying so, where another host has
taken the volume since; kept, it took the volume back by force, and two
hosts fenced each other in turn.
An image is poured onto a volume made beforehand (nbs volume create):
hv convert <image> nbs:<volume> [force] a qcow2 image or a raw disk, written through a session, flushed, let go of
hv create <name> <url> onto=nbs:<volume> a cloud's image downloaded and poured onto it, the VM's first disk
A volume never written before reads as zeros, so a pour onto one leaves the image's zeros out.
Or a cloud's image is made an image of NBS's once, and every VM of it is given a clone (N10d3):
hv image <name> <file|http(s)://image> an image of NBS's: downloaded where it is a URL, a volume of its disk's size made, poured onto, frozen
hv create <vm> image=nbs:<image> [size=N] [onto=nbs:<volume>] [words]
a VM whose first disk is a clone of the image: a volume of the VM's name, or onto='s, grown to size=
The vm provider does the same through the bus, as the panel asks it: its
image is hv image, and its snapshot a snapshot of a VM's volume --
taken where the VM runs, its guest's session paused for it, as nbs volume snapshot pauses a writer's -- of which create's image makes VMs.
hv asks NBS through the management bus, as the host API does: nbs-mds's
nbs provider (volume, volume_create, volume_resize,
volume_freeze, volume_clone, volume_snapshot, volume_delete), so hv.ko links nothing of
nbs-mds's, and without it says to insmod /nbs-mds.ko. hv image holds
the name on its host while it makes the image -- a second of it there is
refused -- refuses one NBS has, downloads a URL into the vms' folder
(.image-<name>, removed after however it ends), makes the volume on this
host of the disk's size, pours, and freezes it; one that fails deletes the
volume while it is a volume still and this host's -- an image another host
made of the name meanwhile is never taken away. hv create ... image=nbs: refuses an image that is a plain volume, a clone's name NBS has
already and a size= under the image's; the clone is made before the
folder's seed and vm.toml, and deleted with the rest when a later step
fails. The clone is thin -- each chunk shared with the image until the VM
writes it, copied up then -- so a VM of a 3 GiB image costs, at first,
only its metadata.
Tested
-
tests/nbs-fuzz.py(host, in CI; testing.md):tests/fuzz/nbs'smetatarget, the state against a model of its own:- every command's answer and everything the state shows held to the model's, the placement to the model's own; among the commands, cs's going down, up and out, and joins begun, ended and given up, from the controller's turns and at random;
- an fsck of its own after every step;
- the snapshot read back in pieces, spoiled and refused, and continued from: bytes spoiled at random, and a set's joiner made one no set can have;
- two members' digests compared, and a divergence said;
- the controller's progress (N9a): from the state of a step, some hosts gone for good and the rest answering, its turns applied until it has nothing to propose -- every proposal answered as the model answers it, none refused -- and then no member down whose cs answers, no place left that a cs had the room to fill, no join under way, every cs that answers up and every one gone out;
- what a host's mds tells its cs (
told_check, the review of 2026-11): a client's request verified, a joiner's ask of a source, a fence, the chunks and volumes still there, the cs's identity and its joins -- mostly as the model has things, now and then a field made another -- each answer the model's, and what names no entry told only by a member that has confirmed a read.
Its
cstarget is the chunk server's engine over disks with a write cache, through histories of fences, writes, reads, flushes, closes and frees, and power cuts: each write not yet flushed kept, lost or torn at a sector's edge, and now and then a record's sector torn inside or a bit rotted. After every cut the cs is opened again and held to every fence and open mark answered, every chunk closed reading what it held, every sector one of the values it may have, no record lost; before each run, a disk with something else on it never taken for blank, another cs's told apart, a blank one of a format the metadata records found lost and one of another format refused, a superblock's copy gone written again at the open, a set's listing that waits for a first write still waiting for the mds's word, a stranger's fence the mds refuses not laid, and a chunk the garbage pass listed, taken at a newer epoch before the mds's word that it is gone, kept. A device failed under a step answers no flush as done and lists no set's chunks for a joiner; and a chunk freed at an epoch it has moved on from since its listing is kept. Itscstarget also frees a chunk through the garbage pass while a read of it waits for the mds's word, the word coming after: the read ends with what the chunk held, never zeros (N12c). Itsinflighttarget (N12b) is the same engine with many requests' operations in flight at once on one task -- tens to a couple of hundred -- over disks whose requests are in flight too, done in any order, a flush covering only the writes done before it was asked, and power cuts with requests out: each write in flight or in the cache kept, lost or torn, a page of the cs's own torn inside now and then. Held to: a read answers the last write of its chunk asked before it; no write of a generation answered once a newer fence of its volume has been; every request answered once nothing more is asked, and then no turn held and no round running; and after every cut no record lost, every fence answered there, every chunk written and not closed since open, a chunk closed reading what it held at its close, and every other sector what the last flush or close answered after its write left, or a value written since. Itswiretarget holds nbswire's messages to their encoding, and reads noise as none.Its
clustertarget is the whole of it end to end (N8): the client's session writing, reading, flushing and closing a volume through six hosts' cs's -- nbswire'sserveover nbscs, as nbs-cs.ko runs it, many requests in flight at once at each, a request's question to its host's mds a message of the network, answered as the state is when it gets there and handed back when the answer does (N12c) -- whose answers are nbswire'smdsover nbsmeta's state, over a network that delivers in any order of pairs, with hosts that crash -- their disks' caches kept, lost or torn -- and come back, writers lost with them, and writers taken over by force: now, or at rest, the old one asked on to write and flush. Every write done held on every replica in, every read to what a write done or out since may have left, every replica that comes back to what a flush made durable, the replicas the same once a writer serves and at the end, and no active replica doing an older generation's write or flush once a newer writer serves. At the end of some runs a host is lost for good, where every chunk can do without it: the writer's writes past it, its flush and its close must be done, and no flush may go to its cs once no set has it in.Its
jointarget is one join's copy (N9b) between two engines, the source served by nbswire'sserveas a cs serves it, and the hosts' mds played from the set's state the target keeps. The network delays, reorders, drops and repeats; up to three writes of the map now are in flight at once, each delivery anywhere among the copy's; writers of maps before the join write the source alone; the set moves on, and the join is given up and made again; a sector being checked rots; and the joiner's power fails mid-copy, now and then a few writes before a chunk is closed. In a third of the runs a member back joins its own place, with chunks found open after a cut; in a tenth each, one race random runs seldom put together is set up on purpose (testing.md). Half of the members back hold a chunk of the set the source has none of. At its end the joiner holds the chunks the source does and no other, every one the source's byte for byte, and every one the copy closed is still so after a power cut. The targets list a page of three chunks at a time (nbswire'ssmall_pages), so that every list goes past its first page.The cluster target asks a flush now and then as its writer attaches anew, and holds every flush and close done to a session that is not attaching anew: one answered during an attach anew's reconcile said durable what the mend had not yet written.
Its
tailtarget is theclustertarget over a volume of 22 MiB, no whole number of chunks: its last chunk, half past its end, written now and then and left open by a crash, mended whole by the next writer's start.Its
scrubtarget is the leader's scrub alone (N9e), over a set of three members whose copies are the target's: writes and closes reach each member on a delay of its own, the scrub's asks on another way, a member holds a chunk once its first write reaches it, granules rot on one member or two, or a member loses a chunk; members are taken down and joined back under the looks, and the races random runs seldom make -- a write raced through a look, a close held back on one member -- set up of a randomness of their own. Every finding is held to the set's version now and to what each member held when it answered, every rot is found, and a chunk held by none takes no turn of the pace.The leader's scrub runs in the
clustertarget too, its asks and answers in a queue and a randomness of their own so that the run goes as it would without them, held to finding nothing the target did not rot; at the end of some runs a sector of a closed chunk rots on one member, or on two, the finding is held to it, and the members taken down as a person would are copied again: every replica alike.The writers' copy-ups run there too (N10c): snapshots of the volume taken between two writers -- the one stopped, the next started -- two kept at most, dropped now and then; the writers after copy up each chunk a snapshot shares before they write there, through joins, drains, crashes and takeovers; each cs's garbage pass frees what copy-ups and drops leave; and every snapshot reads, on every active replica up and at every look, what the volume held at it. Snapshots of the volume attached are taken too, as nbs-mds takes them (N10c2): the writer paused, every chunk closed when it answers, the snapshot applied at its generation; the word to go on now and then lost and the snapshot now and then applied late -- refused, its session having attached anew -- and a pause not answered given up, its writer going on with no attach anew; at some ends the writer stopped while paused, a write held behind the pause. Every stop is held to being done once, as its session ends -- the stops' tags the driver's, as nbsdrive's are: with the bit of the session's own operations set, as they were, a stop's flush done out of turn was told to nobody, and no judge saw it. And a clone of the newest snapshot is made now and then (N10d2), with a writer of its own apart from the volume's: a volume born with every chunk shared, its writes copied up, its reads of chunks recorded as another volume's vouched for by the mds, its writes in place once the snapshot is dropped -- through the crashes, joins and drains the volume's writers meet; dropped now and then, at once where its writer is lost, and at the end once every block it holds is held to what its writer did. And now and then its end starves the writer of memory (#97, nbsclient's
starvefeature, the fuzzer's alone): every host up, its writes' requests find no room for longer than a cs is given to answer, then do -- each done, none failed, no cs asked out anew while it was starved, each read back. And now and then a flush after one that failed (#95): a chunk's last active replica -- the set's two others taken out, as the writer would ask, the controller held off -- holds a write not yet flushed when its host crashes, every write its cache held lost; a flush waits for it to its deadline and fails, and a write elsewhere and a flush after it, asked while that one is out or after it failed, may not be done before the replica is flushed -- the restart's judge holds what the replica has to what the last flush done made durable. And now and then a write past its deadline (N13's decision): one active replica of a chunk, not on the writer's host, cut off from every host but its own -- its host up, the leader's look keeping it in -- while the writer writes the chunk; the write fails at its deadline, the session attaches anew, never closing a chunk of the old generation after it (judged at every send), and, the cut healed, serves again at a newer one, every replica alike. Judged by the replicas, not the client's word: a write failed at its deadline that is on some active replica of a block and not on another must have its session close nothing of its generation after; one that reached none -- the session paused for a snapshot as it came, say, and its fence of the replica cut off waiting -- is not held to it. And a write starved of memory past its deadline, every request of it unmade, reaches no replica: the session goes on at its generation, not attaching anew, and writes the block once its memory is back (thestarved_past_deadlineend, a quarter of the seeds).--mutantstakes two hundred and seventy-five rules out one at a time -- eighty-five of the metadata's, six of them the controller's; sixty of the cs's engine; sixty-two of the wire's, itsserve's, the mds's, the copy's and the scrub's; sixty-eight of the client's -- each caught but three: some only at the cluster seeds the gate runs past its range (GATE_CLUSTER_ALSO), their races moved past the range's by a change of the target's course. The three are known gaps (tests/fuzz/nbs/mutants.py'sKNOWN_GAPS): rules of the client's whose races no campaign has found a seed of, past fifteen thousand each, said as gaps until a seed or a scenario holds them -- an attach anew mending the places of the writes it had out (a mark lost was what it was for, and a join's copy keeps its source's now), the reconcile's writes to a shared chunk in place, and a session detached deaf to the answers of before (the target holds a session over to send nothing, and none has). A close refused as not in the chunk's set done, and what was asked before a stop done before it, gaps before, are caught: the first at the gate's own seeds, the second by the end holding what was asked of a writer right before its stop, every host up, to be done. -
tests/fuzz/raft'snbsmetatarget: the members' part of the state, its commands from pools of names and incarnations that meet and from wide ones that fill the cluster. There are four scenarios and six mutants. -
tests/nbs-test.py(three nos ontests/hub.py's LAN, after the cluster registry is made among them, and a fourth booted late; every nbs-mds without_after=10, the least it takes, andscrub=1). It covers:- an NBS cluster made through the bus, as the panel makes it, and two hosts added, one through a member that does not lead, as a job through the bus;
- an add asked again answered with the same incarnation, and a second init refused, at the shell and through the bus;
- a commit through the majority;
- the leader killed: the other two elect one of themselves and commit, and the killed host comes back from its log;
- a volume's health through the bus:
degradedwhile a host's replicas are down, each of its chunks at two, andokonce the host is back and joined; - the leader cut off by the hub: the other two elect and commit while it commits nothing, then it follows once healed;
- a host's folder removed: the leader says its state is gone, an add of it is refused, and a replace -- a job through the bus -- takes it back as a new member;
- a member removed, and its old incarnation refused when added again;
- what is stored, on a pool disk of three partitions a host (N6):
- no volume before three hosts have a device;
- devices added by id and by name, and the pool's disk itself refused for having no id;
- c's copy of a's partition refused, as a disk moved between servers;
- volumes made, made again, refused for another size, grown, refused smaller, refused past the room;
- the room filled to its last slot, then given back by a delete;
- a device added through the bus, and one drained -- again from another host through the bus, naming its host, and refused naming a host it is not on -- the provider's devices saying each host's and its state;
- the provider answering through the bus as the command does;
- every host's digest at the same entry and the same;
- every log compacted into a snapshot;
- the killed host back from its snapshot and log, the replaced one sent the snapshot, and a removed host's cs taking no new chunks;
- the chunk servers (N7):
- every host's cs taking its devices in, formatting them;
- a device with a filesystem's magic on it refused, and taken with
wipe; - a volume written to its three replicas and read back, across a chunk's end, from the writer's host and read-only from the others; flushed, its chunks open on every cs, then closed;
- a writer on another host refused, then forced: its fences refuse the old writer's writes and reads;
- a host booted again, its cs back with its data, its fence and its open marks;
- the volume deleted, and every cs giving its chunks and its fence back;
- a host's nbs-cs unloaded: a request to it said at once, the others answering; loaded again, serving its devices;
- the client (N8), through
nbs io's sessions: a reader refused while a host writes, a forced writer's fences failing the old session, a host's closed chunks and data kept over its reboot; nbsload'sverifyholding every block to its last write, and a read load on another host; - repair (N9): a volume's two chunks written, the sets moved on past
them (b's cs taken away and back), c killed -- the leader's
controller has its cs down, then out ten seconds on, its places for
others' to take, of which there are none with three hosts -- one chunk
written past it and closed, c booted again: its cs up, joined in its own
place, every chunk at three replicas again with none being copied; a
read on c of what was written while it was gone, which its join's copy
took, and of the chunk not written meanwhile; and what its copies did,
each join's end in its log (
dmesg): the chunk written meanwhile pulled whole, the other found as its source's, nothing more; - a fourth host, d, booted late, invited into the registry and added to the NBS cluster, its cs taking its device in; a volume's two chunks written on a, b and c; one of the three killed for good -- its cs down, then out -- and d joining every set it was in, in its place: every chunk at three replicas, none being copied, the volume read off d's own replica; the lost host removed, three members again;
- rot (N9e): a chunk written and closed, its bytes found in a pool image
on each of its three hosts and a sector of them spoiled on one -- running
-- through QEMU's monitor (
qemu-ioover QMP), under its cs; the leader's scrub saying that host's copy unlike the two others', with the command; the command run, the member joined back in its own place and its copy done; the chunk read off that host's replica as written, the spoiled sector back as it was in the image, and the finding shown as history; at the end, a chunk written and found alike, the volumes' others held by none, and no finding of now; - nothing panicked.
-
tests/hv-nbs-test.py(x86-64, manual: a guest kernel and an initrd): Linux guests of hv's on volumes -- an image poured withhv convertread by a guest on another host; a guest's writes read back past its page cache, and by a guest on another host after it stops; a start refused while another host's guest has the volume, then forced, the old guest's writes failing; a host killed under a guest's write, the write done and read back.
Limits
-
One NBS cluster a host. At most 16 members, five voters.
-
An offer is trusted from any host of the registry: the registry is what says who is trusted.
-
A host that took an offer knows the members by the offer until its log has caught up. A leader elected among members added after the offer is not known to it until then.
-
Membership changes are one server at a time, without joint consensus.
-
A removed member may not hear of its removal, and goes on asking for votes, which PreVote keeps from disturbing the others.
nbson it says it is removed once it applied the entry. -
nbs host removerefuses only the last member. It is NBS's data, not its metadata, that needs three hosts. -
A replica taken out stays out: bringing it back, or a new one in its place, is N9's repair. Until then each host lost takes a replica out of the sets it is in, and a set whose last active replica is lost waits for it.
-
A reconcile copies a chunk left open whole, 4 MiB, from one replica -- this host's, or the first in -- whether the replicas differ or not.
-
A guest's disk offers no discards and no writes of zeros, and copies a read's data out of the session's buffer into the guest's.
-
A cs is one task (N12c): one CPU's worth of checks and copies -- a 1 GbE host's -- and it copies the data into a message and out of one. A look says the task turns, not that its devices answer: a device stuck short of 30 s keeps its replicas in, their writes waiting.
-
A cs serves only with every device the metadata gives it. A device that fails while it serves fails the requests of its chunks; one that is never taken in stops the cs serving. Taking a device out, and rebuilding what was on it, is N9's.
-
A snapshot of an attached volume is asked on its writer's host: the one mds that knows the writer's session. Its writer pauses for it, every chunk it wrote closed -- a moment's stall in the guest.
-
A write to a chunk a snapshot or a clone shares, in a set being joined or with a member drained, waits for the join (N10c), and fails past the operation's deadline: a join of a big set takes minutes. plans/nbs.md has what would lift it.
-
A chunk's data has no checksum of the cs's: a sector is taken to be written whole, as NVMe's atomic write unit promises. The scrub compares the replicas and says what differs; which copy is good a person decides. A copy takes its source's data as it is: rot on a source the scrub has not found yet goes into the joiner.
-
A device drained stays draining until it is added again, and neither a device nor a cs is ever removed. A placement never moves: moving chunks is N9's.
-
Bounds, each refused past:
- 4096 volumes;
- 2^22 chunks, 16 TiB of volumes in all;
- 2^24 places of every map together, 64 TiB: the volumes', the snapshots' and the clones'. A clone's map shares its origin's pieces in memory, but the snapshot and the digest write every map whole (the review of 2026-11: 1,000 clones of a 1 TiB image were a GiB of snapshot -- never taken past raft's limit -- and two seconds of every 64th entry);
- 2^16 replica sets;
- 4096 copy-ups pending at once -- a writer's next attach gives up those the one before left -- refused past as "pending", asked again;
- 64 cs's of 16 devices each;
- 1024 members removed;
- a volume's readers: one a member, 16 at most.
The state is in memory whole: some 4 MiB of heap and 3.75 MiB of snapshot for every TiB of volumes, and 4 bytes more for every 4 MiB a clone or a snapshot maps.