bcachefs
patreon
New rereplicate tool; replication ready for testing
🕑 Added 2018-02-08 21:02:52 +0000 UTC Replication support is finally feature complete; it should have everything implemented that's needed for handling and recoving from device failure.If replication is enabled on a filesystem, a device can fail and be removed while the filesystem is in use without returning any IO errors to userspace - reads/writes will be retried as needed, including on checksum error.
By default, mount is only allowed if all RW and RO devices are present. You can mount with devices missing with mount -o degraded; then, the mount will be allowed if sufficient devices are present that data won't be missing (todo: we need finer grained options for this, for when too many devices have failed and we want to try and read what we can).
bcachefs doesn't have any issues with multiple mounts in degraded mode; we track in the superblock what combinations of devices have data, so we always know _precisely_ which devices we need to mount safely. You can see the replicas section of the superblock with bcachefs show-super -f replicas <device>; this is fairly close to a raw dump, we could use a better tool for displaying this.
In general you should be able to add, remove, offline and online devices at runtime at will - subject to the constraint that we won't allow operations that take away devices we need, and you may have to specify --force if the operation would result in running in degraded mode.
You can offline a device and bring it back online, and all the data that was on that device will still be available. You can remove a device to permenantly remove it from the filesystem; removing a device drops any data on that device, so you'll probably need to specify --force unless you migrated data on that device to other devices first, using the bcachefs device evacuate command (and if it's online you'll want to put it in read only mode first, so we don't keep writing new data to it).
There's still lots of improvements to make on the tooling and documentation - this would be a great area for people to jump in and start contributing patches. I can't think of everything myself :)
For now, you can look at the replication tests to see some example uses of the various commands and to get an idea of what should be currently supported:
https://evilpiepirate.org/git/ktest.git/tree/tests/bcachefs/replication.ktest
