bcachefs
patreon
Status update - debugging and replication
🕑 Added 2017-04-11 05:18:01 +0000 UTC Debugging, debugging, more debugging...If you've been wondering at the slow progress, that's where all my time's been going. The unfortunate reality about creating a filesystem is that a filesystem, much moreso than most software, isn't all that useful if it's only, say, 90% debugged - you don't want a filesystem that doesn't eat your data _most_ of the time. And chasing down those last few bugs, that are the hardest to reproduce and find, is just a long slow slog. And not the fun kind, I'm not one of those weird programmers who enjoys just chasing down really hard bugs.
But it's well worth it to stay on top of, just like eating your spinach or exercising or doing the dishes. Oh well.
As for the bugs I've been working on - xfstest 311 (an fsync test) recently started popping - not sure when exactly, it's quite common that when you find a bug it turns out it was an old latent bug that was only exposed by a recent performance improvement or some other subtle change elsewhere in the code that started stressing things in a slightly different way. Which actually in a way is encouraging - when the bugs you're finding aren't regressions, but have been there since the code was written, you know the total bug count is declining.
So, the bug xfstest 311 exposed initially looked like a bug in the truncate code, but after about a week of debugging (and finding and fixing a couple other bugs that the tests hadn't exposed in the process) it turned out to be a bug in the journalling code - in the course of a btree update, when updating a btree node and journalling the change, the btree code was taking a ref on the most recent journal entry, _not_ the journal entry the insert had a journal reservation for. This bug was introduced around a year ago, when the journalling code was improved to pipeline things better, and allow a new journal entry to be started while there were still outstanding reservations on the previous journal entry, instead of forcing processes to block and wait for all existing journal reservations to be released when a journal entry filled up and we have to start a new one.
That bug was fixed in this commit:
https://evilpiepirate.org/git/bcachefs.git/commit/?id=7c73ce7f1e09e3b2ff968707e3adcb175a052c59
The other thing I'm working on - as time allows, when I'm not chasing bugs - is replication. There's still a fair amount of work to be done before replication is production ready, but it's becoming clearer what exactly the list of things that need to happen is:
The first thing, which I'm working on now, is moving per bucket allocation information (currently just bucket generation numbers) into a btree, which can be replicated and migrated like any other metadata, instead of only being stored on the device that information is for. (The current scheme - see prio_read() and prio_write() in alloc.c - is old, essentially unchanged from the upstream bcache code; that design was always something of a hack, even when I originally wrote it, and has far outlived its usefulness).
The immediate motivation for doing this is that it's necessary for running in degraded mode; we need bucket generation numbers to be available in order to know what pointers are valid, which is required to do much of anything - and we can reconstruct bucket generations if we have to, but that's a tricky repair operation. You especially don't want to a) mount a filesystem in degraded mode b) reconstruct bucket gens for the missing device, and then c) hot add the device, and then read the real bucket gens - that'd just be a huge mess of potential inconsistencies and bugs waiting to happen.
As an aside, the bucket generation number mechanism was originally created for invalidating cached data, but it's turned out to be an _extremely_ good thing to have even when there's no cached data - it makes some problems with bootstrapping allocation information during journal replay much more tractable, and it also makes it possible for us to repair certain kinds of corruption that we wouldn't be able to otherwise.
Also, moving this stuff into a btree isn't just for replication - this is also prep work for persisting full allocation information instead of having to regenerate it at mount time, which is one of the big things that needs to happen to fix mount times.
Next thing on the list is implementing a new mechanism for storing how data is replicated across different devices in the superblock. The most immediate problem this will solve is knowing whether we can mount in degraded mode (with a particular device missing), but there's also a bunch of other issues this infrastructure will help solve.
The issue we need to solve is: say you're already in degraded mode, perhaps you had a device fail - or more interestingly, suppose you had some write errors or discovered some problems during a scrub pass - whatever the reason, you have some replicated extents which are on fewer devices than required. One important thing to point out is that if it was due to only an intermittent failure, or you have many devices in your filesystem, it may be only a small fraction of your data and the particular extents that are degraded may be confined to a subset of all the devices in your filesystem.
Ok, so you're in degraded mode - now, another device fails, or you shutdown and restart and another device is missing. Can you mount, or will you be missing data?
Say you have six devices in your filesystem, doing two way replication, and originally device 1 failed - but then before you could finish a scrub pass device 5 fails. If you want to know if you can mount without having data missing, you need to know if there were any extents that were replicated on only devices 1 and 5 - which may or may not be true, depending on whatever policies were configured when you wrote the data.
To answer that, we need to track in the superblock (which is replicated across every single device, and replicated multiple times within a device) a list of "every unique set of devices extents are replicated across".
Once we've got that, that'll make a lot of replication-related things drastically easier.
After that's done, next thing will be dealing with degraded metadata due to write errors - still haven't decided what I'm going to do for that, probably something simple and stupid that may or may not get replaced later.
Last thing will be implementing scrubbing/rereplication - which will actually be the easiest thing to do out of everything I've described, we've already got data migration code. Main annoying part there will be deciding how to do the userspace interface.
Comments
Hi, Some formating will help to read the text, mean bold, title, I wish read only the section where is interressting for me
Kent Overstreet
They've diverged waaaaaay too much for that to be practical. (Believe me, I tried. I sunk like a month into trying to backport stuff, but the end result was a codebase that was greatly diverged from both bcache and bcachefs and would've required way too much time to stabilize enough to upstream).
NHOrus
How many bugfixes and improvements would backpropagate to bcache?
