bcachefs
patreon
Tiering is dead; long live disk groups
🕑 Added 2018-02-20 21:03:09 +0000 UTC The new disk groups-based code for configuring data placement has been merged, and the notion of configuring disks into "tiers" has been removed. If you have an existing filesystem that uses tiering, you'll have to configure the new interfaces.The reasoning behind the change was that a "disk tier" wasn't really a thing - it was just a hint to a couple different parts of the IO subsystem as to where they should put data and how they should move it around. Instead of having one hint - that was effectively a global setting - we're now exposing settings that correspond to what we're actually doing. As a bonus, these new settings can be overridden for individual files/directories.
The things tiering actually did, and the new equivalents, are:
* It was a hint to read path - it told the read path which disks were faster so it could prefer those. Now, we instead just track the recent IO latency to each device so we know which devices are faster.
* Foreground writes, and metadata allocations, would prefer to use devices in the fastest tier. This is now controlled with the foreground_target option.
* In the background, the tiering thread would look for dirty data on the faster tier and copy it to the slower tier (leaving the original data there, but marking it as cached). This is now controlled with the background_target option; if that option is set, in the background that data will be copied to disks in background_target (as before, leaving the original data in place but marking it cached).
* When reading, if there is no copy of the data we're reading in the fastest tier, we'd write a cached copy of that data to the fastest tier. This is now controlled with promote_target.
So, the old recipe to format /dev/sda and /dev/sdb using /sda as a writeback cache was:
bcachefs format --tier 0 /dev/sda --tier 1 /dev/sdb
The new equivalent is:
bcachefs format /dev/sd[ab] --foreground_target /dev/sda --background_target
/dev/sdb --promote_target /dev/sda
The new options can also be changed at runtime, via sysfs in the /sys/fs/bcache/*/options dir. (They can't be specified at mount time like most other options because when mount options are parsed we don't have what we need to look up disk groups).
Disk groups come in because we need a new way to group disks, and a way for the new settings to refer to more than one disk. A disk can be in at most one "disk group", and you refer to disk groups by label. You don't need to bother with disk groups if you don't actually need to refer to more than one disk - all the new settings that take disk groups can also be passed an individual disk.
The example recipe using disk groups would be:
bcachefs format \
--group ssd /dev/sda /dev/sdb \
--group hdd /dev/sdc /dev/sdd \
--foreground_target ssd \
--background_target hdd \
--promote_target ssd
Note that you don't have to use the options this way - e.g. you could specify /dev/sdb for the foreground target and /dev/sda for the promote target to use /dev/sda as a writearound cache.
There's still more options we'll need to add - for example, we don't yet have a way to specify writethrough caching. For that we need a way to specify that foreground writes should write dirty data to one target, and also a cached copy to another target. But we now have a lot of the infrastructure in place to do lots of interesting things. Other notes, gotchas:
Foreground_target doesn't restrict foreground writes to only use that target; if the device(s) in that target are full the allocation will fall back to other devices in the filesystem - for writethrough caching, we'll need some way to restrict a device to only contain cached data. We do have a --data_allowed option that can be used to restrict the types of data a device can contain (data, btree or user data) - but it can't (yet) be used to restrict a device to only contain cached data.
We don't yet have a mechanism for caching metadata. For writeback caching this is no big deal - you'll probably want your metadata to always live on the fast device anyways - but for writethrough caching this is a bit of a problem (though probably not a huge deal in practice, since bcachefs metadata is fairly compact so it should mostly be cached in RAM).
There was an option added recently, background_compression, to specify that data should be compressed or recompressed with that algorithm in the background - but when the option was added, nothing would actively look for data to be recompressed, it would only be used when tiering or copygc were moving the data around. The new rebalance thread (that replaces the old tiering thread) does actively compress data that isn't compressed with the algorithm specified by the background_compression option.
And, probably the biggest one to be aware of - the ratelimiting algorithm for the rebalance thread is definitely going to need more work. The one big advantage of the tiering approach is that we always had an accurate count of how much work there was for tiering to do - this is no longer really possible (in particular because all these options can be changed at runtime).
So, while you're testing the new functionality please keep an eye on it and report any misbehavior, e.g. not keeping up or spinning and using too much cpu - you can use bcachefs fs usage to see how much data is on all the component devices.
Comments
Just wondering, are these addressed/implemented in the current codes?
Kent Overstreet
Not currently... would have to teach the background threads (copygc/rebalance) to be aware of it and not wake the drive up unnecessarily. Good wishlist feature, though
Just to make sure - does it play well with hdparm -S ? I would like to get my HDD group asleep (e.g. silent) almost all the time, because SSDs should be enough for majority of operations. (The scenario is storage of media. 2 2TB SSDs are not enough to store all the media with reduncy of 2 but more then enough to store all the media that were touched in a last year or two).
Kent Overstreet
That is the idea. Unfortunately it's broken right now though - we need something added to the on disk format for extents to indicate that it was incompressible. I have a patch to add a fake compression type, but I haven't merged it in yet because I haven't decided if I want to go with that approach or not...
Re: background_compression: So this could be used, for example, in a (for lack of a better word) "tiered" compression scheme where when writing data initially, it uses a cheap algorithm like lzo for fast writes, then later when idle a background thread could re-compress with a higher-ratio but more expensive algorithm like zpaq or brotli on the backing device?
