Cow is the main feature of BTRFS.
Every write is a new 'version' of the file with only the change in it and a reference from the old file.
Snapshots will contain those 'versions' without adding much data to the disk.
Those 'Versions' are called extents (group of blocks)
defrag is the act of CREATING a new extents of the file with all update included inside which can grow your space usage if you have snapshots.
Because of CoW, DBs, VMs and logs/journal can be slower:
Files with a lot of random writes can become heavily fragmented (10000+ extents) causing thrashing on HDDs and excessive multi-second spikes of CPU load on systems with an SSD or large amount a RAM.
Quota groups applied to Subvolumes.
Modes allowing mount with damaged filesystem structures, all requires the filesystem to be mounted read-only and doesn’t allow remount to read-write. [2]
improvements to the background block group reclaim (BTRFS can over allocate memory when you write and this process help get that mem back),
automatically dropping the #QGroups of deleted subvolumes,
new modes of "rescue=" that allow to mount (as read-only) images that could have been partially converted by user space tools
- ignoremetacsums - invalid metadata checksums are ignored
- ignoresuperflags - super block flags that track conversion in progress (like UUID or checksums) userspace tool.
memory allocation optimizations, and other improvements[3]
there are some minor performance optimizations and fixes along with more folio API conversions [4]
new BTRFS_IOC_SUBVOL_SYNC_WAIT ioctl() command for the Btrfs filesystem will wait for the cleaning of one or more subvolumes. It is an unprivileged operation, and is intended to allow the "btrfs subvolume sync" command to work without privilege. TLDR New IOCTL that allow you to use the sync command that makes you wait until the deletion of a subvolume is done, this make sense if you have a backup sys that del a subvol and you want to know when you can check for how much place is left [5]
Btrfs now supports performing encoded reads (reading of compressed extents directly, without decompression) via io_uring.[6]
Some Btrfs configurations give the filesystem a choice of multiple devices (Raid 1) when the time comes to read a specific block. In current kernels, the PID of the reading process is used to make that decision, but that will focus all read traffic onto a single device in a single-reader workload. The 6.14 kernel adds a couple of new policy options that can implement either round-robin read balancing or simply focus reads onto a specific device. See this commit to set a specific device.[7]
Added negative compression level for ZSTD so -1 to -15 still getting fairly good result even far in the negatives, probably worth considering on weak configs. (image included in source)[8]
BTRFS getting faster, bunch of small optimization. [9]
Experimental Large Data Folios support. Like in EXT4 6.16, BTRFS now supports Large Folios for regular file. In testing with EXT4 it showed 37% in a synthetic test, but it should bring somewhere in the single digit perf gain in real world use (Still Very good!). For clarity, Large Folios are just folios that are utilized as they should (containing multiple pages) instead of having situations where we have multiple folios where not necessary. It is not a new struct.[10]
Btrfs now caches lookup results of the free space bitmap for around a 20% runtime improvement on an empty file creation benchmark.
Better sequential read performance readahead handling on compressed data.
Using denser keys for the XArray (think of a B-tree in an array) for extent buffers that leads to better packing of nodes and can yield around a 50~70% reduction of leaf nodes.[11]
When you have:
much larger than its memory limit
with no memory limit
The first cgroup will take the read lock and waste time preventing the secound cgroup from performing the right.
This was solved by making read take the read lock of the prior transaction, this is enabled only if the extent being read weren't changed.[12]
Initial support for block sizes (BS) greater than the kernel page size (PS). For now it is experimental and it doesn't have all the features we expect like Raid 5-6, no direct I/O and more...[13]
Adds Experimental Features, Continues Preparations For FSCRYPT[14]
In 6.18 we covered the Initial support for block sizes (BS) greater than the kernel page size (PS).
While still being experimental, the feature is getting more and more improvement, now joined with direct I/O.
Direct I/O allows a program to directly handle transactions with the File System, allowing better performance, especially with DBs.
Initial support for the remap-tree feature, We would need a full BTRFS video for a correct understanding of this feature but in short:
The Copy On Write nature of BTRFS requires management of a lot of metadata when doing relocation.
This load can be lighten by creating a remap-tree and adding a REMAPPED bit to block groups.
While improving performance, the change also brings New possibilities such as larger data extents and reducing
write amplification by removing cow-only metadata items.[15]
This feature is incompatible with the current BTRFS layout and is experimental, so don't expect it to apply to your system without some work.
Small optimization on compression and folio handling.
Switch to crypto library API for checksumming.[16]
Back in 6.17, we saw experimental Large Data Folios support.
After thorough testing, Large folios are now out of the experimental stage and are now enabled by default.
But now, lets make everything more confusing with the addition of:
HUGE FOLIO SUPPORT
Now just to defuse the confusion, Large folios are essentially just handling data without only using pages or single-page folios. (as it is very inefficient to do so)
But HUGE folios specifically refers to 2MB folios. Why 2MB?
Most architecture handle pages that are 4KB in size, these are ordered within a page table.
So here you have the standard page table hierarchy:
So each level contains a bunch of entries of the level under it.
Within the CPU, it is possible to omit the creation of lower level, allowing us to manage what we call a huge page.
As the Page Middle Directory contains 512 Page Table Entries, 512 x 4KB = 2MB giving us the magical number we were looking for.
also, keep in mind, bigger huge pages exist, 1GB is the step above.
Now, because we don't have to manage these 512 Page Table Entries, the handling of that memory is significantly faster.
Coming back to BTRFS, it is a good plan to introduce the handling of huge pages as it will boost performance with big files.
It is currently experimental, but i will keep you updated on the progress. [17]
2KB block size support was also removed, it blocked some of the folio improvement and didn't bring any benefits.[18]
Writeback inhibition's Xarrays replaced by CLOCK LRU.
What the hell did i just say? lets get into how BTRFS' Copy-On-Write works!
We know, BTRFS uses Copy-On-Write (COW in short),
What that means is that when a transaction is written, it is considered immutable (cannot change).
Lets run through a simplified write to get the gist of it.
struct extent_buffer to contain the metadata of the transaction. Within it, leave BTRFS_HEADER_FLAG_WRITTEN unset (to 0).
This flag makes it clear that we can still come back and modify the extent_buffer as we are still writing the data it manages.
extent_buffer again, so we write it to disk and set BTRFS_HEADER_FLAG_WRITTEN to 1, giving it its immutable status.
This is how, when writing synchronously, we get our metadata written to the disk.
But what about when we need to write asynchronously?
remember, when we say asynchronously, that means that it can be out of order.
So lets look at our example again.
extent_buffer, There is no 3rd part here for us to be waiting for.
We cannot wait until the write is done to ensure the metadata is perfect.
so instead, we place the extent_buffer in a dirty buffer, (a section of memory to be flushed/written to disk).
and when written, it will set BTRFS_HEADER_FLAG_WRITTEN to 1. (while its waiting it is still unset.)
Now, it can happen that the extent_buffer is written before the transaction is done, like on big files or deletions.
This would mean that metadata that is still changing is now immutable. This is not acceptable thus we need a new extent_buffer to complete the changes creating more useless metadata on disk.
This is called COW Amplification and it was mostly mitigated on 7.0 by Leo Martins who implemented an Xarray to inhibit writebacks. [22]
So the Xarray stores the _in progress_ extent_buffer and prevents them from setting BTRFS_HEADER_FLAG_WRITTEN ahead of time.
This was only applied to asynchronous transactions as it should.
At the time, in a test, we went from 35 COW being done to 5 using only this method, it was further reduce to 4 with another patch. It is fair to say that they fixed the issue.[22]
But here comes the catch, There is a cost to the management of the Xarrays.
This meant a 22% regression on write operation/sec, not good!
Here, we need to find a solution that is, not only fast, but also the closest possible to the perfect mitigation of the COW Amplification issue.
The right balance was struck using an LRU with a CLOCK eviction policy.
Lets go part by part:
The CLOCK eviction policy needs a bit more explaining, we will use struct btrfs_trans_handle which contains our LRU. [23]
...
/* Extent buffers this handle has inhibited writeback on. */
struct extent_buffer *inhibited_ebs[BTRFS_INHIBITED_EBS_SLOTS];
/* CLOCK reference bit per slot. */
u32 inhibited_ebs_referenced;
u32 nr_inhibited_ebs;
/* CLOCK hand. */
u32 inhibited_ebs_hand;
...
Lets start with a set amount of slots for the pointers (inhibited_ebs), in this patch, they use 8 (set by BTRFS_INHIBITED_EBS_SLOTS).
These pointers will point to our extent_buffers.
We then have the "2nd chance" referenced bits that are stored in inhibited_ebs_referenced.
This bitmap will allow us to know if an extent_buffer was touched.
nr_inhibited_ebs stores the number of filled slot.
and inhibited_ebs_hand stores the index of the CLOCK hand.
Now lets start writing new extent_buffers,
If the extent_buffer is already in the LRU, it will set the inhibited_ebs_referenced bit associated with the slot.[24]
Now that our slots are filled, we need to evict in order to add any new extent_buffers.
To do this we will look at the CLOCK hand to run:
> [!NOTE] The CLOCK hand is only affected by the following code.
If the inhibited_ebs_referenced bit associated with the slot = 1, set it to 0.
If the inhibited_ebs_referenced bit associated with the slot = 0, evict the current extent_buffer.
The evicted extent_buffer will then be set to be written to disk, it will be pushed to a dirty buffer.
Now this might look complex but it is trivial to run, and it is way lighter than maintaining an Xarray.
Now that we understand the change, what is the result?
| tracking | re-COW per first-COW |
|---|---|
| no inhibition | 6.1 |
| non-evicting array, 32 slots | 3.8 |
| CLOCK array, 8 slots (this patch) | 1.6 |
| unbounded xarray (reverted) | 1.4 |
We can see that the Xarray is better than the CLOCK LRU,
but the CLOCK will still give much of the benefits compared to the simpler non-evicting array.
Now, lets talk about performance, a report testing the CLOCK against the Xarray showed a 22% latency reduction in favor of the CLOCK. This would remove the performance regression and still provide the mitigation against COW Amplification.[29]
NEXT we have Stop sleeping for one jiffy in non-ssd mounts during log commit.
Lets go step by step:
CONFIG_HZ. By default, it would amount to 4ms, but it depends on the hardware and the kconfig (around 1ms to 10ms)...-o nossd mount option.Filesystems need to be compatible with calls like fsync().
When fsync() is used on a file, it guarantees that the file is written to storage.
Now, this is an issue for a Filesystem like BTRFS,
We have to manage multiple trees and subvolumes, if we were to have to perfectly write all of them to complete a fsync(),
we would be stalling our system for a fairly simple request.
Here, it would be fair to say, tough luck, you chose a Complex FS, pay the performance premium.
But the BTRFS team didn't like that, so they looked at fsync() and _probably_ thought, Why do we need to write it perfectly?
There is actually no need to complete a full transaction, with all the trees and subvolumes manipulation, in order to fully comply with the fsync() request.
Only that the file not be lost on a sudden crash.
So instead of a full transaction, lets leave the minimum Metadata in a separate tree and store the data.
If we crash, we will be able to read that tree and complete the transaction, without loosing the data.
The tree in question is the Log Tree, and it is what makes fsync() fast.
Lets come back to our patch: Stop sleeping for one jiffy in non-ssd mounts during log commit.
So why were we sleeping for a jiffy during these log commits?
The idea was to give a chance to bundle these log commits together making it more efficient.
The way it worked was that:
BTRFS_ROOT_MULTI_LOG_TASKS flag on the subvolume. This whole strategy was short sighted for 2 reason:
After diagnosing all of that, BTRFS Wizard: Filipe Manana knew what he had to do.
Strip out all of the code that makes log commit sleep a jiffy.
BTRFS_ROOT_MULTI_LOG_TASKS is Gone,
is Gone.
And a wild test appear, 5.7x performance gain. WOW!
I would normally say:
>Good job, Filipe!
But my friends, that patch 1 of 7 in the patchset... Lets see him cook some more!
log_batch which was there to detect when multiple task were logging,It would increment when logging,
so that after sleeping jiffy, if you see change in log_batch you know that you can batch things.
Spoiler, it would never batch things because it required a specific lock to do so, and it would never be able to get it so... RIP.
So byebye log_batch.
The patchset is not the only contribution that Filipe has offered, he also brought the BRUTAL:
This one is funny to me, is Filipe trying to state that 0=0? No.
Lets start by taking a look at why we would want to know if there is hole in the first place.
Back in the days... not that old XD, BTRFS-Progs-5.15 (November 2021, they are following the linux versions)[33],
Brought the "no-holes" option by default.
Before, we used to create a "hole" extent each time we would skip part of a file before writing.
For example, if we create a file, skip 500MB forward and than write 1kb of data, we would have a 500MB "hole" extent along with a 1KB extent for the written data.
so some BTRFS dev said to themself, i can cut that! he cut so deep, that you lost data on your RAID 5.
ok enough jocking, yes they removed the "hole" extent by assuming that if 2 extent are separated by a hole, this hole must be a phantom "hole" extent containing only zeros.
Lets go back to our Log Tree of earlier,
part of the minimum metadata we will need to restore in the event of a crash must include the "hole" extent.
If we don't have that information, we would not know the zeros existed or have to assume the random data that is currently in the "hole" extent is valid (which it isn't).
This means that we need to store the "hole" extents, if they exist, hence the hole detection.
The hole detection consist of scanning each extent of a file making sure that no holes exist between.
If you have a big file made of 20k extents, it will be costly!
So how can we know that a file doesn't have a hole without scanning it? lets look at the code!
BTRFS_INODE_PREALLOC Needs to be disabled. We cannot have pre-allocated extents because it breaks the following code:i_size The logical size of a file needs to be smaller or equal on-disk allocated bytes. /*
* If there are no prealloc extents (which can be located past i_size),
* and disk space used is greater than or equals to i_size, then there
* are no holes.
*/
if (!(inode->flags & BTRFS_INODE_PREALLOC) &&
i_size <= inode_get_bytes(&inode->vfs_inode))
return 0;
With that, we can be sure that there isn't a hole as all the bytes of the file are already allocated.
This is way faster than processing all the extents of a fragmented file and the tests show a 5.3x speedup after applying the patch, again, good job Filipe! [32]
BTRFS now uses IOMAP_DIO_BOUNCE for direct I/O rather than falling back to buffered I/O.
IOMAP is the bridge between the FS and the storage. It was created as a replacement for buffer-heads
AND
Buffered I/O VS Direct I/O
Buffered I/O is the default, it will use the page cache and writes will be written.... one day (The kernel handles it).
Direct I/O will bypass the page cache, it is a preferred solution for programs that manage their cache (DB servers).
By using Direct I/O, the DB server can achieve better performance as it doesn't have to deal with the complexity of the page cache.[21]
Now that we know all that, what is this IOMAP_DIO_BOUNCE?
With Direct I/O, if a write isn't aligned with the underlying software/hardware limitation, the write will fail.
For example, if you use dm-integrity (which makes checksums of the blocks on your disk) set to 4KB block size, a Direct I/O of 5KB would fail because it isn't following the 4KB standard.
IOMAP_DIO_BOUNCE allows for a write that isn't correctly aligned to be sent to a small buffer instead of failing. This is way faster than reverting back to the full buffered I/O.
Test showed that this change brings the performance from 50% of the theoretical limit up to 95%.
This is another example of something that is implemented but not yet used as this BOUNCE feature has been in the kernel since 7.0[19] [20]
So, enough yapping, lets get benching!
This is one of the most insane performance gain i have seen in only 1 release.
The most insane gain is with TigerBeetle, the financial transactions database, where we divide by 3.8 time the latency.
Un-hinge!
In the other test, it sometimes gives good gains or in some it will be within the margin of error of the previous version, so no big change here.
https://maplecircuit.dev/linux/fs/btrfs.html#fn-1" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">1]: [Gotchas - btrfs Wiki (kernel.org)
https://maplecircuit.dev/linux/fs/btrfs.html#fn-2" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">2]: [BTRFS SPECIFIC MOUNT OPTIONS — BTRFS documentation
https://maplecircuit.dev/linux/fs/btrfs.html#fn-3" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">3]: [F2FS, exFAT Btrfs File-System Changes In Linux 6.11 - Phoronix
https://maplecircuit.dev/linux/fs/btrfs.html#fn-4" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">4]: [Btrfs Sees Minor Performance Optimizations With Linux 6.12 - Phoronix
https://maplecircuit.dev/linux/fs/btrfs.html#fn-5" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">5]: [btrfs: add new ioctl to wait for cleaned subvolumes - kernel/git/torvalds/linux.git - Linux kernel source tree
https://maplecircuit.dev/linux/fs/btrfs.html#fn-6" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">6]: [btrfs: add io_uring command for encoded reads (ENCODED_READ ioctl) - kernel/git/torvalds/linux.git - Linux kernel source tree
https://maplecircuit.dev/linux/fs/btrfs.html#fn-7" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">7]: [The first part of the 6.14 merge window [LWN.net]
https://maplecircuit.dev/linux/fs/btrfs.html#fn-8" class="footnote-link is-unresolved" target="_self" rel="noopener nofollow">8]: [Btrfs Adding Fast/Realtime Zstd Compression Other Performance Optimizations - Phoronix
[28]: https://lore.kernel.org/linux-btrfs/12d3c3f07b8610ca13b0f3f792d420541afb7b33.1782949130.git.loemra.dev@gmail.com/
[29]: https://lore.kernel.org/linux-btrfs/CANGjgd=fQkHht2PdDi-+EAdzWH7UtxxWhhJ7b80Rr17PbpgxOw@mail.gmail.com/