Set up bcache to speed up game loading with an HDD/SSD combo

Hello! I decided to write this guide on setting up tiered storage for a game library, as I recently went through figuring it out.

I moved over from windows some time ago, and I used to use PrimoCache to speed up my 6TB game library using a secondary 500GB nvme SSD as cache. (My primary 250GB being my OS drive.)

I'd been wanting something equivalent to it on linux, and I wasn't able to find anything straight forward. This guide aims to make something complicated, a little simpler to set up for the same use case. This is still sorta advanced stuff and we’ll be working entirely in the terminal, but if you've got an SSD with free space and an empty HDD, you should be able to follow along.

After looking around for options, I ended up with lvmcache and bcache, both of which have kernel level support.

But I eventually chose to go with bcache for a couple reasons:

• lvmcache does not support being used in a read-only mode

• bcache can be tuned to accelerate game loading much more reliably (as far as I can tell)

• bcache, especially after tweaks, performs better

The results have been excellent! I did some testing with Satisfactory, loading into a game from the main menu, closing the game completely between each time.

Installed on HDD:

1. 1:09

2. 1:11

3. 1:10

Installed on SSD:

1. 0:22

2. 0:22

Installed on Bcache SSD/HDD:

1. 1:08

2. 1:00

3. 0:48

4. 0:45

5. 0:42

6. 0:38

I stopped here but the cache hitrate was still increasing each time.

I am on Manjaro. But this should work on any distro if you get past step one.

Note that an SSD being used as a cache, will experience somewhat more wear, depending on how often it ends up swapping out what is cached. This means it is likely to die sooner than it would otherwise, something to keep in mind.

Also, I would not set this up with a cache that is smaller than your largest game. Or perhaps even twice that size (100GB). Due to how aggressively this setup will have games be cached, a small cache will simply experience tons of wear, without providing much benefit.

If you want to go ahead anyway, I recommend choosing less aggressive settings in step 5.

Step 1: Install tools

Bcache itself is part of the linux kernel (since 3.10), but to set things up we need to install bcache-tools from AUR. On an Ubuntu based distro, it should be available via apt.

yay bcache-tools

There is also a very handy python script for monitoring bcache, here. Save it, place it in /usr/local/bin and make it executable, to easily run it in the terminal when needed. Or place it wherever and call it through its full path.

Step 2: Prepare your drives

If you want to use an HDD partition that already has data on it, you'll need to convert it using this.

This is not a super reliable process, so back up anything important before attempting it (such as compatdata in steamapps for save files). I have not tried it myself, and as such can’t include instructions, but the github page seems to have all the details needed to give it a try. Please do comment on how it’s done, if you do.

If you want to go the recommended route of starting afresh, you'll want your SSD/s and HDD/s to be either unpartitioned, or if you want to only use part of your SSD/s/HDD/s, prepare a partition on each that is unformatted.

For example, if you only have one SSD which you are booting from, shrink your OS partition and create another unformatted partition on it. (This is sorta what I did, as my secondary SSD also contains a tiny windows partition)

Step 3a: Create the cache devices using new volumes

When creating the backing(HDD) and cache(SSD) devices, you'll be entering “block” and “bucket” sizes. These should correspond to your HDD sector size, and SSD erase block size, respectively. The latter of which is a painful datapoint to find. I just made a guess at mine being 16M. Even if you don't get it right, things will work, but your SSD might experience some more wear.

To create the cache device, run the command:

make-bcache -B /dev/sdX --block 4k --bucket 16M -C /dev/nvme1nXpX

Where "/dev/sdX" is your HDD or partition on it and "/dev/nvme1nX" is your SSD or partition on it.

You can also add more than one of either. If you want to control what volumes cache what other volumes, or want more complex setups using multiple drives, the Arch wiki has some good config examples. If you have multiple backing devices, each will appear as its own volume, named “bcache<N>”. You’ll see “bcache0” in many commands, if you have multiple hard drives, any such command will only apply to one of them.

Once the command completes, check if bcache is doing ok, with the command:

cat /sys/block/bcache0/bcache/state

If it returns "clean", it is, and you can format the volume using the line bellow, replacing "volume-label" with whatever you want to call it:

mkfs.ext4 -L volume-label /dev/bcache0

You have to do this in the terminal, as the cached volume will not show up in partition managers. If you’d like to use some other file system, simply substitute the command with the relevant equivalent (mkfs.btrfs for example).

Step 3b: Create the cache device after converting an existing volume

If you converted your existing data, there's slightly more to do.

You'll be entering “block” and “bucket” sizes. These should correspond to your HDD sector size, and SSD erase block size, respectively. The latter of which is a painful datapoint to find. I just made a guess at mine being 16M. Even if you don't get it right, things will work, but your SSD might experience some more wear.

Turn your SSD into a cache device using the command:

make-bcache --block 4k --bucket 16M -C /dev/nvme1nX

Where "/dev/nvme1nX" is your SSD or partition on it. Once that is done, you'll need to attach it. First run:

sudo bcache-super-show /dev/nvme1nX | grep cset

To find out the UUID of the cache volume, then use that in the following command to attach it:

sudo bash -c 'echo UUID > /sys/block/bcache0/bcache/attach’

Now, check if things worked out with the command:

cat /sys/block/bcache0/bcache/state

If it returns "clean", things should be working, and you can mount the volume!

Step 4: Mount the volume and add fstab entry

Now that there is a cached volume available at /dev/bcache0, we need to mount it, unless your system did already... Simply run:

mount /dev/bcache0 /path/to/mountpoint

We don’t want to do that at every boot, of course. So to create an fstab entry, we’ll want the UUID of the cached volume. Run:

sudo blkid | grep bcache0

Then edit fstab:

sudo nano /etc/fstab

And add a line along the lines:

UUID=the-uuid   /path/to/mountpoint    ext4    nosuid,nodev,nofail,exec    0 0

Hit Control+X to close, press Y to save the change.

Test that the mount works:

mount -a

Step 5: The important part, tweak the bcache config

Now bcache is up, running and mounted. Done? No.

I noticed in my testing that at this point, bcache provided essentially zero performance benefit. It was doing nothing. By default, bcache will cache low latency random read/writes, entirely ignoring large sequential IO. That’s great if you are using it to run an operating system off it, but not so much games.

In fact, if you started playing games, and used bcache-status to check what it was doing, you’d find it would barely be moving anything into cache, and what little it does, it would still access through the HDD instead.

For games, we probably want it to not cache writes, and definitely cache large reads at game file sizes. But these are high latency sequential reads.

Lucky for us, bcache can do just that. To enable read-only operation, run the command:

sudo bash -c 'echo writearound  > /sys/block/bcache0/bcache/cache_mode’

Check that it got applied:

cat /sys/block/bcache0/bcache/cache_mode

The other available modes are the default “writethrough”, which both writes to the HDD, and caches the write while doing so. “writeback”, which writes to the SSD for speed, then slowly transfers the data to the HDD (potential data loss when SSD fails). And “none” which turns caching off. “writearound” does not write to the SSD at all, and only starts caching things when read from the HDD. This is ideal, as we don’t want every install of a new 80 gig game to clear an equivalent amount of stuff out from the cache.

The next command is:

sudo bash -c ‘echo 0 > /sys/block/bcache0/bcache/sequential_cutoff’

Check that it got applied:

cat /sys/block/bcache0/bcache/sequential_cutoff

This disables bcache’s sequential IO detection. By default, this value is set to 4M, meaning any IO operations larger than 4 megabytes won’t be cached. That means our large game files would never benefit. But turn it off, and they will. You could also set this value to simply something large, if you would still like to keep some amount of sequential detection. For a game library, it is likely pointless, but you may want to do this if your SSD is not that large, to limit how much each game hogs up cache.

Next, to find the cache set uuid, run:

sudo bcache-super-show /dev/nvme1nX | grep cset

Then use it for:

sudo bash -c ‘echo 0 > /sys/fs/bcache/<cachesetuuid>/congested_read_threshold_us’

Check that it got applied:

cat /sys/fs/bcache/<cachesetuuid>/congested_read_threshold_us

This will disable bcache’s read latency detection. In some cases, SSDs can be overrun by IO requests, which slows down their response time. In these cases, at a certain threshold, bcache deliberately falls back to reading from the HDD. But we don’t care about latency that much, we want BANDWIDTH.

Especially while loading a ton of game files, the latency of an SSD will suffer, and without this disabled bcache will in those situations read from the HDD. Which might lead to lower latency, but it probably won’t, and it certainly won’t load the game any faster.

Lastly:

sudo bash -c 'echo 4096 > /sys/block/bcache0/queue/read_ahead_kb'

Check that it got applied:

cat /sys/block/bcache0/queue/read_ahead_kb

This will set a readahead value, which by default is zero. I set an even larger value, at 16384. Essentially, bcache will round up to this value, when caching blocks onto the SSD, so that it moves fewer larger chunks over, rather than many small ones. The rounding up will also ensure, that more data that might be needed on future reads, is likely to be there. Again, if your SSD is small, you may want to use a smaller value here.

With these tweaks done, you should now notice the games you play a lot loading faster and faster!

Step 6: Making the tweaks permanent

Those last three are not persistent settings. So we need to configure them to apply on boot.

sudo nano /etc/tmpfiles.d/bcache.conf

Enter the settings like so, remembering to use the UUID from the latency command, and changing the numbers to the right to the values you've decided to use:

w /sys/block/bcache0/bcache/sequential_cutoff                    - - - - 0

w /sys/block/bcache0/queue/read_ahead_kb                         - - - - 16348

w /sys/fs/bcache/<cache set-uuid>/congested_read_threshold_us    - - - - 0

Control+X to exit, press Y to save the changes.

You should now be all set to start using your accelerated game library! And if you installed the bcache-status python script, you can use the command below to monitor how your cache is doing.

bcache-status -a

Add “watch” if you want live updating numbers, if for example you want to see how bcache does as you load up a game.

watch bcache-status -a

Enjoy!

Binary Package: https://yum.oracle.com/repo/OracleLinux/OL9/baseos/latest/aarch64/getPackage/bcache-tools-1.0.8-3.101.0.1.el9.aarch64.rpm

Source Package: https://yum.oracle.com/repo/OracleLinux/OL9/baseos/latest/aarch64/getPackageSource/bcache-tools-1.0.8-3.101.0.1.el9.src.rpm

Mirror: yum.oracle.com

Install Howto

Install bcache-tools rpm package:

# dnf install bcache-tools