Installing & Recovering Ubuntu Server 14.04 using BTRFS

Our new servers are all running Ubuntu Server 14.04 LTS.  This was chosen because it's free, it's fairly mainstream, and it's supported for security updates for 5 years between major releases (after which you can update in-place to a new major release).

An unexpected benefit of this system is that it supports BTRFS.  This was something new to me, but it turns out to offer a simple way to recover a system in the event of a disastrous update, and to recover lost files in the event of the user deleting them by mistake. 

The basic idea with BTRFS is that at any time you can declare a snapshot.  This happens instantly.  It's a point-in-time image.  It doesn't cost you any disk space, except in the sense that if you delete a file, it vanishes from the live filesystem but persists in any snapshots that were created earlier.

For our purposes, this is great stuff because it takes the risk out of patching your servers automatically.  Also, it makes it easy to recover lost data, especially for the mail server (where we need to hold 6 years of backups for legal reasons).

These features are useless unless you know that they exist - and how to drive them - hence I've written these notes to give a quick overview.

HOWTO: Install an Ubuntu Server with BRTFS

OK, you'll probably start by downloading the 64-bit Ubuntu LTS Server ISO file and creating a bootable USB stick from it.  There are several ways to make a bootable USB stick from an .ISO file, but if you're running an Ubuntu desktop, you can click Applications / System Tools / Administration / Startup Disk Creator and off it goes.

Be aware of an annoying little bug in the Ubuntu Server 14.04.2 LTS when installing from USB stick: the install script will only get through the first few screens before announcing that /cdrom is inaccessible (with a Retry/Cancel text dialog box).  This is easily fixed:
     - Use CTRL/ALT/F2 to flip to the second text console.  
     - Type 'mount' to see where the USB stick is mounted (e.g. /dev/sda1)
     - Unmount it from wherever it is, and remount it onto /cdrom :
            umount /dev/sda1
            mount /dev/sda1 /cdrom
     - Exit the shell (Ctrl/D)
     - Use CTRL/ALT-F1 to flip back to the main installer screen.
     - Select the Retry option.

So, you just install Ubuntu Server in the normal way, but when you get to the disk partitioning stage, don't select the "full auto" mode.  Go for manual disk partitioning instead. Don't worry, it's still fairly automated.  Choose to use the BTRFS file system in place of the default EXT4 filesystem.  You want to end up with a small swap partition (say 8 GB max) and a BTRFS root partition. Hit the save option and the installer will finish up as usual.

SO WHAT?  What can BTRFS do for me?  What's so cool about it?

OK, firstly let's install a helper tool to automatically snapshot the root filesystem before each apt-get operation...

    # apt-get install apt-btrfs-snapshot

Now check that all is well with snapshot facility:

    # apt-btrfs-snapshot supported
    Supported

OK that looks good: the option to create snapshots looks to be working.

Let's list the BTRFS subvolumes within our disk partition.

    # btrfs subvolume list /
    ID 257 gen 41 top level 5 path @
    ID 258 gen 35 top level 5 path @home

OK, to begin with, we have two subvolumes off the root filesystem.  "@" is everything except /home.  "@home" is the /home folder any everything below it.  This means that, if push comes to shove, you can roll back system-level changes without impacting users' home folders.

So watch what happens now when we patch the system:-

    # apt-get update && apt-get -y dist-upgrade
    ... 
    ... lots of updates happen
    ...

    # btrfs subvolume list /
    ID 257 gen 41 top level 5 path @
    ID 258 gen 35 top level 5 path @home
    ID 262 gen 38 top level 5 path @apt-snapshot-2015-05-11_13:21:42

Yay!  The apt-btrfs-snapshot package automatically took a snapshot of the root filesystem just before the updates were made.  

We can also find out about APT's automatic snapshots with a slightly different command:

    # apt-btrfs-snapshot list
    Available snapshots:
    @apt-snapshot-2015-05-11_13:21:42

Either way, we have a read-only backup of the root filesystem that we can revert to if we reboot and find the system won't work properly, or at all.

So how would we revert to a snapshot?  Suppose we reboot after taking those updates, only to find that the system is screwed.  
Let's boot from that USB stick again to rescue the system...

    - Pick "Rescue a broken system" from the Ubuntu installer menu
    - Follow the prompts during initialisation
    - At "Enter rescue mode" prompt, you will be prompted to choose a root file system.
        -> Choose: "Do not use a root file system"
        -> Then: "Execute a shell in the installer environment"

OK, if you type "mount" now, you can see which device that you booted from.  So you should be able to work out which device your installation is on, and mount that...

Example output:

    # mount /dev/sdb1 /mnt

    # ls -l /mnt
    drwxr-xr-x    1 root    root    216 May 11 12:22 @
    drwxr-xr-x    1 root    root    166 May 11 10:13 @apt-snapshot-2015-05-11_13:21:42
    drwxr-xr-x    1 root    root     10 May 11 10:16 @home

So if we want to revert to that automatic apt-snapshot of the root filesystem, we just rename the live one to something else, then rename the snapshot to the live one.  Then reboot.

    # mv /mnt/@  /mnt/@-broken-root

    # mv /@apt-snapshot-2015-05-11_13:21:42 /mnt/@

    # reboot

Errr, that's it!  When you reboot, you are using the root filesystem as it was before the failed update.  But you can still access the broken system by mounting it elsewhere.  To see how this might be done, use the mount command to check the physical drive, and the 'btrfs subvolume list /' command to view the subvolumes available:-

    # mount
    ...
    /dev/sda1 on / type btrfs (rw,subvol=@)
    /dev/sda1 on /home type btrfs (rw,subvol=@home)
    ...

OK so if we want to mount our brokenroot snapshot, we can do so like this:

    # btrfs subvolume list /
    ID 257 gen 57 top level 5 path @-broken-root
    ID 258 gen 61 top level 5 path @home
    ID 262 gen 66 top level 5 path @

OK so we can take a look at the failed folder like so:

    # mount -o ro,subvol=@-broken-root /dev/sda1 /mnt

    # ls -l /mnt

If we want to zap it, unmount it, mount the parent volume, and zap it:

    # umount /mnt

    # mount /dev/sda1 /mnt

    # mount
    /dev/sda1 on / type btrfs (rw,subvol=@)
    ..
    /dev/sda1 on /home type btrfs (rw,subvol=@home)
    /dev/sda1 on /mnt type btrfs (rw)

    # ls -l /mnt
    drwxr-xr-x 1 root root 166 May 11 11:13 @
    drwxr-xr-x 1 root root 216 May 11 13:22 @-broken-root
    drwxr-xr-x 1 root root  10 May 11 11:16 @home

    # btrfs subvolume delete /mnt/@-broken-root
    Delete subvolume '/mnt/@-broken-root'

We can also take snapshots of a subvolume at any time.  So we would snapshot the home directory like so:

    # btrfs subvolume snapshot -r /home /snapshots/home-2015-05-11
    Create a readonly snapshot of '/home' in '/snapshots/home-2015-05-11'

Probably best to keep all snapshots read-only to avoid confusion.

You can list all the snapshots of a volume like so:-

    # btrfs subvolume list /

ID 258 gen 79 top level 5 path @home
ID 261 gen 34 top level 5 path @home/itshc/home-TEST
ID 262 gen 80 top level 5 path @
ID 263 gen 78 top level 5 path @home/itshc/home-2015-05-11
ID 264 gen 79 top level 262 path tmp/home-foo

The only thing that annoys me slightly is that the snapshots get mounted at the mount point you specified when you took them.  I think it would be simpler if they were created but not mounted anywhere until requested.  It's probably best to stick them into a '/snapshots' folder to avoid confusion!  

A better way to take snapshots seems to be to temporarily mount the base device onto /mnt and take the snapshots from there...

# mount /dev/sda1 /mnt

# btrfs subvolume snapshot -r /mnt/@home /mnt/@home_snapshot_2015_05_14
Create a readonly snapshot of '/mnt/@home' in '/mnt/@home_snapshot_2015_05_14'

# umount /mnt

# btrfs subvolume list /
ID 258 gen 92 top level 5 path @home
ID 262 gen 91 top level 5 path @
...
ID 270 gen 95 top level 5 path @home_snapshot_2015_05_14

This avoids all your snapshots showing up until you actually mount them.

BTW, some forums reckon it's best to mount BTRFS filesystems as 'noatime', particularly if there are lots of snapshots, to prevent unnecessary copy-on-write operations.  But I can't see why that's needed if you make sure that all your snapshots are read-only.

Grateful for any pointers from sysadmins who are already using BTRFS in anger.