Why is umount so slow? Why does it take so long?

Part of the answer was already given in an earlier blog post; here’s some more explanation.

The write() syscall typically writes into RAM only. In Linux we call that “page cache“, or “buffer cache“, depending on what exactly the actual target of the write() system call was.

From that RAM (cache inside the operating system, high in the IO stack) the operating system does periodically do writeouts, at its leisure, unless it is urged to write out particular pieces (or all of it) now.

A sync (or fsync(), or fdatasync(), or …) does exactly that: it urges the operating system to do the write out.
A umount also causes a write out of all not yet written data of the affected file system.

Note:

• Of course the “performance” of writes that go into volatile RAM only will be much better than anything that goes to stable, persistent, storage. All things that have only been written to cache but not yet synced (written out to the block layer) will be lost if you have a power outage or server crash.
  The linux block layer has never seen these changes, DRBD has never seen these changes, they cannot possibly be replicated anywhere.
  Data will be lost.

There are also controller caches which may or may not be volatile, and disk caches, which typically are volatile. These are below and outside the operating system, and not part of this discussion. Just make sure you disable all volatile caches on that level.

Now, for a moment, assume

• you don’t have DRBD in the stack, and
• a moderately capable IO backend that writes, say, 300 MByte/s, and
• around 3 GiByte of dirty data around at the time you trigger the umount, and
• you are not seek-bound, so your backend can actually reach that 300 MB/s,

you get a umount time of around 10 seconds.

Still with me?

Ok. Now, introduce DRBD to your IO stack, and add a long distance replication link. Just for the sake of me trying to explain it here, assume that because it is long distance and you have a limited budget, you can only afford 100 MBit/s. And “long distance” implies larger round trip times, so lets assume we have a RTT of 100 ms.

Of course that would introduce a single IO request latency of > 100 ms for anything but DRBD protocol A, so you opt for protocol A. (In other words, using protocol A “masks” the RTT of the replication link from the application-visible latency.)

That was latency.

But, the limited bandwidth of that replication link also limits your average sustained write throughput, in the given example to about 11MiByte/s.
The same 3 GByte of dirty data would now drain much slower, in fact that same umount would now take not 10 seconds, but 5 minutes.

You can also take a look at a drbd-user mailing list post.

So, concluding: try to avoid having much unsaved data in RAM; it might bite you. For example, you want your cluster to do a switchover, but the umount takes too long and a timeout hits: the node (should) get fenced, and the data not written to stable storage will be lost.

Please follow the advice about setting some sysctls to start write-out earlier!

As you know DRBD marks the to-be-changed disk areas in the Activity Log.

Until now that meant that for random-write workloads a DRBD speed penalty of up to 50%, ie. each application-issued write request translated to two write requests on storage.

With DRBD 8.4.3 Lars managed to reduce that overhead¹, from 1:2 down to 64:65, ie. to about 1.6%. (In sales speak “up to 64 times faster” )

Here are two graphics showing the difference on one of our test clusters; both using 10GigE and synchronous replication (protocol C):

The raw LVM line shows the hardware limit of 350 IOPS; while 8.4.2 and 8.3.15 are quickly limited by harddisk seeks, the 8.4.3 bars go up much further – in this hardware setup we get 4 times the randwrite performance!

When using SSDs the difference is even more visible ­— the 8.4.2 to 8.4.3 speedup is a factor ~16.7.

Again, the raw LVM line shows the hardware limit of 50k IOPS; 8.4.2 needs to wait for the synchronous writes (at 1.5k IOPS), but 8.4.3 gives 25k IOPS, at least half the pure SSD speed.

Please note that every setup is different — and storage subsystems are very complex beasts, with many, non-linear, interacting parts. During our tests we found many “interesting” (but reproduceable) behaviours – so you’ll have to tune your specific setup^2,3.

Furthermore, the activity log can now be much bigger⁴; but, as the impact on performance of leaving the “hot” area is now very much reduced, you may even want to lower the al-extents – ie. tune the AL-size to the working set, to reduce re-sync times after a failed Primary.

And, last but not least, the AL can be striped – this might help for some hardware setups, too.
Please see the documentation for more details.

BTW: these changes are in the DRBD 9 branch too, so you won’t lose the benefits.

That was not that complicated, really; as our core product DRBD works on (nearly) every Linux distribution, we simply

1. live-migrated all VMs to one of the nodes;
2. reinstalled the root filesystem on the other node with RHEL 6.3¹ and configured GRUB to boot into that one;
3. installed matching DRBD modules
4. waited a few seconds for the resync to complete (which was really that fast, because we didn’t touch the existing logical volumes, and so the changed data were only a few GiB);
5. and then let Pacemaker take control over the cluster again, allowing us to migrate the VMs to the newly installed node. Without any service interruption.

The key to this was that DRBD and Pacemaker are available in compatible versions on most current distributions — and that’s not a big problem, because we make such packages available for our customers in our repositories.

Upgrading DRBD from 8.3 to 8.4 at the same time is only a small, secondary change; after all, its network code can talk to different versions by design.

As 2π is proposed to be named τ we’ve chosen the name “Raspberry Tau” for this proof-of-concept.

We’ve connected two Raspberry Pis via their on-board ethernet interfaces (via a switch, so we can simply SSH into them), booted via 2GB SD-cards with a Raspbian image on them. After upgrading to a kernel that has kernel-headers available we built DRBD modules, and voilá! A Raspberry Tau cluster is born.

We’re replicating the data on the USB-Sticks; their performance nicely matches the available network. Here’s /proc/drbd (shortened and line-wrapped for readability):

root@raspberry-alice:~# cat /proc/version 
Linux version 3.2.0-3-rpi (Debian 3.2.21-1+rpi1) 
  (debian-kernel@lists.debian.org) (… Debian 4.6.3-1.1+rpi2)…)
root@raspberry-alice:~# cat /proc/drbd 
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
    root@raspberry-bob, 2012-09-18 12:58:08
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:805304 nr:0 dw:348628 dr:818596 al:127 bm:70 lo:0 pe:0
      ua:0 ap:0 ep:1 wo:d oos:0

As Raspbian is Debian-based, there are Pacemaker (and Heartbeat resp. Corosync) packages available … so a cheap, low-power, High-Availability cluster is easily built.

Disclaimer: for a real HA-cluster you’d need a few more things.

• a STONITH device (if power is supplied via a Linux-PC, you could turn off the USB port by software), and
• redundant network connectivity (USB ethernet adapter).

Of course, if you’re just clustering your media library these things might not be mandatory.

Packages are available for everyone – just drop an email to sales@linbit.com, and we will be happy to provide them.

For DRBD 8.4.2 we’ve removed the double-stacked check in the userspace tools; now it’s easier to do a multi-stacked setup.

A picture says more than a thousand words; please see here:

The special thing here is that the dashed DRBD backup connection is not always connected; via a cronjob the backup node disconnects itself, does a (file-based) backup of the data to another storage (tape, harddisk, DVD-RW, paper tape…), and reconnects afterwards – to get the newer data, by a standard synchronization.

The advantage is that the IO load on the backup node has no impact on the primary node – you can even start a local database, and do some CPU- and IO-intensive evaluations. As the backup node is not connected to the HA cluster it cannot slow down the normal operation.

The backup itself can be taken with automatic LVM Snapshots, which prevents a split brain situation when mounting the backup drbd resource – or you need to do a drbdadm connect --discard-my-data before re-connecting.

And: all of that is possible while having a continuous connection to the disaster-recovery site.

Credit for this also goes to Mark Olliver from Thermeon, who was involved in testing and developing this setup.

Basically, the first setup is having two servers, one of them being actively driving a DM-mirror (RAID1) over (eg.) two iSCSI volumes that are exported by two SANs; the alternative is using a standard DRBD setup. Please note that both setups need some kind of cluster manager (like Pacemaker).

Here are the two setups visualized:
The main differences are:

So the Open-Source solution via DRBD has some clear technical advantages — not just the price.

And, if that’s not enough — with LINBIT you get world-class support, too!

DRBD is designed as a storage solution to provide High Availability, Disaster Recovery and Cross Site High Availability to your systems.  As developers of DRBD, we sometimes get community feedback that some folks are using DRBD as a “pseudo” backup solution, and in response to this we wanted to share some abstract guidelines on utilizing DRBD properly by following some key best practice methodologies.

Although DRBD is not backup software, it doesn’t mean you can’t use it in your backup procedures. Utilizing DRBD with LVM as a backing device, one can create backups with minimal to no interference to performance. This is done by utilizing LVM snapshotting as outlined in LINBIT’s DRBD User’s Guide.  Although this page outlines how to do snapshots before and after a resync, these could easily be adapted to a cron job.  Essentially one would disconnect the Secondary, snapshot the backing device, mount the snapshot, perform the backups, umount the snapshot, reconnect the Secondary.  These point in time backups are great for technology such as iSCSI targets, Virtual Machine storage or Databases such as MySQL and PostgreSQL.  As you can imagine, this methodology is quite popular in the Linux HA and DRBD communities.

LINBIT advises systems administrators to:

1. Utilize DRBD for High Availability, Disaster Recovery and Cross Site High Availability (business continuity) purposes.
2. Plan, review and execute a full backup strategy that makes sense for your organization and data.   Be sure to keep in mind how much data you’re planning on storing, backing up and at what intervals.  It is important to choose the point in time to make your backups to minimize things such as user error.  In many cases, backing up every day is the appropriate strategy.
3. Test, test, test.  We cannot say this enough.We develop software that is designed to prevent loss from failure, so you could say we’re experts on this topic.  It’s very important that you not only test DRBD’s configuration, but the components that make up your backup system as well.  Then, on a scheduled basis, you should be reviewing your data to ensure its completeness and correctness.  As well, on an annual basis it would be wise to review your top level strategy and make updates if your requirements have changed.  In summation, it is advised to routinely test your backup procedure and also verify (checksum) your backups to ensure their completeness.

In closing, DRBD is designed to prevent loss of service as the result of equipment failure.  LINBIT strongly advises systems administrators to implement a strategy that incorporates “point in time” backups so administrators can restore, rewind and rejoice knowing that they’re not only backed by the best open source replication technology: DRBD, but a comprehensive backup solution that is designed for the organization’s needs in mind.

How do you backup your DRBD cluster?

Share your thoughts or comments below!

While writes occur on both sides of the cluster, by default the reads are served locally (ie., the value is prefer-local). This might not be optimal if you’ve got a big pipe to the other node and a heavily loaded IO subsystem.

read-balancing has several options to choose from:

• 32K-striping up to 1M-striping chooses the node to read from via the block address – eg. for 512K-striping the first half of each MiByte would be read from one machine, and the second half from the other¹.
  This is a simple, static load-balancing.
• round-robin just passes the request to alternating nodes.
  This might go wrong if your application reads 4kiB, 1MiB, 4kiB, 1MiB, and so on – but this is fairly unlikely.
• least-pending chooses the node with the smallest number of open requests.
• when-congested-remote uses the remote node if there are local requests².
• prefer-remote is implemented for completeness, however as of this writing there is no viable use case.

Please note that all this is still below the filesystem layer – so even if the secondary is used for reading, this won’t speed up a failover, as the pages read are not kept anywhere.

Be strict in what you emit, liberal in what you accept¹

is simply not true when dealing with mission-critical systems.

It’s ok to be alerted on upgrading a machine because the “old, working” RegEx that did the parsing doesn’t match anymore²; it’s not a problem to get an email when someone adds the 100th DRBD resource and causes the grep to fail; and so on.

Better to have a few false positives when you’re actively changing things than to get a false negative that costs you months of data; that’s what an assert (and monitoring isn’t that different) is for, after all.

Keep monitoring strict, and let it fail loudly on unexpected things – after the first few occurrences they’re not unexpected anymore and can be dealt with.