While writes occur on both sides of the cluster, by default the reads are served locally (ie., the value is prefer-local). This might not be optimal if you’ve got a big pipe to the other node and a heavily loaded IO subsystem.

read-balancing has several options to choose from:

• 32K-striping up to 1M-striping chooses the node to read from via the block address – eg. for 512K-striping the first half of each MiByte would be read from one machine, and the second half from the other¹.
  This is a simple, static load-balancing.
• round-robin just passes the request to alternating nodes.
  This might go wrong if your application reads 4kiB, 1MiB, 4kiB, 1MiB, and so on – but this is fairly unlikely.
• least-pending chooses the node with the smallest number of open requests.
• when-congested-remote uses the remote node if there are local requests².
• prefer-remote is implemented for completeness, however as of this writing there is no viable use case.

Please note that all this is still below the filesystem layer – so even if the secondary is used for reading, this won’t speed up a failover, as the pages read are not kept anywhere.

Be strict in what you emit, liberal in what you accept¹

is simply not true when dealing with mission-critical systems.

It’s ok to be alerted on upgrading a machine because the “old, working” RegEx that did the parsing doesn’t match anymore²; it’s not a problem to get an email when someone adds the 100th DRBD resource and causes the grep to fail; and so on.

Better to have a few false positives when you’re actively changing things than to get a false negative that costs you months of data; that’s what an assert (and monitoring isn’t that different) is for, after all.

Keep monitoring strict, and let it fail loudly on unexpected things – after the first few occurrences they’re not unexpected anymore and can be dealt with.

I want to use a 10TiB volume with DRBD, is that supported”?

The easiest way to answer things like that is to say look for yourself on the public DRBD usage page – the biggest public device size is ~220TiB, so go figure

The current maximum device size is 1EiB (1 ExiByte = 1024 TibiByte¹), so there’s a bit of room left.

DRBD needs about 32MB RAM per TB storage, so for 1 EB storage you’ll need 32GiB of RAM just for the DRBD bitmap². Having a bit more for the OS, userspace and buffer cache is left as an exercise for the reader.

If you’ve got questions, ask the DRBD experts at LINBIT – we wrote the code, after all!

But, as per the old saying Trust, But Verify¹ it might be a good idea to periodically test whether the nodes really have identical data, similar to the checks that are² done for RAID sets.

The verify-alg digest is used to save bandwidth during online verification; while without this setting the whole data has to be transferred³, a value of md5 means that only 20 bytes are needed for each 4KiByte block, resulting in bandwidth savings of about 99.5%.

The DRBD Users’ Guide has a nice chapter describing the configuration and usage, so I won’t get into this topic here.

If the volume you’re checking is actively used, you might see a few false positives in the log messages:

kernel: block drbd0: Out of sync: start=56079768, size=8 (sectors)

This is because some data block writes might have just been on the wire while the two nodes calculated their checksums, and so would see different generationsof the data. If you do this check eg. every week and get different block numbers every time, you’re fine. If you get the same block number(s), your storage might have stuck bits, and be unable to correctly write data in these blocks!

Please note that the needed verify-alg setting here sounds similar to the data-integrity-alg option, but serves a different purpose. data-integrity-alg means more CPU-usage for every write; but, similar to verify-alg, it is subject to false-positives, see here for details on both of these points.

It’s configured as follows:

• Set c-plan-ahead to approximately 10 times the RTT; so if ping from one node to the other says 200msec, configure 2 seconds (ie. a value of 20, as the unit is tenths of a second).¹
  Please note that the controller is only polling every 100msec; so c-plan-ahead values below 5 don’t make sense, as the controller hasn’t collected enough information to decide whether to request more data. We recommend to use at least 1 second (configured value is 10).
  This value specifies the “thinking ahead” time of the controller, ie. the time period the controller has to achieve the actual sync-rate.
• Configure minimum and maximum values via c-min-rate and c-max-rate; these depend mostly on the available bandwidth per resource.
  The c-min-rate is the minimum bandwidth that will be used during a resync, whereas c-max-rate is the most bandwidth that can be used by a resync.
• Now decide whether to use c-fill-target or c-delay-target – you can choose only one.

Difference between delay and fill based control

If you set c-fill-target to a non-zero value, DRBD will try to keep that much data on the wire; if application IO gets in, it will temporarily displace the synchronization traffic. This means that application data will have only a limited amount of synchronization data in the buffers before it, which helps latency a bit.
The data still has to fit into the socket buffers, along with the application IO, so using multi-MB sizes here doesn’t make sense; 100kByte is a good starting value.

With a proxy you should use c-delay-target, so set the c-fill-target value to zero. This way the time interval that the synchronization data is on the wire is measured; if application IO gets in, this triggers the controller, and it will turn back the synchronization speed, to keep the communication latency at the specified value. Use 5 times the RTT as a starting point.

The users guide (8.3 version) describes the data-integrity-alg as

DRBD can ensure the data integrity of the user’s data on the network by comparing hash values. [...]

Too many people think this is a must-have setting – but are sadly wrong.

During initial installation and testing it does make sense to use this – it’s an easy way to find out whether the hardware (CPU, memory, network card, etc.) work as they should – if you get the famous Digest integrity check FAILED message¹ you can be worried (but not too much, since you found that during testing (;).

But in production this should not be set – apart from causing a lot of CPU load² it might cause frequent connection abort – and that means a short bit of time (re-sync) during which the secondary is inconsistent.

So: don’t use this in production.

Use syntax highlight. This helps to see unmatched quote characters easily. Whether it’s too colorful can be discussed, though
A current version can be found here, and the mailing list post is here.

For correlating resource names I recommend the Mark plugin. I’ve put an additional mapping in my ~/.vimrc:

 nnoremap  <F4>   :call mark#MarkCurrentWord()

With this I just put the cursor on a resource name and press F4 – all occurrences of this word are highlighted, so it’s very easy to see where the name is used¹. If I mark other words they get different colors, and with F4 the highlightning can be turned off again, too.

I think that for people looking at more-than-minimal Pacemaker configurations this helps to quickly get (and keep!) an overview.

Imagine a node having a DRBD resource in the secondary role. When it crashes, gets powered down, or looses connection to the primary for any other reason, the primary will remember the changed blocks in its bitmap; so when the secondary node connects to the primary again, the primary can tell exactly what data has to be transferred to get it Up-to-Date again.

Now, when the node with the Primary role crashes, no one can say with certainty which blocks have been changed on its storage, but haven’t been sent to the secondary, or vice-versa.
So the primary has to persistently remember which blocks it touched lately, so that on a crash it can ask the secondary node (which might have been promoted to primary in the meantime) for these blocks; they are fetched and written to the crashed primaries’ storage, to get its data consistent with the other node.

The al-extents value defines how big the hot area (=active set) can get. As the AL has to be stored on persistent storage, each change to this set has to be written to the meta-data area, before the actual data can be written – and waiting for that can slow down DRBD. So you can choose between

• a small activity log – this might make DRBD slower during load;
• or a big activity log, which will cause more data to be re-synced in case of a primary crash.

So, for the most part people are happy choosing a bigger activity log; in the 8.3 series the maximum value is 3833; for 8.4 the limit got raised to 6433, and who knows what 9.0 brings?

To cite a bit of an lwn.net article:

Imagine a machine with lots of memory – say 100Gig. [...]
Suppose this filesystem is being written to steadily so that the maximum amount of memory is always dirty. With the default vm.dirty_ratio of 40%, this could be 40Gig.[...]
Waiting for 40gig to flush for an atime update to complete is clearly unsatisfactory.

Now this isn’t entirely correct anymore; recent kernels do some tuning of these values, but you get the gist.

Even if you have a super-fast storage subsystem, you’ll have to process that data – and if you use DRBD for that filesystem, DRBD has to push that data through your network to the other node(s) – which will take some time, even with 10Gbit Ethernet …
Another advantage is that you will not as easily get into OOM situations.

So, the simple conclusion: for (possibly) heavily I/O loaded systems, set some sane value for the vm.dirty_background_bytes sysctl; again, see the recent post for more details.