Comments on: DRBD and MySQL: Just Say Yes http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/ Linux, DRBD, and other stuff of interest Tue, 06 Jan 2009 05:43:41 +0000 http://wordpress.org/?v=MU hourly 1 By: Morgan Tocker http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1753 Morgan Tocker Wed, 30 Apr 2008 12:36:02 +0000 http://fghaas.wordpress.com/?p=50#comment-1753 @Bill: FWIW, you are correct assuming it is statement based replication. In the case of row based replication, it's possible that a bit flip could cause a syntactically correct 'event' that will corrupt data. @Bill: FWIW, you are correct assuming it is statement based replication. In the case of row based replication, it’s possible that a bit flip could cause a syntactically correct ‘event’ that will corrupt data.

]]> By: Florian Haas http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1743 Florian Haas Tue, 29 Apr 2008 07:57:39 +0000 http://fghaas.wordpress.com/?p=50#comment-1743 Bill, you're right, a MySQL replication statement is probably more likely to altogether fail due to network corruption, rather than propagate garbage. But "more likely" doesn't mean "certain". :-) Just like for DRBD, without replication integrity checking, there is a chance that network corruption affects the DRBD protocol header rather than the packet payload. This would cause DRBD on the remote end to receive (and discard) a malformed packet. You just can't tell for sure. Which is why in DRBD we adopted the end-to-end approach. Bill,

you’re right, a MySQL replication statement is probably more likely to altogether fail due to network corruption, rather than propagate garbage. But “more likely” doesn’t mean “certain”. :-)

Just like for DRBD, without replication integrity checking, there is a chance that network corruption affects the DRBD protocol header rather than the packet payload. This would cause DRBD on the remote end to receive (and discard) a malformed packet. You just can’t tell for sure. Which is why in DRBD we adopted the end-to-end approach.

]]> By: Florian Haas http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1742 Florian Haas Tue, 29 Apr 2008 07:52:49 +0000 http://fghaas.wordpress.com/?p=50#comment-1742 Kris Buytaert adds an interesting angle to the discussion in a post titled "DRBD and MySQL: often say NO"; see http://www.krisbuytaert.be/blog/node/657. Kris Buytaert adds an interesting angle to the discussion in a post titled “DRBD and MySQL: often say NO”; see http://www.krisbuytaert.be/blog/node/657.

]]> By: MySQL Replication vs DRBD Battles | MySQL Performance Blog http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1741 MySQL Replication vs DRBD Battles | MySQL Performance Blog Tue, 29 Apr 2008 03:50:50 +0000 http://fghaas.wordpress.com/?p=50#comment-1741 [...] these days we see a lot of post for and against (more, more) using of MySQL and DRBD as a high availability [...] [...] these days we see a lot of post for and against (more, more) using of MySQL and DRBD as a high availability [...]

]]> By: Bill http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1740 Bill Mon, 28 Apr 2008 19:12:35 +0000 http://fghaas.wordpress.com/?p=50#comment-1740 How is mysql replication worse off the dbrd in the case of nic /network corruption, if the binary logs contain full sql statements? If it mangles a byte changing INSERT to INSERQ. replication will break, but it won't destroy your data on the slave. Now if it hits a blob field of binary data, then yes that might be a problem. I would think hat would be less likely to happen, or at least more dependent upon the schema and usage patterns. How is mysql replication worse off the dbrd in the case of nic /network corruption, if the binary logs contain full sql statements? If it mangles a byte changing INSERT to INSERQ. replication will break, but it won’t destroy your data on the slave. Now if it hits a blob field of binary data, then yes that might be a problem. I would think hat would be less likely to happen, or at least more dependent upon the schema and usage patterns.

]]> By: Florian Haas http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1738 Florian Haas Sun, 27 Apr 2008 19:43:02 +0000 http://fghaas.wordpress.com/?p=50#comment-1738 Eric, in your post you refer broadly to "any corruption on the primary master" getting propagated over to the DRBD peer. Now as I've stated and will continue to state, you're right as far as upper I/O layers are concerned. And this won't get "fixed" unless someone redesigns the complete Linux I/O stack. But looking at possible issues _below_ DRBD, we're chipping away at possible sources of corruption one by one (note that these are sources of corruption _outside_ DRBD that DRBD just happens to handle gracefully or rectify): - Disk I/O errors on any node: Automatic detach, introduced pre-DRBD 8.0 - Network bit flips/network traffic corruption/NIC driver bugs: End-to-end replication integrity checks, introduced in 8.2.0 - Subtle disk I/O errors, bit flips, local-disk data corruption: Online device verification, introduced in 8.2.5 The latency concerns you mentioned have also greatly been mitigated by better CPU affinity handling introduced in 8.2.3. This has been back-ported to the 8.0 branch as well, and makes a particularly big difference on multi-core systems. Eric,

in your post you refer broadly to “any corruption on the primary master” getting propagated over to the DRBD peer. Now as I’ve stated and will continue to state, you’re right as far as upper I/O layers are concerned. And this won’t get “fixed” unless someone redesigns the complete Linux I/O stack.

But looking at possible issues _below_ DRBD, we’re chipping away at possible sources of corruption one by one (note that these are sources of corruption _outside_ DRBD that DRBD just happens to handle gracefully or rectify):

- Disk I/O errors on any node: Automatic detach, introduced pre-DRBD 8.0
- Network bit flips/network traffic corruption/NIC driver bugs: End-to-end replication integrity checks, introduced in 8.2.0
- Subtle disk I/O errors, bit flips, local-disk data corruption: Online device verification, introduced in 8.2.5

The latency concerns you mentioned have also greatly been mitigated by better CPU affinity handling introduced in 8.2.3. This has been back-ported to the 8.0 branch as well, and makes a particularly big difference on multi-core systems.

]]> By: Eric Bergen http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1736 Eric Bergen Sun, 27 Apr 2008 19:02:46 +0000 http://fghaas.wordpress.com/?p=50#comment-1736 What specifically has changed to address my concerns and in which versions of drbd? I'll gladly add updated sections to that blog entry so people aren't getting outdated information. What specifically has changed to address my concerns and in which versions of drbd? I’ll gladly add updated sections to that blog entry so people aren’t getting outdated information.

]]> By: Phil Hildebrand http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1734 Phil Hildebrand Sun, 27 Apr 2008 16:16:20 +0000 http://fghaas.wordpress.com/?p=50#comment-1734 Though we haven't run DRBD yet in production, it is our plan (we're setting up in QA currently for perf and fail over testing). We run all our databases (SQLServer and MySQL) on a SAN for multiple reasons, and for most of the reasons you talk about (snapshots, DRBD sitting below the file system, etc) we have decided on this as a great HA solution for us. It also allows us to have each node attached to separate storage processors on the SAN, which will allow for a full storage processor failure without taking out our database system. This is also way faster than any non-block device replication that I know of. The issue with replication as a sole setup for HA is that it requires intelligent coding in the application should you have to switch to the slave as 'master' because there is not a shared ip that get's migrated. Though there are patches for auto promoting masters from google, it's not really the most straight forward process. Ideally, if you can afford both (as in our case) from a disk usage perspective, then you get the benefits from both.... Though we haven’t run DRBD yet in production, it is our plan (we’re setting up in QA currently for perf and fail over testing).

We run all our databases (SQLServer and MySQL) on a SAN for multiple reasons, and for most of the reasons you talk about (snapshots, DRBD sitting below the file system, etc) we have decided on this as a great HA solution for us. It also allows us to have each node attached to separate storage processors on the SAN, which will allow for a full storage processor failure without taking out our database system.

This is also way faster than any non-block device replication that I know of.

The issue with replication as a sole setup for HA is that it requires intelligent coding in the application should you have to switch to the slave as ‘master’ because there is not a shared ip that get’s migrated. Though there are patches for auto promoting masters from google, it’s not really the most straight forward process.

Ideally, if you can afford both (as in our case) from a disk usage perspective, then you get the benefits from both….

]]> By: Morgan Tocker http://fghaas.wordpress.com/2008/04/27/drbd-and-mysql-just-say-yes/#comment-1733 Morgan Tocker Sun, 27 Apr 2008 15:53:04 +0000 http://fghaas.wordpress.com/?p=50#comment-1733 > NIC and network corruption is also propagated. That one is interesting - since it was mentioned as a minus for DRBD, when it's just as true for MySQL replication ;) The worklog for event checksums in MySQL is <a href="http://forge.mysql.com/worklog/task.php?id=2540" rel="nofollow">#2540</a>. Until that's implemented, DRBD is doing *better* (not worse) in this regard. > NIC and network corruption is also propagated.

That one is interesting - since it was mentioned as a minus for DRBD, when it’s just as true for MySQL replication ;)

The worklog for event checksums in MySQL is #2540. Until that’s implemented, DRBD is doing *better* (not worse) in this regard.

]]>