Raspberry Tau: a Pi cluster

The Raspberry PI is a small ARM computer (hardware specifications in wiki, outline and FAQs). Of course, you can build a cluster with it!

As 2π is proposed to be named τ we’ve chosen the name “Raspberry Tau” for this proof-of-concept.

We’ve connected two Raspberry Pis via their on-board ethernet interfaces (via a switch, so we can simply SSH into them), booted via 2GB SD-cards with a Raspbian image on them. After upgrading to a kernel that has kernel-headers available we built DRBD modules, and voilá! A Raspberry Tau cluster is born.

We’re replicating the data on the USB-Sticks; their performance nicely matches the available network. Here’s /proc/drbd (shortened and line-wrapped for readability):

root@raspberry-alice:~# cat /proc/version 
Linux version 3.2.0-3-rpi (Debian 3.2.21-1+rpi1) 
  (debian-kernel@lists.debian.org) (… Debian 4.6.3-1.1+rpi2)…)
root@raspberry-alice:~# cat /proc/drbd 
version: 8.4.2 (api:1/proto:86-101)
GIT-hash: 7ad5f850d711223713d6dcadc3dd48860321070c build by
    root@raspberry-bob, 2012-09-18 12:58:08
 0: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:805304 nr:0 dw:348628 dr:818596 al:127 bm:70 lo:0 pe:0
      ua:0 ap:0 ep:1 wo:d oos:0

As Raspbian is Debian-based, there are Pacemaker (and Heartbeat resp. Corosync) packages available … so a cheap, low-power, High-Availability cluster is easily built.

Disclaimer: for a real HA-cluster you’d need a few more things.

  • a STONITH device (if power is supplied via a Linux-PC, you could turn off the USB port by software), and
  • redundant network connectivity (USB ethernet adapter).

Of course, if you’re just clustering your media library these things might not be mandatory.

Packages are available for everyone – just drop an email to sales@linbit.com, and we will be happy to provide them.

20 thoughts on “Raspberry Tau: a Pi cluster

  1. Hi

    You say drop a mail to sales@ but I dont know at what?

    I would love to build a PI cluster to mirror my media so if you can send me the packages and any details etc…..

    Thanks in advance

    David

  2. 2.d Interface : USB ethernet :-) i.e rtl8150

    Corosync fencing with stonith ssh sample :

    crm configure primitive st-ssh stonith:external/ssh params hostlist=”tau1 tau2″ clone fencing st-ssh

    crm configure property $id=”cib-bootstrap-options”
    dc-version=”1.1.7-2.fc17-ee0730e13d124c3d58f00016c3376a1de5323cff” \
    cluster-infrastructure=”corosync” \
    stonith-enabled=”true” \
    no-quorum-policy=”ignore” \
    expected-quorum-votes=”2″ \
    default-resource-stickiness=”5000″ \
    maintenance-mode=”false”

  3. Hi,

    I’ve succeeded to this configuration, with one Primary and one Secondary node.

    But do you know if Raspberries can support DRBD with 2 primary nodes, GFS, Cluster-manager, etc… ?

    Thank you !

    • It should be easily supported in terms of CPU and RAM.

      But — the Raspberry has only got a single network interface, so you’d need to use some USB-to-USB-cable as second communication path.
      Then, the ethernet is 100MBit; USB connections won’t be much faster, so the speed is inherently limited.
      For a cluster filesystem you’d need reliable fencing – how would you solve that?

      So sure, you could tinker around and make it work in some way – but I guess the sane solution would be a simple NFS server.

      • In fact, I’ve tried to make that. And yes, there is bugs or configuration errors (especially CLVM / DLM).

        I’m don’t know really what “fencing” is, but I used human manual option in ccs_tool…
        (‘ccs_tool addfence human manual’ command).

        However, I agree with you, it seems maybe too unstable.

        Last question please : Why this kind of cluster needs two network cards ?? (or Usb, like you say).

        THX

  4. Is it possible to use just ust2usb cable and usb-eth module to cover absence of second network adapter? I know it will be slow, but as proof of concept it would be interesting… maybe next gen rpi will have gigabit ethernet.

    • Yes, of course that should work. For a proof-of-concept, that is.

      But you still won’t have STONITH — it’s much easier to test with VMs, IMO.

  5. Hi,
    I tried the Rapsberry PI packages on my PI’s with Wheezy.
    Download and install with:
    sudo dpkg -i drbd8-module-3.12-1-rpi_8.4.5-1+3.12.9-1+rpi1_armhf.deb
    sudo dpkg -i drbd8-utils_8.9.1+linbit-4_armhf.deb

    Then: sudo /etc/init.d/drbd restart

    Problem reported:
    /etc/init.d/drbd: line 142: 2481 Segmentation fault $DRBDADM sh-nop

    Any help appreciated ….

    Regards,
    Dirk

    • Hi Dirk,

      hmmm, sorry… it works for us here. Perhaps you can get some more details via gdb, to find out what’s going wrong?

      • Hi,

        thanks for your reply.
        I have no experiences with GDB, so could be tricky.
        So you have used a fresh install of wheezy on two RPI’s an did the same commands as I did? In this case, I would try a fresh install on both RPI’s…..

        Thanks,
        Dirk

      • Back again. Fresh Wheezy on both PI’s. Kernel DRBD is 8.4.5, from Backports I installed Utils 8.9. The segmentation fault persists.
        global_common.conf:
        global {
        usage-count yes;
        }
        common {
        net {
        protocol C;
        }
        }

        /etc/drbd.d/r0.res:

        resource r0 {
        net {
        protocol C;
        cram-hmac-alg sha1;
        shared-secret “FooFunFactory”;
        }
        disk {
        resync-rate 10M;
        }
        on rpi1 {
        volume 0 {
        device minor 1;
        disk /dev/sda1;
        meta-disk internal;
        }
        address 192.168.1.12:7789;
        }
        on rpi2 {
        volume 0 {
        device minor 1;
        disk /dev/sda1;
        meta-disk internal;
        }
        address 192.168.1.13:7789;
        }
        }

        gdb -ex run -ex bt –args drbdadm create-md r0:

        GNU gdb (GDB) 7.4.1-debian
        Copyright (C) 2012 Free Software Foundation, Inc.
        License GPLv3+: GNU GPL version 3 or later
        This is free software: you are free to change and redistribute it.
        There is NO WARRANTY, to the extent permitted by law. Type “show copying”
        and “show warranty” for details.
        This GDB was configured as “arm-linux-gnueabihf”.
        For bug reporting instructions, please see:

        Reading symbols from /sbin/drbdadm…(no debugging symbols found)…done.
        Starting program: /sbin/drbdadm create-md r0

        Program received signal SIGSEGV, Segmentation fault.
        0x00059b02 in ?? ()
        #0 0x00059b02 in ?? ()
        #1 0x0000aa4a in ?? ()
        #2 0x0000aa4a in ?? ()
        Backtrace stopped: previous frame identical to this frame (corrupt stack?)

        • Well, it works for me.

          root@raspi:~# drbdadm sh-nop ; echo $?
          0
          root@raspi:~# drbdadm -V
          DRBDADM_BUILDTAG=GIT-hash:\ f861210be7433b670787b8fd9ea0915cccffcb01\ build\ by\ root@raspi\,\ 2014-12-01\ 10:59:25
          root@raspi:~# strings `which drbdadm` | grep “build by”
          GIT-hash: f861210be7433b670787b8fd9ea0915cccffcb01 build by root@raspi, 2014-12-01 10:59:25
          root@raspi:~# cat /proc/cpuinfo
          Processor : ARMv6-compatible processor rev 7 (v6l)

          Hardware : BCM2708

          What do you get on these commands?

          • First: Many thanks for your help!!

            That is, what I got:

            # cat /proc/drbd:
            version: 8.4.5 (api:1/proto:86-101)
            srcversion: EDE19BAA3D4D4A0BEFD8CDE

            # apt-cache policy linux-image-$(uname -r) drbd8-utils

            Installed: 2:8.9.2~rc1-1~bpo70+1
            Candidate: 2:8.9.2~rc1-1~bpo70+1
            Version table:
            *** 2:8.9.2~rc1-1~bpo70+1 0
            100 http://http.debian.net/debian/ wheezy-backports/main armhf Packages
            100 /var/lib/dpkg/status
            2:8.3.13-2 0
            500 http://mirrordirector.raspbian.org/raspbian/ wheezy/main armhf Packages

            # sudo drbdadm sh-nop ; echo $?
            Segmentation fault
            139

            # drbdadm -V
            Segmentation fault

            # sudo drbdadm -V
            (Nothing, no output at all)

            # cat /proc/cpuinfo
            processor : 0
            model name : ARMv6-compatible processor rev 7 (v6l)
            BogoMIPS : 2.00
            Features : half thumb fastmult vfp edsp java tls
            CPU implementer : 0x41
            CPU architecture: 7
            CPU variant : 0x0
            CPU part : 0xb76
            CPU revision : 7
            Hardware : BCM2708
            Revision : 0010
            Serial : 000000009561ef73

          • Ah, you installed the debian upstream packages for armhf – these are incompatible with the Raspberry Pi A, B, and B+.

            The debian branch works only for the Raspberry Pi 2 series; but Raspbian already has 8.9.2rc packages too, so you can simply install these.

  6. I am a newbie on that, so sorry for my question.
    I followed your Link and I have this in my sources.list:
    deb http://archive.raspbian.org/raspbian wheezy main contrib non-free
    deb-src http://archive.raspbian.org/raspbian wheezy main contrib non-free

    So I thought, that apt-get could do the installation and started a simulation:
    # apt-get -s install drbd-utils

    But I got this:
    Reading package lists… Done
    Building dependency tree
    Reading state information… Done
    Note, selecting ‘drbd8-utils’ instead of ‘drbd-utils’
    The following extra packages will be installed:
    heirloom-mailx
    Suggested packages:
    heartbeat exim4 mail-transport-agent
    The following NEW packages will be installed:
    drbd8-utils heirloom-mailx
    0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded.
    Inst heirloom-mailx (12.5-2+deb7u1 Raspbian:7.0/stable [armhf])
    Inst drbd8-utils (2:8.3.13-2 Raspbian:7.0/stable [armhf])
    Conf heirloom-mailx (12.5-2+deb7u1 Raspbian:7.0/stable [armhf])
    Conf drbd8-utils (2:8.3.13-2 Raspbian:7.0/stable [armhf])

    So, again sorry, but how can I install the mentioned package from here: http://mirrordirector.raspbian.org/raspbian/pool/main/d/drbd-utils/

    Many thanks in advance!!!

    Dirk

    • You’ll need to use not the “wheezy” repositories, but “jessie”, “jessie-staging”, and/or “testing”.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

What is 9, multiplied by 7 ?
Please leave these two fields as-is:
IMPORTANT! To be able to proceed, you need to solve the following simple math (so we know that you are a human) :-)