Monthly Archives: February 2019

Active-Active HA cluster on Hetzner servers

Hetzner is a dedicated server provider that gives extremely good value to its customers but has a few quirks. The particular quirk that made my life so difficult, and I guess of many others, is its implementation of floating IPs.

Hetzner’s implementation of floating IPs is not ARP based as an orthodox sysadmin would expect. I will hazard a guess that it is done in such a fashion so that data streams do not leak across customers.

But despair not, there is a straightforward way to do exactly what one would expect to do with linux-ha. One can setup a pair of floating IPs between a pair of hosts and use keepalived to manage IP migration thus effectively creating an active-active setup.

First you need two servers with a private interconnection between them and one floating IP per server. Configure the IP on each server following the instructions on the Hetzner Wiki . Make sure that you configure both floating IPs on each host so that on failover the interfaces will be ready. Also remember to setup your internal IPs and set aside two IPs of your own for the internal floating scheme that vrrp needs. In our example we will use 192.168.1.1 for maste1 and 192.168.2.1 for master2. Obviously the internal network is 192.168.0.0/16

Now setup the Hetzner api scripts as shown in their failover script and you are almost done. Next step is to install keepalived and configure it so that only its vrrpd module is active. As an example on a Debian server edit the file /etc/default/keepalived and insert the following line in there

DAEMON_ARGS="-P

Now the trick is to have two virtual routers configured in keepalived each having as master one of your hosts and slave the other one. A sample configuration for master1 follows:

#
# This runs on master1
# 192.168.[12].1 are the internal floating IPs we manage as a necessity
# but the notification scripts do the actual work
# eth1 is the internal interconnect
#
global_defs {
   notification_email {
       devops@yourdomain
   }
   notification_email_from noreply@yourdomain
   smtp_server smtp.yourdomain
   smtp_connect_timeout 60
   script_user root
}

vrrp_instance virtualrouter1 {
   state MASTER
   interface eth1
   virtual_router_id 1
   priority 200
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 1234pass
   }
   virtual_ipaddress {
       192.168.1.1
   }
   preempt_delay 5
   notify_master "/usr/local/bin/hetzner.sh ip1 set master1"
   notify_backup "/usr/local/bin/hetzner.sh ip1 set master2"
}

vrrp_instance virtualrouter2 {
   state BACKUP
   interface eth1
   virtual_router_id 2
   priority 100
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass pass4321
   }
   virtual_ipaddress {
       192.168.2.1
   }
   preempt_delay 5
   notify_master "/usr/local/bin/hetzner.sh ip2 set master1"
   notify_backup "/usr/local/bin/hetzner.sh ip2 set master2"
}

obviously the configuration on master2 is the inverse of the above.

  • Beware fallback and failback times are a bit longer than what one would expect , but the setup adds a nine to your overall availability!