Active-Active HA cluster on Hetzner servers

Hetzner is a dedicated server provider that gives extremely good value to its customers but has a few quirks. The particular quirk that made my life so difficult, and I guess of many others, is its implementation of floating IPs.

Hetzner’s implementation of floating IPs is not ARP based as an orthodox sysadmin would expect. I will hazard a guess that it is done in such a fashion so that data streams do not leak across customers.

But despair not, there is a straightforward way to do exactly what one would expect to do with linux-ha. One can setup a pair of floating IPs between a pair of hosts and use keepalived to manage IP migration thus effectively creating an active-active setup.

First you need two servers with a private interconnection between them and one floating IP per server. Configure the IP on each server following the instructions on the Hetzner Wiki . Make sure that you configure both floating IPs on each host so that on failover the interfaces will be ready. Also remember to setup your internal IPs and set aside two IPs of your own for the internal floating scheme that vrrp needs. In our example we will use 192.168.1.1 for maste1 and 192.168.2.1 for master2. Obviously the internal network is 192.168.0.0/16

Now setup the Hetzner api scripts as shown in their failover script and you are almost done. Next step is to install keepalived and configure it so that only its vrrpd module is active. As an example on a Debian server edit the file /etc/default/keepalived and insert the following line in there

DAEMON_ARGS="-P

Now the trick is to have two virtual routers configured in keepalived each having as master one of your hosts and slave the other one. A sample configuration for master1 follows:

#
# This runs on master1
# 192.168.[12].1 are the internal floating IPs we manage as a necessity
# but the notification scripts do the actual work
# eth1 is the internal interconnect
#
global_defs {
   notification_email {
       devops@yourdomain
   }
   notification_email_from noreply@yourdomain
   smtp_server smtp.yourdomain
   smtp_connect_timeout 60
   script_user root
}

vrrp_instance virtualrouter1 {
   state MASTER
   interface eth1
   virtual_router_id 1
   priority 200
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass 1234pass
   }
   virtual_ipaddress {
       192.168.1.1
   }
   preempt_delay 5
   notify_master "/usr/local/bin/hetzner.sh ip1 set master1"
   notify_backup "/usr/local/bin/hetzner.sh ip1 set master2"
}

vrrp_instance virtualrouter2 {
   state BACKUP
   interface eth1
   virtual_router_id 2
   priority 100
   advert_int 1
   authentication {
       auth_type PASS
       auth_pass pass4321
   }
   virtual_ipaddress {
       192.168.2.1
   }
   preempt_delay 5
   notify_master "/usr/local/bin/hetzner.sh ip2 set master1"
   notify_backup "/usr/local/bin/hetzner.sh ip2 set master2"
}

obviously the configuration on master2 is the inverse of the above.

  • Beware fallback and failback times are a bit longer than what one would expect , but the setup adds a nine to your overall availability!
Advertisements

Bust your users’ chops by finding their passwords

#!/bin/bash

# rockyou is @ https://github.com/brannondorsey/naive-hashcat/releases/download/data/rockyou.txt
# wordlists are @ http://www.md5this.com/tools/wordlists.html 
#increase performance if you feel like it
performance=3
performance=1

LDAPSERVER=
LDAPBASE=
LDAPBINDDN=
LDAPPASS=

cd /root/infosec/hashcat
(ldapsearch -Z -x -h $LDAPSERVER -b ou=users,$LDAPBASE -D $LDAPBINDDN -w "$LDAPPASS" uid UserPassword |\
grep ^userPassword |\
awk '{ print $2}' | while read line
do
echo $line| base64 --decode 2>/dev/null | grep SSHA
done ) > /root/infosec/hashcat/passwords

/root/infosec/hashcat/hashcat \
--quiet \
-O -w $performance \
-D 2 \
--gpu-temp-abort 80 \
-m 111 \
-r rules/generated2.rule \
-o cleartext_passwords \
passwords dicts/rockyou.txt dicts/www.md5this.com/Wordlist.txt dicts/www.md5this.com/wordlists/*.dic

if [ -f cleartext_passwords ]
then
echo found some passwords
cat cleartext_passwords
fi

 

Beautifully Track Certificate Expirations with Grafana , Influxdb and Collectd.

In my previous write up I had a sample python script to display the number of days a certificate is valid. I have moved forward and created a complete certificate tracking solution using Collectd, Influxdb and Grafana. I will not go through the complete setup here because there are just too many tutorials about this kind of thing. What follows is the recipe for just tracking certificates’ expirations, and arduous and perilous task for admins of all ages.

First you need a collectd config file that uses the exec plugin, normally it would go inside /etc/collectd/collectd.conf.d/certs_valid.conf

LoadPlugin exec

   Exec nobody "/etc/collectd/collectd.conf.d/collectd_certs_valid.py" "one_domain:443"
   Exec nobody "/etc/collectd/collectd.conf.d/collectd_certs_valid.py" "another_domain:443"

Now copy the following python code in /etc/collectd/collectd.conf.d/collectd_certs_valid.py and make sure you can run by hand. A good test is collectd_certs_valid.py http://www.google.com:443 it should start giving out lines like: PUTVAL “www.google.com/x509-seconds/timeleft” interval=10 N:5948602  every ten seconds.

#!/usr/bin/python3 -u
#
# Calculate the expiration days for a cert angelos@multiwave.fr
#
import OpenSSL
import ssl
import sys
import datetime
import time
import os


if len(sys.argv) <=1:
  print("Usage: expires host:port")
  exit(1)

try:
  [hostname,port]=sys.argv[1].split(":")
except:
  hostname=sys.argv[1]
  port=443

try:
  conn = ssl.create_connection((hostname, port))
except:
  print("ssl connection failed")
  exit(1)

try:
  interval=int(os.environ['COLLECTD_INTERVAL'])
except:
  interval=10

while True:
  conn = ssl.create_connection((hostname, port))
  context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
  sock = context.wrap_socket(conn, server_hostname=hostname)
  certificate = ssl.DER_cert_to_PEM_cert(sock.getpeercert(True))
  x509 = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, certificate)
  expires=time.strptime(x509.get_notAfter().decode('ascii'), '%Y%m%d%H%M%SZ')

  now=datetime.datetime.now().timetuple()
  diff=time.mktime(expires)-time.mktime(now)

  print("PUTVAL \"%s/x509-seconds/timeleft\" interval=%d N:%d"% (hostname,interval,diff))

  time.sleep(interval)

The above effectively allows collectd to insert proper expiration time data into your Influxdb. Note the -u parameter to python to use unbuffered output.

Now all you have to do is use  Grafana and create a new dashboard/panel combination and use the x509 metrics retrieved from the python script.

Remember to adjust your Y axes for time data in seconds and the net result of the Graph will be pretty nice. Adjust it to your heart’s content:

The really truly awesome part is Grafana’s Alerts. You can create an alert if a certificate’s time is less than a week to get and alert.

Never Be surprised by an expired certificate ever again!

P.S. many thanks to all the net folks who know the innards of openSSL and x509 for python

Certificate Expiration Tracking

Days a cert is still valid

#!/usr/bin/python3
#
# Calculate the expiration days for a cert
#
import OpenSSL
import ssl
import sys
import datetime
import time

try:
    [hostname,port]=sys.argv[1].split(":")
except:
    hostname=sys.argv[1]
    port=443

try:
    conn = ssl.create_connection((hostname, port))
except:
    print("ssl connection failed")
    exit(1)

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1)
sock = context.wrap_socket(conn, server_hostname=hostname)
certificate = ssl.DER_cert_to_PEM_cert(sock.getpeercert(True))
x509 = OpenSSL.crypto.load_certificate(OpenSSL.crypto.FILETYPE_PEM, certificate)
expires=time.strptime(x509.get_notAfter().decode('ascii'), '%Y%m%d%H%M%SZ')

now=datetime.datetime.now().timetuple()
diff=((time.mktime(expires)-time.mktime(now))/3600/24)

print("%s:%s days left:%d" % (hostname, port, diff))

use flake8 to check recursively a python dir/project

For all those lazy yet finicky python coders. Code to be read while listening to David Byrne’s Lazy


#!/bin/bash

TOSCAN=$1
FLAKE8="flake8 --max-line-length=100"

if [ "X$TOSCAN" == "X" ]
then
        TOSCAN="."
fi

TMPDIR=/tmp/$$
mkdir $TMPDIR

echo "0" > $TMPDIR/flakeerrors.txt
flakeerrors=0
find $TOSCAN -type f -name "*.py" |\
while IFS= read -r file
do
        echo ">>>>>>>> flake8 checking file:$file "
        res=0
        $FLAKE8 "$file" >&  $TMPDIR/flake8out.txt
        res=$?
        cat $TMPDIR/flake8out.txt
        if (( $res > 0 ))
        then
                echo "======== flake8 Return:$res"
                (( flakeerrors++ ))
        fi
        # subshells do not propagate vars
        echo $flakeerrors > $TMPDIR/flakeerrors.txt
done
# back from the while subshell
flakeerrors=$(cat $TMPDIR/thisbuild/flakeerrors.txt)
if (( $flakeerrors > 0 ))
then
        echo " **********************
                flake8 error
                count: ${flakeerrors}
                *********************" | /usr/games/cowsay -e \*\*
fi


buildbot for python with gitlab queues

Here is a quick and dirty recipe for using a buildbot pipeline to build python projects from gitlab. gitlab does indeed have its own CI but one has to remember to create all there .gitlab-ci.yml files all over the place.There is no real way to have a global build system.

buildbot has a nifty feature called anyBranchScheduler that will try to build for any branch and two really nasty design problems.

a) the workers do not normally clean up the build dir because they expect things to be normalized at branch level

b) the way it calls external shells is execve style attrocious

and

c) there is no straightforward  way to build one’s python tools under a virtualenv

The solution is to first create a factory that will call our very own build script and pass it as a parameter the branch and the repo. gitlab thankfully passes this info to buildbot which graciously accepts and puts into util.Property(‘branch’) and util.Property(‘repository’)  respectively. So here is the factory shellsequence definition

myfactory.addStep(
  steps.ShellSequence(
    commands=[
        util.ShellArg(command=['/usr/local/bin/runabuild.bash' , 
            util.Property('branch') , util.Property('repository')],
            logfile='build.log', warnOnFailure=True),
    ]
  )
)

c['builders'] = [
    util.BuilderConfig(
        name="test_and_build",
        workernames=["local-worker"],
        factory=myfactory
    ),
]


So far so good , now you need the script and here it goes. It will create a tmp dir, check out the code, run dependencies, unit tests, build the package and upload it!

#!/bin/bash

SSHPARAMS="-i /home/buildbot/.ssh/id_rsa.rsync -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no"
SSHDEST="python-packages@python.mydomain.com"
TMPDIR=/tmp/buildbot/$$

branch=$1
giturl=$2

if [ "X$branch" == "X" ]
then
	echo "No branch passed"
	exit 1
fi

if [ "X$giturl" == "X" ]
then
	echo "No branch passed"
	exit 1
fi

function finish {
	cd /home/buildbot
	/bin/rm -rf $TMPDIR
	echo "*** cleanup done ***"
}


# sometimes gitlab sends https:// urls not git@
giturl=`echo $giturl | sed -e 's/https:\/\/gitlab.mydomain.com\//git@gitlab.mydomain.com:/g'`
echo "******************** giturl is:$giturl branch is:$branch ********************"

mkdir -p $TMPDIR
cd $TMPDIR

# Exit on error
set -e 
set -x
trap finish EXIT TERM HUP

echo "Setting up python virtualenv under $TMPDIR/venv"
virtualenv $TMPDIR/venv -p python3
source $TMPDIR/venv/bin/activate

#########################################################
#
# dependencies & requirements
#
##########################################################

if [ -f $TMPDIR/thisbuild/setup.py ]
then
	cd $TMPDIR/thisbuild
	echo "******************** Installing dependencies via pip ********************"
	pip install --process-dependency-links  $TMPDIR/thisbuild/
else
	echo "!!!!!!!!!!!!!!!!!!!! No setup.py found !!!!!!!!!!!!!!!!!!!!"
fi

if [ -f $TMPDIR/thisbuild/requirements.txt ]
then
	cd $TMPDIR/thisbuild
	echo "******************** Installing requirements via pip ********************"
	pip install -r  $TMPDIR/thisbuild/requirements.txt
else
	echo "!!!!!!!!!!!!!!!!!!!! No requirements.txt found !!!!!!!!!!!!!!!!!!!!"
fi

#########################################################
#
#  tests
#
##########################################################

if [ -d $TMPDIR/thisbuild/tests ]
then
	cd $TMPDIR/thisbuild
	echo "******************** Running unit tests ********************"
	python -m unittest
else 
	echo "!!!!!!!!!!!!!!!!!!!! No unit tests found !!!!!!!!!!!!!!!!!!!!"
fi

#########################################################
#
#  build & upload
#
##########################################################



if [ -f $TMPDIR/thisbuild/setup.py ]
then
	cd $TMPDIR/thisbuild
	echo "******************** Building the project ********************"
	python setup.py sdist bdist_egg
	if [ $? -ne 0 ]
	then
		echo "!!!!!!!!!!!!!!!!!!!! Build failed !!!!!!!!!!!!!!!!!!!!"
		exit 1
	fi
	# now upload the stuff :-)
	ls -alR dist
	dstdir=$(python3 setup.py --name 2> /dev/null | tail -n 1)
	echo "******************** uploading to $dstdir ********************"
	chmod 644 dist/*
	ssh $SSHPARAMS $SSHDEST mkdir -p files/${dstdir}
        scp $SSHPARAMS dist/* $SSHDEST:files/${dstdir}/
else
	echo "!!!!!!!!!!!!!!!!!!!! No setup.py found !!!!!!!!!!!!!!!!!!!!"
fi

echo "******************** build Job completed ********************"
finish

 

Seafile Rescue

I just had seafile installed as a file sharing system. Unfortunately being a bit lazy , I did not go through the whole documentation where it explicitly says LDAP users , use the mail attribute or else and used the uid attribute.

Now if you just make this config change and restart seafile all your old libs are gone ! To help restore owners and ownerships of libraries to their rightful email owners  , I used the following incantation:

update seafile_db.RepoInfo  set last_modifier='Email@Address' where last_modifier='$uid';

update seafile_db.RepoOwner set owner_id='email@address' where owner_id='$uid';

update seahub_db.options_useroptions set email='email@address' where email='$uid';

update seahub_db.base_userlastlogin set username='email@address' where username='$uid';

update ccnet_db.LDAPUsers set email='email@address' where email='$uid';

 

Standard disclaimers apply such as:

  • Make a backup first
  • This may damage your system
  • etc.

Don’t blame sendmail^H^H^H^H^H^H^HFirefox.

On Firefox performance,

For the last couple of months I have been plagued by real bad firefox performance. My colleagues kept harping about ditching this old browser and switching to chromium, the latest sexiest thingie. But you see I am an old dog and I really like to keep good software, so I tried to follow every blog site about firefox performance tuning. The eye opener was about:addons which showed that LastPass was “probably problematic”.

Well I disabled it and lo and behold Firefox is no longer using two of my ‘top’s CPUs at full throttle.

So, don’t blame Firefox! Remember:  the net elders did not blame sendmail either!

Dev. Defense

Customers, internal or external for that matter, regularly ask for one thing , imagine another and actually need a third. How can a developer defend oneself before such customers ? Use this; really.

  1. Please tell me what is expected of this product. Better yet draft it on a napkin.
  2. I have limited telepathy which decreases exponentially by distance. Please communicate succinctly via other means.
  3. I did not think of the other 400 possibilities you could have used  because my creativity is limited by my imagination. I never imagined this, or that for that matter.

Artifactory Stats via PERL

#!/usr/bin/perl
#
# Glean data from artifactory
#

use JSON;
use Data::Dumper;

# using api key for akarageo, fix as necessary
$CREDS=”user:apikey”;
$API=’https://artifactory.mydomain.com/artifactory/api&#8217;;

%MAG = (bytes => 1 ,
KB => 1024,
MB => 1024*1024,
GB => 1024*1024*1024,
TB => 1024*1024*1024*1024);

# if you want to sort by percentage
sub byperc {
$a->{percentage} <=> $b->{percentage} || $a->{filesCount} <=> $b->{filesCount}
}
# if you want to sort by storage size

sub bysize {
# convert to numbers and then compare
my ($bytes,$mag)=split(‘ ‘,$a->{usedSpace});
$a->{realSpace}=$bytes * $MAG{$mag};

($bytes,$mag)=split(‘ ‘,$b->{usedSpace});
$b->{realSpace}=$bytes * $MAG{$mag};

$b->{realSpace} <=> $a->{realSpace} || $a->{filesCount} <=> $b->{filesCount} ||   $a->{itemsCount} <=> $b->{itemsCount}
}

# the most interestinf part
sub printrepositoriesSummaryList {
my ($perl_scalar)=@_;
# data size page
print “<h2>Repositories Summary List</h2>\n”;
print “<table><tr>\n<th>Repository Name</th><th>Percentage</th><th>Used   Space</th><th>Folders</th><th>Items</th><th>Files</th></tr>\n”;
foreach $arr ($perl_scalar->{repositoriesSummaryList}) {
my @sarr = sort bysize @$arr;
foreach $hashref (@sarr) {
print “<tr>”;
print “<td>”.$hashref->{‘repoKey’}.”</td>”;
print “<td>”.$hashref->{‘percentage’}.”<ctd>”;
print “<td>”.$hashref->{‘usedSpace’}.”</td>”;
print “<td>”.$hashref->{‘foldersCount’}.”</td>”;
print “<td>”.$hashref->{‘itemsCount’}.”</td>”;
print “<td>”.$hashref->{‘filesCount’}.”</td>”;
print “</tr>\n”;
}
}
print “</table>”;
}

# global stats
sub printfileStoreSummary{
my ($perl_scalar)=@_;
print “<h3>File Store Summary</h2>”;
print “<table><tr><th>Directory</th><th>Total Space</th><th>Used Spacee</th>  <th>Free Space</th></tr><tr>”;
print “<td>” .$perl_scalar->{fileStoreSummary}{storageDirectory} .”</td>\n”;
print “<td>” .$perl_scalar->{fileStoreSummary}{totalSpace} .”</td>\n”;
print “<td>” .$perl_scalar->{fileStoreSummary}{usedSpace} .”</td>\n”;
print “<td>” .$perl_scalar->{fileStoreSummary}{freeSpace} .”</td>\n”;
print “</tr></table>\n”;
}
# may be interesting some time
sub printbinariesSummary{
my ($perl_scalar)=@_;
print “<h2>Binaries Summary</h2>\n<table>\n”;
foreach $key (keys($perl_scalar->{binariesSummary})) {
print “<tr><td>$key</td><td>” .$perl_scalar->{binariesSummary}{$key} .”</td></tr>\n”;
}
print “</table>\n”;
}
#main Code
$content=`/usr/bin/curl -s -u ${CREDS} ${API}/storageinfo`;
$json = JSON->new->allow_nonref;
$perl_scalar = $json->decode($content);

#$pretty_printed = $json->pretty->encode( $perl_scalar ); # pretty-printing
#print $pretty_printed ;

print “<h2> Artifactory Statistics </h2>”;

printfileStoreSummary($perl_scalar);
#printbinariesSummary($perl_scalar);
printrepositoriesSummaryList($perl_scalar);