Monthly Archives: December 2015

Swiss Dormant Accounts Scraper

It has been quite a while since I have used PERL and this dormant accounts’ web page seemed like a challenge. Since I am always up for a good challenge, here is the solution in good old fashioned PERL.

 


#!/usr/bin/perl 
#
# Get Rich, Dump stuff from Swiss banks angelos karageorgiou angelos@unix.gr
# 
use HTML::Form;
use WWW::Mechanize;
use HTML::Parser ();
use Data::Dumper;
use HTML::TableExtract;

my $mech = WWW::Mechanize->new();
my $url='https://www.dormantaccounts.ch/narilo/';

# first into the page, click on Publications
$mech->get( $url );
$mech->form_number(2);
$mech->click();

my $html = $mech->content();
dump_table($html);

my $cont=1;
while ($cont) {
    print "-" x 80 ."\n";
    $cont=0;
    my $form=$mech->form_number(2);
    my $saved_input=undef;
    foreach $input ($form->inputs) {
        if ($input->value eq 'Next') {
            $saved_input=$input;
            $cont=1;
        }
    }
    # just in case
    $mech->click_button( input => $saved_input);
    dump_table($mech->content());
}

sub dump_table () {
 my $html=shift;
 $te = HTML::TableExtract->new( );
 $te->parse($html);

 # Examine all matching tables
 foreach $ts ($te->tables) {
   next if $ts->coords == "0,0";
   foreach $row ($ts->rows) {
        foreach $col (@$row) {
            $col=~s/[\s][\s]*/ /g;
            print "'$col' ;";
        }
    print "\n";
   }
 }
}

 

 

Saltstack script testing using docker

There are literally thousands of ways for a saltstack  script to go awry! One simple way to test these scripts is to create a single salt-master many salt-minions autmated testing rig.

First one needs a build script


 

#!/bin/bash
#
# build the docker images and start the test environment
# angelos@unix.gr
#
echo "Cleaning Up"
docker rm -f salt-master
for minion in 1 2 3
do
 docker rm -f salt-minion${minion}
done
echo "Building Master Image"
echo "Copying my ssh credentials to be used for git"
mkdir -p creds 2> /dev/null
cp $HOME/.ssh/id_rsa creds/
cp $HOME/.ssh/known_hosts creds/
cp $HOME/.gitconfig creds/gitconfig
docker build . 2>&1 | tee master.log
id=`grep 'Successfully built' master.log | cut -d" " -f 3`
if [ "X${id}" == "X" ]
then
 echo "Salt Master build failed"
 exit 1
fi
docker run -d \
 -h salt-master --name salt-master \
 -v $PWD/bath:/srv/salt \
 -v $PWD/pillar:/srv/pillar \
 --memory-swappiness=1 $id
docker tag -f ${id} salt-master
echo "Building Minion image"
docker build -f Dockerfile.minion . 2>&1 | tee minion.log
id=`grep 'Successfully built' minion.log | cut -d" " -f 3`
if [ "X${id}" == "X" ]
then
 echo "Salt Minion build failed"
 exit 1
fi
for minion in 1 2 3
do
 hash=`docker run -d \
 -h salt-minion${minion} --name=salt-minion${minion} \
 --link salt-master \
 --memory-swappiness=1 $id`
 docker tag -f ${id} salt-minion${minion}
done
docker ps

The big trick now is not so much the Dockerfiles that create the relevant docker images as much as each image’s startup script

Here is the salt-master’s docker CMD script


~/saltstack (master *% u=)$ cat master-start.sh 
#!/bin/bash

service rsyslog start
service salt-minion start
service salt-master start
service sshd start
echo "Sleeping a bit: 15 secs"
sleep 15
Echo "Auto-accepting All minion keys"
salt-key -A -y
echo "Going into infinity"
sleep infinity;true

Here is the salt-minions’ docker CMD script


 

~/saltstack (master *% u=)$ cat minion-start.sh 
#!/bin/bash
service rsyslog start
salt-minion --daemon --log-level debug
echo "Going into infinity"
sleep infinity;true

 

The above couple of almost brain dead scripts  will allow you to create an automated test platform for saltstack scripts.

 

 

 

 

SaltStack: encrypted VSphere Credentials

I got the idea from  https://clinta.github.io/random-local-passwords/

It integrates GPG GIT and SALT for relatively secure centralized credentials storage
You mileage will vary

Here is the diff to allow for storage of credentials in encrypted form for cloud.providers


*** /usr/lib/python2.6/site-packages/salt/cloud/clouds/vmware.py.old 2015-12-15 11:47:05.703214983 +0000
--- /usr/lib/python2.6/site-packages/salt/cloud/clouds/vmware.py 2015-12-15 12:56:18.067154711 +0000
***************
*** 67,72 ****
--- 67,73 ----
 import time
 import os.path
 import subprocess
+ import re
 
 # Import salt libs
 import salt.utils
***************
*** 197,202 ****
--- 198,212 ----
 port = config.get_cloud_config_value(
 'port', get_configured_provider(), __opts__, search_global=False, default=443
 )
+ ext_auth_method = config.get_cloud_config_value(
+ 'password_program', get_configured_provider(), __opts__, search_global=False, default=''
+ )
+ pw_store = config.get_cloud_config_value(
+ 'pw_store', get_configured_provider(), __opts__, search_global=False, default='/opt/passdb'
+ )
+ 
+ if ext_auth_method=='pass':
+ password=_get_pw_from_pass(username,pw_store)
 
 return salt.utils.vmware.get_service_instance(url,
 username,
***************
*** 3569,3571 ****
--- 3579,3605 ----
 return False
 
 return {datastore_cluster_name: 'created'}
+ 
+ 
+ 
+ def _get_pw_from_pass(pw_name, pw_store):
+     '''
+     Get a password, from pass utility ( GPG must be active) remember to patch pass with gpg secret
+     '''
+     my_env = os.environ
+     my_env["PASSWORD_STORE_DIR"] = pw_store
+ 
+     # synchonize first
+     devnull = open(os.devnull, 'w')
+     subprocess.call(['/usr/bin/pass','git', 'pull'],env=my_env,cwd=pw_store,stdout=devnull, stderr=devnull)
+ 
+     pw_file = '{0}/{1}.gpg'.format(pw_store, pw_name)
+     log.info("trying to get pass from '{0}'".format(pw_file))
+     if os.path.isfile(pw_file):
+         log.info("trying to get pass for '{0}'".format(pw_name))
+         proc = subprocess.Popen(["/usr/bin/pass",  pw_name], env=my_env,cwd=pw_store,stdout=subprocess.PIPE)
+         pass_plaintext = proc.stdout.readline().rstrip()
+         return pass_plaintext
+     else:
+         log.info("GPGed password file not found '{0}'".format(pw_file))
+ 
+     return 'Pass Not Found'
+