Monthly Archives: July 2016

Easily add parallelization to any (PERL) program

Here is the catch, suppose you have a little program that can naturally be parallelized. Suppose you have an array, and for each row of the array you want to execute a certain procedure but all the procedures can run in parallel. Here is a little PERL tidbit that can do just that. It spawns a thread per array row but in batches so you can keep tabs:-).
Imagine this working in tandem with memoize.
$par_tabs is a global that defines the maximum number of parallel tabs.

You need a sub per element like so:

sub per_entry_sub($$$$$){
     my ($game,$evtime,$href,$extraparam1,$extraparam2) = @_;
     ...
     ...
}

and of course you call the parallel excutor like so:

par_do_threaded(\&per_entry_sub,$extraparam1,$extraparam2,\%games);

The \&per_entry_sub is a reference to the sub itself and is used in the executor as
$func->(parameters);

 

 

Aaand here is the executor:

######################################################################
#
# Parallel Threaded execution, one thread per additional firefox TAB
#
######################################################################
sub par_do_threaded($$$$) {
    my ($func,$extraparam1,$extraparam2,$dataref)=@_;
    # dereference data hash
    my %data = %$dataref;
    #Now fetch each entry in parallel
    my $count=0;    
    print "*****************************************************\n";
    print "**                 ENTERING PARALLEL OPERATION     **\n";
    print "*****************************************************\n";
    my $numdata=scalar keys %data;
    print "NUMBER OF ENTRIES=$numdata\n";
    return if ($numdata<=0);
    my $entrynum=0;
    foreach my $key (sort keys %data) {
        $entrynum++;
        print "ENTRY NUMBER: $entrynum ENTRY $key\n";
        my $entry  = $key;
        my $href   = $data{$key}{'href'};
        my $evdate = $data{$key}{'date'};
        # do it in batches of $par_tabs
        if ( $count < ($par_tabs - 1) ) { $count++; my $tid=threads->create($func,$entry,$evdate,$href,$extraparam1,$extraparam2)->detach();
            print "THREAD $tid created\n";
        } else {
            # This is the linear execution part of the code
            $func->($entry,$evdate,$href,$extraparam1,$extraparam2);
            $count=0;
        }
    }
}
Advertisements