Register
It is currently Mon Nov 24, 2014 12:51 am

Parallel


All times are UTC - 6 hours


Post new topic Reply to topic  [ 3 posts ] 
Author Message
 PostPosted: Fri Mar 25, 2011 1:12 pm   
User avatar

Joined: Tue Apr 27, 2010 2:28 pm
Posts: 172
Location: Czech Republic
I often use this script to execute several commands in parallel (on a machine with more than 1 CPU):
Code:
#! /bin/bash

## Tries to run commands in parallel. Commands are read from STDIN one
## per line, or from a given file specified by -f.
## Author: E. Choroba

file='-'
proc_num=$(grep -c ^processor'\b' /proc/cpuinfo)
prefix=$HOSTNAME-$USER-$$
sleep=10

children=()
names=()

if [[ $1 =~ ^--?h(elp)?$ ]] ; then
    cat <<-HELP
        Usage: ${0##*/} [-f file] [-n max-processes] [-p tmp-prefix] -s [sleep]
          Defaults:
            STDIN for file
            $proc_num for max-processes (number of processors)
            $prefix for tmp-prefix
            $sleep for sleep interval
        HELP
    exit
fi

function child_count () {
    child_count=0
    for child in "${children[@]}" ; do
        kill -0 $child 2>/dev/null && let child_count++
    done
    echo $child_count
}

while getopts 'f:n:p:s:' arg ; do
    case $arg in
        f ) file=$OPTARG ;;
        n ) proc_num=$((OPTARG)) ;;
        p ) prefix=$OPTARG;;
        s ) sleep=$OPTARG;;
        * ) echo "Warning: unknown option $arg" >&2 ;;
    esac
done

i=0
while read line ; do
    name=$prefix.$i
    let i++
    names+=($name)

    while ((`child_count`>=proc_num)) ; do
        sleep $sleep
    done

    eval $line 2>$name.e >$name.o &
    children+=($!)

done < <(cat $file)
wait

cat "${names[@]/%/.o}"
cat "${names[@]/%/.e}" >&2
rm "${names[@]/%/.o}" "${names[@]/%/.e}"

Typically, I just send a list of commands to be run to its standard input. The commands can be created on the fly like this:
Code:
for file in *.txt ; do echo "process $file" ; done | parallel.sh
or
Code:
ls *.txt | xargs -n2 | sed 's/^/process /' | parallel.sh
to process files in tuples.


Top
 Profile  
 PostPosted: Tue May 03, 2011 11:40 am   
Site Admin
User avatar

Joined: Sun May 15, 2005 9:36 pm
Posts: 673
Location: Des Moines, Iowa
I'm shamelessly stealing this ! :)

We have a computer that has 96 processors at work.... now I can harness that power with the shell :)
It will need some modifications as there is no /proc/cpuinfo ;) but I'll give it a shot.
Thanks for posting it.


Top
 Profile WWW  
 PostPosted: Tue May 03, 2011 3:55 pm   
User avatar

Joined: Tue Apr 27, 2010 2:28 pm
Posts: 172
Location: Czech Republic
crouse wrote:
I'm shamelessly stealing this ! :)

We have a computer that has 96 processors at work.... now I can harness that power with the shell :)
It will need some modifications as there is no /proc/cpuinfo ;) but I'll give it a shot.
Thanks for posting it.


cpuinfo is used just to obtain the number of processors to set the default value of parallel processes. You can set it to 96 :) We have only 4 - 8 processors here, but on 40 machines in a cluster :)

I hope the script will serve you well.


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP