Register
It is currently Fri Nov 28, 2014 5:00 am

Simple loop to array for a newbie


All times are UTC - 6 hours


Post new topic Reply to topic  [ 13 posts ] 
Author Message
 PostPosted: Mon Jan 04, 2010 7:32 pm   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
I am pretty new at this, so while it might seem like a very simple request, I'm not at the point where I can write it myself.

I want to schedule a script to run each day, capturing the logins I have had for the previous day. I already wrote a script that scours the logs of my app and gives me the date, the total number of logins and the number of unique users. I would like to have a script that parses the two numbers into two strings that I can use to insert into a Google Charts URL for a WGET.

Here is what I have:
2010-01-01, 4485, 3148
2010-01-02, 5817, 3954
2010-01-03, 10671, 6503

This is what I want:
$LOGINS_TOTAL=4485,5817,10671
$LOGINS_UNIQUE=3148,3954,6503

Is this pretty easy? Anyone want to take a crack at it?

Thanks
A


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 3:10 am   

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221
I'm thinking the use of arrays would be suitable for this.
Code:
#!/bin/bash
# Make a human-readable version of the first argument passed to the script (./script.sh <argument 1>)
file=$1
# reset the counter
i=0
# Do a line by line loop of the file supplied in argument 1 (or as it is now known, $file)
# The IFS variable tells us what we should look for to create an itteration inside for(1)
# old_IFS is just a way to reset the IFS when we're done :)
old_IFS=$IFS
IFS="
"
# The for(1) loop will read the $file and sort it according to column 1, the date in your supplied example.
for i in $(cat $file | sort -t',' -k1,1); do
   # This adds the second column to an array named "array1"
   array1[$i]=$(echo $i | cut -d',' -f2)
   # This adds the third column to an array named "array2"
   array2[$i]=$(echo $i | cut -d',' -f3)
   # Increment the elemental counter :)
   ((i++))
done
IFS=$old_IFS
# Echo the entire array (and since it's space separated we replace all spaces with commas)
total_logins=$(echo ${array1[*]} | replace " " ",")
total_unique=$(echo ${array2[*]} | replace " " ",")

echo "total_logins=$total_logins"
echo "total_unique=$total_unique"


Something like this might give you a clue on how to do it :) This isn't very well tested... i don't have a proper linux bash shell around here at the moment.

Best regards
Fredrik Eriksson


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 10:18 am   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
Not sure how this is supposed to work? The file I want to open is in the format of:
DATE, TOTAL LOGINS, UNIQUE LOGINS

I want to parse out the total logins, then the unique logins. Each needs to be in a comma separated string that will end up like: 123,234,345,456,567

This is so I can use it in a WGET to a Google Charts URL

Thx
A


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 10:57 am   

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221
You didn't read the comments did you?

The script as it is now (i copied it flat out and did the same with your example) creates 2 variables at the end.
$total_logins
$total_unique

These 2 are filled with a comma-separated list of the second column's data (the one you designated as total logins) and the third column's data (total unique).
Then it just echo's it as a proof of concept.
Code:
hanna:~> ./test.sh test.txt
total_logins=4485,5817,10671
total_unique=3148,3954,6503


You'll have to modify the output so it suits your needs.
If you don't have a file then you can just replace the line:
for i in $(cat $file | sort -t',' -k1,1); do

with
for i in $(command which you want to run goes here); do

As long as the list is in the same order as your example it doesn't really matter where the data comes from

Best regards
Fredrik Eriksson


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 11:54 am   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
Actually, I didn't get a chance to read the comments. I was surfing on my iPhone this morning to see if there were any replies to my post. The iPhone doesn't render the code field properly so I only saw about 4 lines.

Sorry about that!

The script looks great - thanks for the help.

A


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 12:11 pm   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
On Solaris, I don't seem to have replace. At least the errors I get indicate such:
Code:
./login-count: line 23: replace: command not found
./login-count: line 24: replace: command not found

Should I be trying sed instead of replace?


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 12:29 pm   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
Got it to work using this:
Code:
total_logins=$(echo ${array1[*]} | sed -e 's/[ ]\{1,\}/,/g')
total_unique=$(echo ${array2[*]} | sed -e 's/[ ]\{1,\}/,/g')

Now, is there a way to get the last 10 sets of numbers out of that file?


Top
 Profile  
 PostPosted: Tue Jan 05, 2010 1:28 pm   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
Okay, I think I figured it all out:
Code:
#!/bin/bash
# Parse data from login file to pass to Google Charts

# Locate the stat file
statfile="../statsdata/login.stats"

# Read in the last 10 entries
file=`tail -10 $statfile`

# reset the counter
i=0

# Do a line by line loop of the file supplied in argument 1 (or as it is now known, $file)
# The IFS variable tells us what we should look for to create an itteration inside for(1)
# old_IFS is just a way to reset the IFS when we're done :)
old_IFS=$IFS
IFS="
"
# Sort the total and unique logins
for i in $file; do
   # This adds the second column to an array named "array1"
   array1[$i]=$(echo $i | cut -d',' -f2)
   # This adds the third column to an array named "array2"
   array2[$i]=$(echo $i | cut -d',' -f3)
   # Increment the elemental counter :)
   ((i++))
done
IFS=$old_IFS

# Echo the entire array (and since it's space separated we replace all spaces with commas)
total_logins=$(echo ${array1[*]} | sed -e 's/[ ]\{1,\}/,/g')
total_unique=$(echo ${array2[*]} | sed -e 's/[ ]\{1,\}/,/g')

# Print the result
echo "total_logins=$total_logins"
echo "total_unique=$total_unique"


Thanks very much for your help! It works great.


Top
 Profile  
 PostPosted: Mon Jan 11, 2010 4:27 pm   

Joined: Mon Jan 04, 2010 7:26 pm
Posts: 8
Okay, ran into a bit of a snag. The script worked great until the other day. Then I started getting the following error:

Code:
line 53: 2010-01-08: value too great for base (error token is "08")


Line 53 is part of the following:
Code:
   50  # The for(1) loop will read the $file and sort it according to column 1, the date in your supplied example.
    51  for i in $MYFILE; do
    52     # This adds the second column to an array named "array1"
    53     array1[$i]=$(echo $i | cut -d',' -f2)
    54     # This adds the third column to an array named "array2"
    55     array2[$i]=$(echo $i | cut -d',' -f3)
    56     # Increment the elemental counter :)
    57     ((i++))
    58  done
    59  IFS=$old_IFS


I get the feeling that it is because the 08 in the date is being interpreted as an octal. I can't change the way the date is being stored, as it is an enterprise application I don't control the configuration of. Is there a way I can get the script to interpret the number as base-10 instead?

A


Top
 Profile  
 PostPosted: Sun Jan 24, 2010 7:28 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
a_calder wrote:
Quote:
I already wrote a script that scours the logs of my app and gives me the date, the total number of logins and the number of unique users. I would like to have a script that parses the two numbers into two strings that I can use to insert into a Google Charts URL for a WGET.

Hi, that sounds like fun...

It sounds like something I would write as a function in the larger problem. Can I see the script that gets the stats? :)

g


Top
 Profile  
 PostPosted: Mon Feb 01, 2010 1:40 pm   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
Here's how I would do it:
Code:
#!/bin/bash

echo; echo "[fn: distinguish_logins]"; echo

#function distinguish_logins ()
#{
rm ./logins_total.log
rm ./logins_unique.log
unset LOGINS_TOTAL[*]
unset LOGINS_UNIQUE[*]

echo "cat ./log_records.log"
echo "~~~~~~~~~~~~~~~~~~~~~"
cat ./log_records.log
echo

printf "%-11s" "Record"
printf "%+11s" "Log Total"
printf "%+11s\n" "Log Unique"
printf "%-11s" "~~~~~~"
printf "%+11s" "~~~~~~~~~"
printf "%+11s\n" "~~~~~~~~~~"

let -i count=0
for LOG in $(cat ./log_records.log); do
   let count=count+1
   echo $LOG | cut -d, -f2 >> ./logins_total.log
   echo $LOG | cut -d, -f3 >> ./logins_unique.log
   LOG_DATE[$count]=$(echo $LOG | cut -d, -f1)
   LOGINS_TOTAL[$count]=$(echo $LOG | cut -d, -f2)
   LOGINS_UNIQUE[$count]=$(echo $LOG | cut -d, -f3)
   printf "%-11.10s" ${LOG_DATE[$count]}
   printf "%+11.10s" ${LOGINS_TOTAL[$count]}
   printf "%+11.10s\n" ${LOGINS_UNIQUE[$count]}
#   echo "${LOGINS_TOTAL[$count]} ${LOGINS_UNIQUE[$count]}"
done
echo

echo "Logins Total:"
echo "~~~~~~~~~~~~~"
cat ./logins_total.log
echo
echo "Logins Unique:"
echo "~~~~~~~~~~~~~~~"
cat ./logins_unique.log
echo

echo -n "LOGINS_TOTAL="
for (( i = 1; i <= $count; i++ )); do echo -n "${LOGINS_TOTAL[$i]},"; done; echo
echo -n "LOGINS_UNIQUE="
for (( j = 1; j <= $count; j++ )); do echo -n "${LOGINS_UNIQUE[$j]},"; done; echo
echo "[done]"; echo

#}

exit 0


Giving this as output:
Code:
[fn: distinguish_logins]

cat ./log_records.log
~~~~~~~~~~~~~~~~~~~~~
2010-01-01,4485,3148
2010-01-02,5817,3954
2010-01-03,10671,6503

Record       Log Total Log Unique
~~~~~~       ~~~~~~~~~ ~~~~~~~~~~
2010-01-01        4485       3148
2010-01-02        5817       3954
2010-01-03       10671       6503

Logins Total:
~~~~~~~~~~~~~
4485
5817
10671

Logins Unique:
~~~~~~~~~~~~~~~
3148
3954
6503

LOGINS_TOTAL=4485,5817,10671,
LOGINS_UNIQUE=3148,3954,6503,
[done]



Of course, I would want to insist that the spaces between fields be taken out of the original log records.
>:)


Top
 Profile  
 PostPosted: Mon Feb 01, 2010 2:55 pm   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 574
eww!
Code:
#!/bin/bash

while IFS=' ,' read DATE TOTAL UNIQUE
do LOGINS_TOTAL+=( "$TOTAL" )
   LOGINS_UNIQUE+=( "$UNIQUE" )
done < <(tail -10 ../statsdata/login.stats)

# Echo the entire array (and since it's space separated we replace all spaces with commas)
# and Print the result
echo "total_logins=$(sed -e 's/[ ]\{1,\}/,/g' <<<"${LOGINS_TOTAL[@]}")"
echo "total_unique=$(sed -e 's/[ ]\{1,\}/,/g' <<<"${LOGINS_UNIQUE[@]}")"


Top
 Profile  
 PostPosted: Wed Feb 03, 2010 10:48 am   
User avatar

Joined: Mon Jan 18, 2010 8:10 pm
Posts: 40
Nice, Watael!

I am saving this for future reference.

g


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 13 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Majestic-12 [Bot] and 8 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP