BashScripts.org
http://bashscripts.org/forum/

Simple loop to array for a newbie
http://bashscripts.org/forum/viewtopic.php?f=8&t=951
Page 1 of 1

Author:  a_calder [ Mon Jan 04, 2010 7:32 pm ]
Post subject:  Simple loop to array for a newbie

I am pretty new at this, so while it might seem like a very simple request, I'm not at the point where I can write it myself.

I want to schedule a script to run each day, capturing the logins I have had for the previous day. I already wrote a script that scours the logs of my app and gives me the date, the total number of logins and the number of unique users. I would like to have a script that parses the two numbers into two strings that I can use to insert into a Google Charts URL for a WGET.

Here is what I have:
2010-01-01, 4485, 3148
2010-01-02, 5817, 3954
2010-01-03, 10671, 6503

This is what I want:
$LOGINS_TOTAL=4485,5817,10671
$LOGINS_UNIQUE=3148,3954,6503

Is this pretty easy? Anyone want to take a crack at it?

Thanks
A

Author:  fredrik.eriksson [ Tue Jan 05, 2010 3:10 am ]
Post subject:  Re: Simple loop to array for a newbie

I'm thinking the use of arrays would be suitable for this.
Code:
#!/bin/bash
# Make a human-readable version of the first argument passed to the script (./script.sh <argument 1>)
file=$1
# reset the counter
i=0
# Do a line by line loop of the file supplied in argument 1 (or as it is now known, $file)
# The IFS variable tells us what we should look for to create an itteration inside for(1)
# old_IFS is just a way to reset the IFS when we're done :)
old_IFS=$IFS
IFS="
"
# The for(1) loop will read the $file and sort it according to column 1, the date in your supplied example.
for i in $(cat $file | sort -t',' -k1,1); do
   # This adds the second column to an array named "array1"
   array1[$i]=$(echo $i | cut -d',' -f2)
   # This adds the third column to an array named "array2"
   array2[$i]=$(echo $i | cut -d',' -f3)
   # Increment the elemental counter :)
   ((i++))
done
IFS=$old_IFS
# Echo the entire array (and since it's space separated we replace all spaces with commas)
total_logins=$(echo ${array1[*]} | replace " " ",")
total_unique=$(echo ${array2[*]} | replace " " ",")

echo "total_logins=$total_logins"
echo "total_unique=$total_unique"


Something like this might give you a clue on how to do it :) This isn't very well tested... i don't have a proper linux bash shell around here at the moment.

Best regards
Fredrik Eriksson

Author:  a_calder [ Tue Jan 05, 2010 10:18 am ]
Post subject:  Re: Simple loop to array for a newbie

Not sure how this is supposed to work? The file I want to open is in the format of:
DATE, TOTAL LOGINS, UNIQUE LOGINS

I want to parse out the total logins, then the unique logins. Each needs to be in a comma separated string that will end up like: 123,234,345,456,567

This is so I can use it in a WGET to a Google Charts URL

Thx
A

Author:  fredrik.eriksson [ Tue Jan 05, 2010 10:57 am ]
Post subject:  Re: Simple loop to array for a newbie

You didn't read the comments did you?

The script as it is now (i copied it flat out and did the same with your example) creates 2 variables at the end.
$total_logins
$total_unique

These 2 are filled with a comma-separated list of the second column's data (the one you designated as total logins) and the third column's data (total unique).
Then it just echo's it as a proof of concept.
Code:
hanna:~> ./test.sh test.txt
total_logins=4485,5817,10671
total_unique=3148,3954,6503


You'll have to modify the output so it suits your needs.
If you don't have a file then you can just replace the line:
for i in $(cat $file | sort -t',' -k1,1); do

with
for i in $(command which you want to run goes here); do

As long as the list is in the same order as your example it doesn't really matter where the data comes from

Best regards
Fredrik Eriksson

Author:  a_calder [ Tue Jan 05, 2010 11:54 am ]
Post subject:  Re: Simple loop to array for a newbie

Actually, I didn't get a chance to read the comments. I was surfing on my iPhone this morning to see if there were any replies to my post. The iPhone doesn't render the code field properly so I only saw about 4 lines.

Sorry about that!

The script looks great - thanks for the help.

A

Author:  a_calder [ Tue Jan 05, 2010 12:11 pm ]
Post subject:  Re: Simple loop to array for a newbie

On Solaris, I don't seem to have replace. At least the errors I get indicate such:
Code:
./login-count: line 23: replace: command not found
./login-count: line 24: replace: command not found

Should I be trying sed instead of replace?

Author:  a_calder [ Tue Jan 05, 2010 12:29 pm ]
Post subject:  Re: Simple loop to array for a newbie

Got it to work using this:
Code:
total_logins=$(echo ${array1[*]} | sed -e 's/[ ]\{1,\}/,/g')
total_unique=$(echo ${array2[*]} | sed -e 's/[ ]\{1,\}/,/g')

Now, is there a way to get the last 10 sets of numbers out of that file?

Author:  a_calder [ Tue Jan 05, 2010 1:28 pm ]
Post subject:  Re: Simple loop to array for a newbie

Okay, I think I figured it all out:
Code:
#!/bin/bash
# Parse data from login file to pass to Google Charts

# Locate the stat file
statfile="../statsdata/login.stats"

# Read in the last 10 entries
file=`tail -10 $statfile`

# reset the counter
i=0

# Do a line by line loop of the file supplied in argument 1 (or as it is now known, $file)
# The IFS variable tells us what we should look for to create an itteration inside for(1)
# old_IFS is just a way to reset the IFS when we're done :)
old_IFS=$IFS
IFS="
"
# Sort the total and unique logins
for i in $file; do
   # This adds the second column to an array named "array1"
   array1[$i]=$(echo $i | cut -d',' -f2)
   # This adds the third column to an array named "array2"
   array2[$i]=$(echo $i | cut -d',' -f3)
   # Increment the elemental counter :)
   ((i++))
done
IFS=$old_IFS

# Echo the entire array (and since it's space separated we replace all spaces with commas)
total_logins=$(echo ${array1[*]} | sed -e 's/[ ]\{1,\}/,/g')
total_unique=$(echo ${array2[*]} | sed -e 's/[ ]\{1,\}/,/g')

# Print the result
echo "total_logins=$total_logins"
echo "total_unique=$total_unique"


Thanks very much for your help! It works great.

Author:  a_calder [ Mon Jan 11, 2010 4:27 pm ]
Post subject:  Re: Simple loop to array for a newbie

Okay, ran into a bit of a snag. The script worked great until the other day. Then I started getting the following error:

Code:
line 53: 2010-01-08: value too great for base (error token is "08")


Line 53 is part of the following:
Code:
   50  # The for(1) loop will read the $file and sort it according to column 1, the date in your supplied example.
    51  for i in $MYFILE; do
    52     # This adds the second column to an array named "array1"
    53     array1[$i]=$(echo $i | cut -d',' -f2)
    54     # This adds the third column to an array named "array2"
    55     array2[$i]=$(echo $i | cut -d',' -f3)
    56     # Increment the elemental counter :)
    57     ((i++))
    58  done
    59  IFS=$old_IFS


I get the feeling that it is because the 08 in the date is being interpreted as an octal. I can't change the way the date is being stored, as it is an enterprise application I don't control the configuration of. Is there a way I can get the script to interpret the number as base-10 instead?

A

Author:  gofree [ Sun Jan 24, 2010 7:28 am ]
Post subject:  Re: Simple loop to array for a newbie

a_calder wrote:
Quote:
I already wrote a script that scours the logs of my app and gives me the date, the total number of logins and the number of unique users. I would like to have a script that parses the two numbers into two strings that I can use to insert into a Google Charts URL for a WGET.

Hi, that sounds like fun...

It sounds like something I would write as a function in the larger problem. Can I see the script that gets the stats? :)

g

Author:  gofree [ Mon Feb 01, 2010 1:40 pm ]
Post subject:  Re: Simple loop to array for a newbie

Here's how I would do it:
Code:
#!/bin/bash

echo; echo "[fn: distinguish_logins]"; echo

#function distinguish_logins ()
#{
rm ./logins_total.log
rm ./logins_unique.log
unset LOGINS_TOTAL[*]
unset LOGINS_UNIQUE[*]

echo "cat ./log_records.log"
echo "~~~~~~~~~~~~~~~~~~~~~"
cat ./log_records.log
echo

printf "%-11s" "Record"
printf "%+11s" "Log Total"
printf "%+11s\n" "Log Unique"
printf "%-11s" "~~~~~~"
printf "%+11s" "~~~~~~~~~"
printf "%+11s\n" "~~~~~~~~~~"

let -i count=0
for LOG in $(cat ./log_records.log); do
   let count=count+1
   echo $LOG | cut -d, -f2 >> ./logins_total.log
   echo $LOG | cut -d, -f3 >> ./logins_unique.log
   LOG_DATE[$count]=$(echo $LOG | cut -d, -f1)
   LOGINS_TOTAL[$count]=$(echo $LOG | cut -d, -f2)
   LOGINS_UNIQUE[$count]=$(echo $LOG | cut -d, -f3)
   printf "%-11.10s" ${LOG_DATE[$count]}
   printf "%+11.10s" ${LOGINS_TOTAL[$count]}
   printf "%+11.10s\n" ${LOGINS_UNIQUE[$count]}
#   echo "${LOGINS_TOTAL[$count]} ${LOGINS_UNIQUE[$count]}"
done
echo

echo "Logins Total:"
echo "~~~~~~~~~~~~~"
cat ./logins_total.log
echo
echo "Logins Unique:"
echo "~~~~~~~~~~~~~~~"
cat ./logins_unique.log
echo

echo -n "LOGINS_TOTAL="
for (( i = 1; i <= $count; i++ )); do echo -n "${LOGINS_TOTAL[$i]},"; done; echo
echo -n "LOGINS_UNIQUE="
for (( j = 1; j <= $count; j++ )); do echo -n "${LOGINS_UNIQUE[$j]},"; done; echo
echo "[done]"; echo

#}

exit 0


Giving this as output:
Code:
[fn: distinguish_logins]

cat ./log_records.log
~~~~~~~~~~~~~~~~~~~~~
2010-01-01,4485,3148
2010-01-02,5817,3954
2010-01-03,10671,6503

Record       Log Total Log Unique
~~~~~~       ~~~~~~~~~ ~~~~~~~~~~
2010-01-01        4485       3148
2010-01-02        5817       3954
2010-01-03       10671       6503

Logins Total:
~~~~~~~~~~~~~
4485
5817
10671

Logins Unique:
~~~~~~~~~~~~~~~
3148
3954
6503

LOGINS_TOTAL=4485,5817,10671,
LOGINS_UNIQUE=3148,3954,6503,
[done]



Of course, I would want to insist that the spaces between fields be taken out of the original log records.
>:)

Author:  Watael [ Mon Feb 01, 2010 2:55 pm ]
Post subject:  Re: Simple loop to array for a newbie

eww!
Code:
#!/bin/bash

while IFS=' ,' read DATE TOTAL UNIQUE
do LOGINS_TOTAL+=( "$TOTAL" )
   LOGINS_UNIQUE+=( "$UNIQUE" )
done < <(tail -10 ../statsdata/login.stats)

# Echo the entire array (and since it's space separated we replace all spaces with commas)
# and Print the result
echo "total_logins=$(sed -e 's/[ ]\{1,\}/,/g' <<<"${LOGINS_TOTAL[@]}")"
echo "total_unique=$(sed -e 's/[ ]\{1,\}/,/g' <<<"${LOGINS_UNIQUE[@]}")"

Author:  gofree [ Wed Feb 03, 2010 10:48 am ]
Post subject:  Re: Simple loop to array for a newbie

Nice, Watael!

I am saving this for future reference.

g

Page 1 of 1 All times are UTC - 6 hours
© 2000, 2002, 2005, 2007 phpBB Group • http://www.phpbb.com