Register
It is currently Thu Jul 24, 2014 3:41 pm

Get Size of Data in folder and email the result, plus ....


All times are UTC - 6 hours


Post new topic Reply to topic  [ 12 posts ] 
Author Message
 PostPosted: Tue Jan 13, 2009 1:31 am   

Joined: Tue Jan 13, 2009 1:10 am
Posts: 5
Hi,

I run a server that stores a vast amount of information in this format:

/data/username1/2778346321/files
/data/username1/2892029482/files
/data/username1/4433229482/files

/data/username2/4556732282/files
/data/username2/1123736738/files
etc

I need some help to put together a bash script to do..

1. Read from a plain text file what usernames to check (located anywhere on server - not important where)
2. Check usage for the users individual directories (du -h)
2. Read an xml file in the files directory to extract a translation of what the numbered directory is. Our system stores a plain English name of the numbered directory, IE: JohnsPc in the XML file
3. Produce and email like so ..

User: Username1
JohnsPc = 12GB (this is actually df -h of the 2778346321 directory)
MarysPc = 4GB
JacksPc = 1GB
etc

It would then email to an address located in the plain text file with the username, the results above.

IE: The plain text file, is so I can edit it and easily add new usernames and email addresses to it.

EG, Text File would look like
username1,admin@xyz.com
username2,admin@xyz.com
username3,monitor@abc.com
username4,support@sss.com
etc


The xml file for each numbered dir looks like this, below. And is in a file called info.xml in the root of the each numbered dir.
<Set>
<Name="JohnsPc" id="2778346321" />
</Set>


I feel I am asking a lot here. Is it possible to even do this?
My command line experience is very limited to very basic things only. But always looking to learn more.

Appreciate any feedback at all.


Top
 Profile  
 PostPosted: Tue Jan 13, 2009 12:48 pm   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
Hi,

A few clarifications required:

> 1. Read from a plain text file what usernames to check (located anywhere on server - not important where)

Do you want to check multiple users or a single user at a time?

> 2. Read an xml file in the files directory to extract a translation of what the numbered directory is. Our system stores a plain English name of the numbered directory, IE: JohnsPc in the XML file
> The xml file for each numbered dir looks like this, below. And is in a file called info.xml in the root of the each numbered dir.

Where is the XML file actually located? In /data/username1/ (a single xml file for all numbered dirs) or in /data/username1/<numbered dir>/ (a separate xml fille for each numbered dir) ?

Regards,
Lijeesh


Top
 Profile WWW YIM  
 PostPosted: Tue Jan 13, 2009 7:58 pm   

Joined: Tue Jan 13, 2009 1:10 am
Posts: 5
Hi lijeesh,

> Do you want to check multiple users or a single user at a time?

Check all users who are in the text file. So ideally loop through each one at a time.

> Where is the XML file actually located? In /data/username1/ (a single xml file for all numbered dirs) or in /data/username1/<numbered dir>/ (a separate xml fille for each numbered dir) ?

There is one file for every numbered dir in there own files folder: /data/username1/2778346321/files/info.xml
The contents of the xml file look exactly like this...
<Set>
<Name="JohnsPc" id="2778346321" />
</Set>

There is nothing else in the file except the above.

So basically one username may have multiple numbered dirs with the "files" folder and inside that each having an xml file referencing a human readible name for the numbered dir.

I hope that helps.
Appreciate your help.


Top
 Profile  
 PostPosted: Sun Jan 18, 2009 1:36 am   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
Hi,

Here is the script. The script should be run in the following format:

script_name -u user_file -e email_file

Working of this script can be summarised as below:

1. Loop through the user file and for each user, perform 2 - 3.
2. Find out the logical name and disk usage of each numbered directory in /data/username.
3. Find out user's email address from email file and email the disk usage stats.


Code:

#!/bin/bash


# data directory
datadir=/data

# user file
email_file=''

# email file
email_file=''

# report file
report=/tmp/report.$$

usage()
{
  echo "Usage: $0 -u user_file -e email_file"
}

while getopts :e:u: opt
do
  case $opt in
    e)  if [ -f $OPTARG ]
        then
          email_file=$OPTARG
        else
          echo "Invalid email file. Aborting."
          exit
        fi
   ;;
    u)  if [ -f $OPTARG ]
   then
     user_file=$OPTARG
   else
          echo "Invalid user file. Aborting."
          exit
        fi
   ;;
  esac
done

if [ "$user_file" = "" -o "$email_file" = "" ]
then
  echo "user/email file not set."
  usage
  exit
fi

# loop through the user file

for user in $(cat $user_file)
do
  if [ ! -e /$datadir/$user -o ! -d /data/$user ]
  then
    echo "$user: invalid or non-existent directory /$datadir/$user"
    continue
  fi
  echo "User: $user" > $report
  for dir in $(ls -d /$datadir/$user/[0-9]*)
  do
    logical_name=$(grep "Name" $dir/files/info.xml | sed -r 's/.*Name="(.*)".*id.*/\1/g')
    disk_usage=$(du -sh $dir | awk '{print $1}')
    echo "$logical_name = $disk_usage" >> $report
  done
  user_email=$(grep -E "\s*$user\s*," $email_file | awk -F, '{print $2}')
  echo "emailing $user"
  # emai the user
  cat $report | mail -s "Disk usage stats" $user_email
  cat /dev/null > $report
done

rm -f $report



Thanks,
Lijeesh


Top
 Profile WWW YIM  
 PostPosted: Sun Jan 18, 2009 11:33 pm   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
BTW the script expects the user and email files in the following formats:

user file

user1 user2 user3

OR

user1
user2
user3

email file

user1,email1
user2,email2
user3,email3


Top
 Profile WWW YIM  
 PostPosted: Tue Jan 20, 2009 3:12 pm   

Joined: Tue Jan 13, 2009 1:10 am
Posts: 5
Hi Lijeesh,

Thank you very much.

That works perfectly, exactly what I was after.

There was a couple of small typos that I needed to correct in order for it to work of which I will post here, for the sake of the forum.

1. The user_file variable had the wrong name

Was
Code:
# user_file
email_file=''


Changed to
Code:
# user_file
user_file=''


2. An If statement had a missing variable (probably missed out when you added the variables after testing)

Was
Code:
if [ ! -e /$datadir/$user -o ! -d /data/$user ]


Changed to
Code:
if [ ! -e /$datadir/$user -o ! -d /$datadir/$user ]


----

Curious to know...

How easy or difficult would it be to have an html template file that I could create, and then insert the results and send as an html formatted email?

Many thanks!


Last edited by axrock on Wed Jan 21, 2009 9:19 pm, edited 1 time in total.

Top
 Profile  
 PostPosted: Tue Jan 20, 2009 11:41 pm   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
Dear Friend,

You are right, those are typos, sorry for that. May be I was a bit sleepy at that time :-P

Regards,
Lijeesh


Top
 Profile WWW YIM  
 PostPosted: Wed Jan 21, 2009 12:02 am   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
Hi,

The difficulty of inserting the results in a html template depends on the structure of the template. Let me show you a simple example. Suppose the template is as follows and you want to put the results within the <pre></pre> tags.

<html>
<body>
<pre>
</pre>
</body>
</html>

Then you can divide the template file into two as follows:

template1:

<html>
<body>
<pre>

template2:
</pre>
</body>
</html>

Now put template1 at the beginning and tempate2 at the end of the results. So the following code

Code:
  echo "User: $user" > $report
  for dir in $(ls -d /$datadir/$user/[0-9]*)
  do
    logical_name=$(grep "Name" $dir/files/info.xml | sed -r 's/.*Name="(.*)".*id.*/\1/g')
    disk_usage=$(du -sh $dir | awk '{print $1}')
    echo "$logical_name = $disk_usage" >> $report
  done


can be modified as below:

Code:
  cat $tempate1 > $report
  echo "User: $user" >> $report
  for dir in $(ls -d /$datadir/$user/[0-9]*)
  do
    logical_name=$(grep "Name" $dir/files/info.xml | sed -r 's/.*Name="(.*)".*id.*/\1/g')
    disk_usage=$(du -sh $dir | awk '{print $1}')
    echo "$logical_name = $disk_usage" >> $report
  done
  cat $tempate2 >> $report


You need to set the variables template1 and template2 to the path of corresponding template file.

This is the quick and simple idea on my head. But it depends on the structure of template file and how you want to integrate the results with the template.

Regards,
Lijeesh


Top
 Profile WWW YIM  
 PostPosted: Wed Jan 21, 2009 12:20 am   

Joined: Tue Jan 13, 2009 1:10 am
Posts: 5
Hello again,

What you have suggested will do the job nicely.

Does the mail function require any special headers for html, or does it just work. I know when I used to work with PHP you had to specify that the email was html and not just text.

Thanks for the help.


Top
 Profile  
 PostPosted: Wed Jan 21, 2009 1:40 am   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
Hi,

For MIME support, better we use the sendmail program instead of mail program. So the code

Code:
  echo "User: $user" > $report
  for dir in $(ls -d /$datadir/$user/[0-9]*)
  do
    logical_name=$(grep "Name" $dir/files/info.xml | sed -r 's/.*Name="(.*)".*id.*/\1/g')
    disk_usage=$(du -sh $dir | awk '{print $1}')
    echo "$logical_name = $disk_usage" >> $report
  done
  user_email=$(grep -E "\s*$user\s*," $email_file | awk -F, '{print $2}')
  echo "emailing $user"
  # emai the user
  cat $report | mail -s "Disk usage stats" $user_email
  cat /dev/null > $report


can be modified as below:

Code:
  # Find user's email address
  user_email=$(grep -E "\s*$user\s*," $email_file | awk -F, '{print $2}')

  # Add email header to report file
  { echo "To: $user_email"
    echo 'From: Monitor <you@domain.com>'   # You can remove From if you want
    echo 'Subject: Disk Usage'
    echo 'Content-Type: text/html; charset="us-ascii'
  } > $report

  # Add template1 to report file
  cat $tempate1 >> $report

# Add results to report file
  echo "User: $user" >> $report
  for dir in $(ls -d /$datadir/$user/[0-9]*)
  do
    logical_name=$(grep "Name" $dir/files/info.xml | sed -r 's/.*Name="(.*)".*id.*/\1/g')
    disk_usage=$(du -sh $dir | awk '{print $1}')
    echo "$logical_name = $disk_usage" >> $report
  done

  # Add template2 to report file
  cat $tempate2 >> $report
  echo "emailing $user"

  # email the user
  cat $report | sendmail $user_email

  # Empty the report file for next user
  cat /dev/null > $report


Thanks,
Lijeesh


Top
 Profile WWW YIM  
 PostPosted: Wed Jan 21, 2009 9:22 pm   

Joined: Tue Jan 13, 2009 1:10 am
Posts: 5
Thanks Lijessh!

Really appreciate the time and effort you have given to put this together for me.

I finally have it all working just as i had hoped thanks to your scripting skills.

When I get som free time, I really need to study this up a bit more. But working with your script has already taught me quite a bit.

Many thanks.


Top
 Profile  
 PostPosted: Wed Jan 21, 2009 9:38 pm   

Joined: Tue Dec 05, 2006 8:37 pm
Posts: 30
Location: Thiruvananthapuram / India
I'm glad that I could help you :-)


Top
 Profile WWW YIM  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 12 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Bing [Bot] and 7 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP