Register
It is currently Thu Dec 14, 2017 6:27 pm

script to tar/rsync/rm multiple folder names


All times are UTC - 6 hours


Post new topic Reply to topic  [ 14 posts ] 
Author Message
 PostPosted: Tue Apr 12, 2016 4:39 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
hi all,

i attach a link with what im trying to do automatically via script but i have some questions i need answering please, bear in mind i am really new to bash scripting, the only thing i know how to do is commands in scripts like cd rm tar rsync cp stuff like that

http://s20.postimg.org/q9hpgtlzh/archive.jpg

i have mutiple project folders in the "to_be_archived" folder ie

batman
superman
hulk
spiderman
iron_man
etc etc...

so it makes a tar file of the folder, rsyncs tar file to another folder "archived_projects" and deletes the tar file and folder in "to_be_archived"

my example i just did a test with a untitled folder to check to see if it works and it does

i want it to -

do all folders in the list and not just one specific folder, so once it does the 1st one it does the 2nd one so on so forth

i want to check before it deletes the folder and tar file in the (to_be_archived) i want to double check if it has rsynced the whole file across to (archived_projects)

cheers,

rob


Top
 Profile  
 PostPosted: Tue Apr 12, 2016 6:54 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
What's the real goal of all this?

rsync has a very sophisticated algo to transfer a minimum of data only and does even compressing on the fly.
One rsync command can do this all.


Top
 Profile  
 PostPosted: Tue Apr 12, 2016 7:14 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
i work in a media post house and we work on different projects ie media we get from clients and we save all there media on a NAS, so once all the different departments have worked on that project folder to finalise it, i want to move it to the "to_be_archived" folder and from there i want to devise a script that will do what i mentioned above


Top
 Profile  
 PostPosted: Tue Apr 12, 2016 7:25 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
So just use one rsync command and you are done.


Top
 Profile  
 PostPosted: Tue Apr 12, 2016 8:31 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
thanks but how do i do all folders in the list and not just one specific folder, so once it does the 1st one it does the 2nd one so on so forth


Top
 Profile  
 PostPosted: Wed Apr 13, 2016 9:56 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
getting there

#!/bin/bash
cd /to_be_archived/
for DIR in * ; do
fSaveDir="${DIR##*/}"
tar -cf "${fSaveDir}".tar "${fSaveDir}"
rsync -a "${fSaveDir}".tar /archived_projects/
rm -f "${fSaveDir}".tar
rm -rf "${fSaveDir}"
done


Top
 Profile  
 PostPosted: Wed Apr 13, 2016 3:43 pm   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
please use code tags and indent your code properly.
Press "Full Editor".

And read the man page of rsync.
You want some options to not delete the already archived tar files.


Top
 Profile  
 PostPosted: Fri Apr 15, 2016 6:31 am   
Moderator
User avatar

Joined: Thu Oct 11, 2007 7:12 am
Posts: 265
Location: London - UK
Not sure why you would need this line...
Code:
fSaveDir="${DIR##*/}"

...but maybe your version of bash is different to mine.

Also why not create the tar file in it's destination rather than moving it later?
Code:
tar -cf "/archived_projects/${fSaveDir}".tar "${fSaveDir}"

You might also want to add compression (-z) to that.

An option to check the tar command finished correctly might be to check it's exit status.
Code:
tar -cf "/archived_projects/${fSaveDir}".tar "${fSaveDir}"
if [ $? -gt 0 ]
then
  echo  an error occured
fi


I hope these help,
DW


Top
 Profile  
 PostPosted: Fri Apr 15, 2016 10:29 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
job done -

Code:
#!/bin/bash
cd /to_be_archived/
for DIR in * ; do
fSaveDir="${DIR##*/}"
tar -cf "${fSaveDir}".tar "${fSaveDir}"
rsync -a "${fSaveDir}".tar /archived_projects/
if [ $? -ne 0 ]
then
mail -s "${fSaveDir}" [email protected] <<< "project "${fSaveDir}" aborted due to error"
else
rm -f "${fSaveDir}".tar
rm -rf "${fSaveDir}"
fi
done


Top
 Profile  
 PostPosted: Fri Apr 15, 2016 12:52 pm   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
Again no proper indenting of posted code
and useless copying of data.


Top
 Profile  
 PostPosted: Mon Apr 25, 2016 8:13 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
Code:
#!/bin/bash
cd /to_be_archived/
for DIR in * ;

   do
   MailAddress="[email protected]"

   if ! tar -cf "$DIR".tar "$DIR"
   then
   mail -s "tar failed $DIR" $MailAddress <<< "creating of tar $DIR failed due to error, removing $DIR.tar"
   rm -f "$DIR".tar
   continue
   fi

   if [ -f /archived_projects/"$DIR".tar ]; then
     mail -s "duplicate exists $DIR" $MailAddress <<< "$DIR.tar already exists"
   rm -f "$DIR".tar
   continue
   fi

   if ! rsync -a "$DIR".tar /archived_projects/
   then
   mail -s "rsync failed $DIR" $MailAddress <<< "rsync of $DIR failed due to error, removing $DIR.tar"
   rm -f "$DIR".tar
   continue
   fi

   if ! rm -f "$DIR".tar
   then
   mail -s "remove tar failed $DIR" $MailAddress <<< "removing of tar $DIR failed due to error"
   continue
   fi

cd "$DIR"
date >> /ifs/data/cha/cha-it/it/archive_details.txt
echo -n  "Folder to be archived = " >> /ifs/data/cha/cha-it/it/archive_details.txt
pwd >> /ifs/data/cha/cha-it/it/archive_details.txt
echo -n  "Number of files =       " >> /ifs/data/cha/cha-it/it/archive_details.txt
find . -type f | wc -l >>  /ifs/data/cha/cha-it/it/archive_details.txt
echo -n  "Size in GB =            " >> /ifs/data/cha/cha-it/it/archive_details.txt
du . -s -B 1G >>  /ifs/data/cha/cha-it/it/archive_details.txt
echo " " >>  /ifs/data/cha/cha-it/it/archive_details.txt
cd ..

   if ! rm -rf "$DIR"
   then
   mail -s "remove folder failed $DIR" $MailAddress <<< "removing of $DIR failed due to error"
   continue
      else
      mail -s "success $DIR" $MailAddress <<< "successfully completed archiving $DIR"
   fi

   done


also i want to add to my script that if the directory stays the same size then it is safe to start of the script but if the directory changes size ie someones is still moving the directory in "to_be_archived" dont do it and wait till the directoey doesnt increase in size and then it can start off the script


Top
 Profile  
 PostPosted: Mon Apr 25, 2016 8:31 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
Inendation is meant to make code readable.
It is not a matter of spicing code up using a salt shredder.

You want some kind of waiting loop.
Read man inotifywait


Top
 Profile  
 PostPosted: Thu Apr 28, 2016 5:38 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
inotifywait doesnt work for files remotley ie NFS shares

really struggling with my last command in this script im trying to do

i want to capture the disk usage of the DIR in "to_be_archived" folder and a sleep command for 10 minutes and then run the disk usage command again to see if the two values change

if they have stop doing this DIR and do the next DIR in the "to_be_archived" folder

if the same values match, it can continue doing the tar/rsync/rm commands

im kind of thinking like this -


Code:

du --time "$DIR"/
sleep 600
du --time "$DIR"/


but i dont know how get it to do if the two values change exit, if they stay the same continue

the reason i have used the du --time command is it tells me the modification time and i would rather go by this than the folder size, if that makes sense

thanks


Top
 Profile  
 PostPosted: Thu Apr 28, 2016 7:47 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
something like:
Code:
while true; do
   checkDirSizeAndSetNewSize
   (( oldSize == newSize )) && break
   sleep 10
done
This loop runs forever. After having set the newSize var it does a math compare and breaks the loop, if oldSize equals newSize.
The math operation (( .... )) combined with the conditional concatenation of commands is essentially the same like if ! [[ $oldSize = $newSize ]] ; then break; else doNothing; fi


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 13 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP