It is currently Fri May 25, 2018 2:35 am

backup script to split and tar files and put it on amazon s3

All times are UTC - 6 hours

Post new topic Reply to topic  [ 3 posts ] 
Author Message
 PostPosted: Thu Jun 21, 2012 8:40 pm   

Joined: Thu Jun 21, 2012 8:11 pm
Posts: 3
Hi all,

I'm very new to bash scripting. Kindly please help me on this.

Here's the situation I've created a directory, let's call it /backup it has a subdirectories named after each of the users (e.g user1, user2, user3....). All users will then backup their files (large files) on their respected folders. My task now is to automatically compress all their files and split it if the file is larger than 5GB. Say for example user1 copies all his files that needs to be backup on his folder (/backup/user1). The script will now check that directory and do the necessary actions I mentioned above.

I'm in need of a backup script which does the ff.

1. If a file is larger than 5GB. split it and tar the file. In preparation of,
2. Monthly or weekly backup file to amazon s3 using s3rsync (or if you have recommended tool for putting files to amazon s3)
3. If a file is unchanged it doesn't need to copy to amazon s3

I hope someone can help me on this. Any help is highly appreciated.

I've searched google and can't find a script that suit my needs.

Can anyone post some of their scripts or links for reference which may help?

Thank you.

 PostPosted: Sun Jun 24, 2012 5:39 pm   
User avatar

Joined: Fri Jul 22, 2011 5:39 pm
Posts: 47
I use fuse-over-amazon on a few boxen and s3cmd on another that can't support fuse-over-amazon, so s3cmd was the choice.
Where fuse-over-amazon is used, I mount the s3 bucket(s).
On the box that doesn't support fuse-over-amazon, I use "s3cmd rsync" myself. it allows writing directly to s3 buckets.
but it should be something like this...

tar -pczf /path/to/what/to/tar split -b 5G archive.tar.gz

my script calls this:
/usr/bin/s3cmd sync /c9backups/c9wiki/ s3://c9internal/c9wiki/

NOTE: s3cmd does NOT require mounts. :)
NOTE: I haven't used split before and tar versions "may" differ slightly so the syntax above may not be exactly correct.
If you open a terminal and type "sudo modprobe fuse" and if it doesn't error out, you can use fuse-over-amazon. If it does error, then s3cmd or your choice (s3rync), or another...

Check out


 Profile WWW  
 PostPosted: Wed Jul 11, 2012 12:02 am   

Joined: Thu Jun 21, 2012 8:11 pm
Posts: 3
Thanks HTH,

Another thing, How can I able to compress an unknown file (I don't know the filename)? Say for example. user1 copy his/her file on his/her respected directory /backup/user1 . How should I compress the file that he/she copied on the said directory? Without me knowing the filename of it. What basic script should I use.?


Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 3 posts ] 

All times are UTC - 6 hours

Who is online

Users browsing this forum: No registered users and 9 guests

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  

BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group