Register
It is currently Mon Sep 01, 2014 3:03 pm

Copy Modified Files Recursively


All times are UTC - 6 hours


Post new topic Reply to topic  [ 14 posts ] 
Author Message
 PostPosted: Mon Aug 21, 2006 7:18 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
I've got a script that will get all the files modified today and I want to copy them to my webserver which has the same directory structure. The only problem I'm having it cp isn't copying it to the directory, its just copying to the root and if I use -r it copies the entire directory. Here's my script so far

find DIR1/* -mtime 0 -name '*' -type f -exec cp -v {} DIR2 \;

any help would be great


Top
 Profile  
 PostPosted: Mon Aug 21, 2006 8:45 pm   
User avatar

Joined: Mon Jul 03, 2006 8:58 pm
Posts: 52
Location: Rochester, NY
I'd do a simple bash for loop to solve this one:

Code:
for file in `find DIR1/* -mtime 0 -name '*' -type f`; do
    cp -r $file TODIR/
done;


Pay particular attention to the difference between a backtick (` unshifted tilde ~ key) before "find" and after "f"and an apostrophe (' unshifted quote " key) around "*" - it's very important!


Top
 Profile WWW  
 PostPosted: Mon Aug 21, 2006 10:38 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
that didn't work either. It didn't copy the files into their respective directory in the new folder


Top
 Profile  
 PostPosted: Mon Aug 21, 2006 10:58 pm   
Site Admin
User avatar

Joined: Sun May 15, 2005 9:36 pm
Posts: 666
Location: Des Moines, Iowa
jgallen23 wrote:
I've got a script that will get all the files modified today and I want to copy them to my webserver which has the same directory structure.


uh..... not to be a killjoy, but isn't that what rsync does ?????


Top
 Profile WWW  
 PostPosted: Tue Aug 22, 2006 4:19 am   
User avatar

Joined: Mon Jul 03, 2006 8:58 pm
Posts: 52
Location: Rochester, NY
Sorry, my command was a bit wrong, but the concept works. This loop actually does what you want it to do:
Code:
for file in `find DIR1/* -mtime 0 -name '*' -type f`; do
    echo $file
    cp $file `echo $file | sed 's/DIR1/DIR2/'`
done;


It'll echo the file being copied to the screen -- remove the first echo to get rid of that.

This script assumes your base directory (DIR1) is different than your target directory DIR2, but the structure of each are identical.


Top
 Profile WWW  
 PostPosted: Tue Aug 22, 2006 12:56 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
crouse - can rsync just sync after a certain time? I want to be able to just transfer stuff changed today


Top
 Profile  
 PostPosted: Tue Aug 22, 2006 1:09 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
BrionS,

I'm pretty new to bash scripting, could you explain this line of code:

cp $file `echo $file | sed 's/DIR1/DIR2/'`

Thanks
JGA


Top
 Profile  
 PostPosted: Tue Aug 22, 2006 4:22 pm   
User avatar

Joined: Mon Jul 03, 2006 8:58 pm
Posts: 52
Location: Rochester, NY
Sure thing!

That's a compound line - it's running two commands at once through use of the backticks "`".

Like order of operations - the script will execute what's in the backticks first, and then send its output to the containing statement.

So first it executes
Code:
echo $file | sed 's/DIR1/DIR2/'

which is a non-elegant way to substitute all the occurances of "DIR1" in the $file (path) with "DIR2". In this case, DIR1 and DIR2 would be replaced with your source and target root folders.

The output of that statement is the modified file path to point to DIR2/path instead of DIR1/path, so the subsequent statement becomes:
Code:
cp DIR1/path DIR2/path


The end result is that a file from the source directory that was found by your 'find' command will be copied to the target directory in the same location. Here's an example of a "real" file as it's transformed by this script:

Code:
# we want to syncronize this file: /source/my/dir/testfile to /dest/my/dir/testfile
# so we'll change the script to look like this:
for file in `find /source/* -mtime 0 -name '*' -type f`; do
    echo $file
    cp $file `echo $file | sed 's/source/dest/'`
done;

The output of this program will be:
Code:
/source/my/dir/testfile

indicating that it copied that file (though it doesn't indicate where to) -- this behavior can easily be modified.

The end result is now you have a copy of the /source/my/dir/testfile in /dest/my/dir/.

So, I'll modify this script to be a little more verbose and a little more parameterized...

Code:
#!/bin/bash
#
# Script to synchronize files modified today from $SRC to $DEST as
# specified by the first and second command-line parameters

SRC = $1
DEST = $2

echo Synchronizing $SRC to $DEST...
for file in `find $SRC/* -mtime 0 -name '*' -type f`; do
    dfile = `echo $file | sed 's/$SRC/$DEST/'`
    echo copying $file to $dfile
    cp $file $dfile
done;
echo Synchronization complete.


Now you can call the program thusly (saved as bsync.sh) using the previous example:
Code:
brion> sh bsync.sh /source /dest
Synchronizing /source to /dest...
copying /source/my/dir/testfile to /dest/my/dir/testfile
Synchronization complete.
brion>


sed is a Stream EDitor that permits you to, among other things, substitute text in a stream with other text -- in this case, the $SRC with the $DEST directory names in the source file path.

Does that help?


Top
 Profile WWW  
 PostPosted: Tue Aug 22, 2006 5:30 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
thanks BrionS, that explanation helped a lot.

I found a couple of issues. For some reason the $dfile ends up the same as $file so obviously, it doesn't get copied. The other issue I found is if the file name has a space in it. I'll probably just rename those files as file names should probably not have a space in them.

JGA


Top
 Profile  
 PostPosted: Tue Aug 22, 2006 5:51 pm   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
I found out the sed problem has to do if I pass a / in the directory

so if I do ~/copytest, sed returns an error


Top
 Profile  
 PostPosted: Tue Aug 22, 2006 9:47 pm   
User avatar

Joined: Mon Jul 03, 2006 8:58 pm
Posts: 52
Location: Rochester, NY
I apologize. I wrote a script without testing it. :( :(

You're right - it's got all sorts of problems which frankly I don't quite understand. However, I'm pretty sure if you hardcode the source and dest directories you can get it to work (instead of using variables).

Also in sed, to use a '/' character, you need to escape it with '\' such as:
Code:
sed 's/\/source/\/dest/'


Another thing wrong with my second script is the SRC and DEST need to remove the spaces
Code:
SRC=$1
DEST=$2


As I get time I'll debug the rest - maybe Crouse or jbsnake will pop by and correct my syntax.


Top
 Profile WWW  
 PostPosted: Wed Aug 23, 2006 1:29 am   
Site Admin
User avatar

Joined: Sun May 15, 2005 9:36 pm
Posts: 666
Location: Des Moines, Iowa
again...... i REALLY think you need to "man rysnc" ;) Here is one of the scripts i use......

Code:
---------- Begin remote_backup1.sh -------------------------------------------------------------------------------------

#!/bin/sh

# remote_backup.sh
# first let us designate the host or ip address
# of the remote server and local target directory
# we wish to send to

REMOTE_HOST="enter host here"
LOCAL_TARGET="/mnt/backup/backup1/"

# Now for the actual work part of this script

/usr/bin/rsync --exclude proc/ --exclude var/spool/ --exclude var/mail/ \
--exclude var/tmp/ --exclude dev/ --exclude boot/ --exclude initrd/ \
--exclude tmp/ --exclude home/*/mail/ --exclude var/log/dcpumon/ \
-azv -H -e ssh root@$REMOTE_HOST:/ $LOCAL_TARGET

# make sure the directory reflects the date and time last updated

touch $LOCAL_TARGET

------ end script ------------------------------------------------------------


Daily, this backs up ONLY the CHANGED files from certain directories.... notice the --exclude stuff.
On initial run... it backs up EVERYTHING ....then it only "syncs" the new data from there on out..... very cool, and when used in conjunction with cron, makes for a hands free painless backup setup.

Hope that is of some help


Top
 Profile WWW  
 PostPosted: Wed Aug 23, 2006 4:05 am   
User avatar

Joined: Mon Jul 03, 2006 8:58 pm
Posts: 52
Location: Rochester, NY
I guess was just hoping for some pointers as to why my script didn't work - rsync aside. I'm not completely familiar with sed yet and from a theoretical standpoint I don't see why this script doesn't at least copy the files it finds.

Consider it practice in script writing - just like writing a compiler to learn how one works, it's not like the world needs another compiler. :)


Last edited by BrionS on Wed Aug 23, 2006 1:41 pm, edited 1 time in total.

Top
 Profile WWW  
 PostPosted: Wed Aug 23, 2006 8:03 am   

Joined: Mon Aug 21, 2006 12:37 pm
Posts: 10
the reason I didn't want to use rsync is somebody in a previous post said that rsync doesn't work well syncing two windows boxes (can somebody confirm this?). Another reason is that this isn't a backup script, this is an upload script to my webserver. This lets me quickly get the files I've changed today and upload them.

Thank you to both crouse and BrionS for your scripts.

JGA


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 14 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 6 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP