Register
It is currently Thu Dec 14, 2017 6:57 am

Program or bash script to see total progress of copy


All times are UTC - 6 hours


Post new topic Reply to topic  [ 9 posts ] 
Author Message
 PostPosted: Thu Sep 14, 2017 6:04 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
hi all,

i want a program or to make a bash script to find out the total ETA/percent (would be nice aswell a progress bar) of a copy recursive command

so lets say i do -

Code:
cp -r /source_folder/ /destinatation_folder/


and when i run it i get no information on the screen of how the copy is progressing

i have tried using "pv" and "progress" commands but to no avail, they just give me the ETA/progress of individual files copied in the directory but not the total ETA/progress of the whole directory

any help,

rob


Top
 Profile  
 PostPosted: Fri Sep 15, 2017 3:04 pm   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
smashed it -

[[email protected] data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
[=> ] 2% ETA 2:34:31

and to find the dir size i did -

du -s call_the_midwife_7_1708/

but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes


Top
 Profile  
 PostPosted: Fri Sep 15, 2017 4:56 pm   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
This is a not that trivial task.
There are some programs doing this, like gcp.

And there are some scripting solutions out addressing this.
Either they are very slow speed or inaccurate.

To roll you own one, you have to gather first the size of the complete directory tree.
Hence you have to walk the entire tree and sum up the size of each file before start copying.

And this demands the parsing of the cmdline on your own.
Which is only doable, if /proc is mounted the normal way.
Then you can read the unaltered cmdline from /proc/$$/cmdline

You have to deal with all kinds of file/path globbing bash offers
to get proper results.

Solutions admitting a fair amount of incorrectness use constructs like "du -s /target/dir".

In general all these computed sizes are inaccurate.

Rethink your question and specify exactly which way you wanna go.


Top
 Profile  
 PostPosted: Fri Sep 15, 2017 4:57 pm   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
robertkwild wrote:
smashed it -

[[email protected] data]# tar -c call_the_midwife_7_1708/ | pv -lep -s 32455212 | tar -x -C /mnt/local/data/new/
[=> ] 2% ETA 2:34:31

and to find the dir size i did -

du -s call_the_midwife_7_1708/

but doing it via this method takes ages as its creating the tar and extracting the tar, normally doing a normal copy only takes roughly 18 minutes
This is one of the inaccurate solutions.


Top
 Profile  
 PostPosted: Wed Sep 20, 2017 2:32 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
just thought of another idea -

il get the size of the source path -

Code:
du -s /source_path/


then i will start the copy -

Code:
cp -r /source_path/ /destination_path/


while im copying i will monitor the progress -

Code:
watch -n 0.5 du -s /destination_path/


but i want to do this all in a bash script but my issue is it wont watch the destination path while the copy is going on, how do i do both at the same time

rob


Top
 Profile  
 PostPosted: Thu Sep 21, 2017 12:02 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
Read man watch and you'll see that this not easily done, if at all.
watch uses an entire screen of the terminal.
You have to deal with all kinds of internals of tty, ptty, kernel features and alike.

It might be easier to delegate the counting work to some background function and feed back the output to the copying script by using some trap function.

But again: will be slow or inaccurate.


Top
 Profile  
 PostPosted: Thu Sep 21, 2017 7:05 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
i have seen this -

https://stackoverflow.com/questions/238 ... es#tab-top

if you read the answer by mitch he has made a script that does this with echo commands

but when i try it and run it on my linux box i just get 33, 66, 100% with the hashes, how can i get this to count up from 1-100

also how can i implement this with my copy command "cp -r /source /dest"

many thanks,

rob


Top
 Profile  
 PostPosted: Thu Sep 21, 2017 9:59 am   

Joined: Mon Oct 20, 2014 9:53 am
Posts: 560
Pseudocode:
Code:
getTotal BytesToCopy
trap 'cleanup' EXIT
trap 'progressbar' SIGUSR1

count_function(){
    PidOfMainScript=$1
    bytes_now=$(countAlredyCopiedBytes)
    count_percentage=$(calculatePercentage)  # get it somehow
    (( count_persentage > 100 )) && count_percentage="done"
    echo $count_percentage > PercentageTempFile
    kill -SIGUSR1 $PidOfMainScript
}
cleanup(){
    cleanUpTheMess
}

progressbar(){
    until [[ $percentage == "done" ]]; do
       percentage=$(cat PercentageTempFile)
       # use a token to indicate that job has finished.
       echoCommandsHere  # use printf ; it's easier
    done   
}
startCopyJobHere & # and send it to background

this_pid=$$
count_percentage $this_pid

progressbar


Top
 Profile  
 PostPosted: Fri Sep 22, 2017 5:35 am   

Joined: Tue Apr 12, 2016 4:11 am
Posts: 51
sorted it,


yum remove rsync


https://download.samba.org/pub/rsync/rsync-3.1.2.tar.gz


installed it by untarring it cd'd into the dir and running "./configure.sh" "make" "make install"


and now i get the result i wanted -


[[email protected] data]# rsync -a --info=progress2 call_the_midwife_7_1708/ new/
14,874,971,690 44% 27.58MB/s 0:10:48 xfr#16, to-chk=2/143)


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 9 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 22 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  
cron


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP