Register
It is currently Tue Sep 16, 2014 3:28 pm

wget: Return An Error to "if [ ]"


All times are UTC - 6 hours


Post new topic Reply to topic  [ 5 posts ] 
Author Message
 PostPosted: Sat Aug 15, 2009 12:37 pm   

Joined: Sat Aug 15, 2009 11:07 am
Posts: 6
Location: South Africa
I have written a script to download and web links that are pasted in a simple text file.
It works through the list, downloading each one to a directory (created by date) and deleting the line of the downloaded link.
My aim is to have this run every night via 'cron',
Here it is:

Code:
#!/bin/bash
#
# ----------
# Successively download each link in a file,
# and remove each link after downloading.

# --------------------
# VARIABLES
# --------------------
#
# set the download file
links=~/wget.links
i=1
# set the download directory
directory=$(date "+%a.%d.%b.%y")
#
# --------------------
# FUNCTIONS
# --------------------
#
# set the function to download each line of the download file
function download_links()                     
   {                                 
           echo "---------- downloading "$each" ----------"      
      echo ""                              
      wget -c --tries=5 --waitretry=1 --directory-prefix $directory $each
           echo ""                        
   }                              
#
#   
# set the function to delete the links just downloaded
function delete_lines()
   {
      echo "removing "$each" from file"
      sed -i ''$i'd' $links
      echo ""
   }      
#
# --------------------
# MAIN BODY
# --------------------
#
# for each line fron file...
for each in $(cat $links)
   do
      # call the function to download each line
      download_links
      # if wget doesn't return true then don't delete the link from the file
# ------------------------
#      if [  ]
#         then
#            delete_lines
#         else
#            i=`expr $i + 1`
#      fi
# -------------------------
      # call the function to remove downloaded file each time
      delete_lines
      # if there are no links left to download...
      if [ 0 -eq $(cat $links | wc -l) ]
         # then print the following
         then
            echo "all files finished downloading"
         # otherwise, we still have the following to download
         else
            echo "still have the following to download:"
            cat $links
      fi
done


It works well, except if the download doesn't work then I would like it not to delete the link and print an error message, skipping to the next one and so on.

Is there any way I can get wget to return 'false' to the "if [ ]" condition?
Or is there a better way to do it?

Cheers


Last edited by adamcbrewer on Sun Aug 16, 2009 2:55 pm, edited 1 time in total.

Top
 Profile  
 PostPosted: Sun Aug 16, 2009 2:51 pm   

Joined: Sat Aug 15, 2009 11:07 am
Posts: 6
Location: South Africa
Found it ( I hope!).

The problem isn't in wget at all. I just overlooked the "$?" return variable from the last command.
This way any failed downloads return a 1 (false), instead of a 0 (true). :))

e.g.
Code:
wget http://www.something.com
if [ $? -eq 0 ]
        then ...
        else ...
fi


However, I'm not sure if this works for a partially downloaded file? :-/


Top
 Profile  
 PostPosted: Mon Aug 17, 2009 12:40 am   

Joined: Mon Nov 17, 2008 7:25 am
Posts: 221
I would guess that wget returns an error if it for some reason exits in the middle of a download.

If the file exists and you don't use any parameters wget automagically adds .# as an extension (starting with file.txt.1, file.txt.2 and so on).

Best regards
Fredrik Eriksson


Top
 Profile  
 PostPosted: Mon Aug 17, 2009 12:57 am   

Joined: Sat Aug 15, 2009 11:07 am
Posts: 6
Location: South Africa
Thanks for the reply!

I suppose that's true, will have to test an actual link and then cancel it halfway through.

And as for the naming of files that wget does, maybe can find someway of using it to check it the files has already downloaded? Maybe in the same loop it does the original download, and then remove the corresponding line in the text file?
Or tell wget not to create ".1" and return another result (if it wants to overwrite to an if [ ] conditional.

I'm sure the solution is just a line or two of code! [-o<


Top
 Profile  
 PostPosted: Mon Aug 17, 2009 1:54 am   

Joined: Sat Aug 15, 2009 11:07 am
Posts: 6
Location: South Africa
Finally, I have it!

I made a the wget.links file, with dummy links, and it works right the way through. This was actually my first script so I am pretty chuffed with myself. =D> lol

Anyway, you could set it to retry as many times as you want. I suppose it could also be condensed as well, so feel free! One things though: I'm not very experienced with wget, so there might be better options to put in there.

Code:
#!/bin/bash
#
# ----------
# Successively download each link in a file,
# and remove each link after downloading.
# ----------
# Created by Adam, 14.08.09
# ----------
#
# --------------------
# VARIABLES
# --------------------
#
# set the download file
links=~/wget.links
i=1
# set the download directory
directory=$(date "+%a.%d.%b.%y")
#
# --------------------
# FUNCTIONS
# --------------------
#
# set the function to download each line of the download file         
function download_links()                           
   {                                 
      echo "---------- downloading "$each" ----------"            
      echo ""                              
      wget -c --tries=10 --waitretry=1 --directory-prefix $directory $each      
      result=$?
      echo ""                                 
   }                                 
#   
# set the function to delete the links just downloaded               
function delete_lines()
   {
      echo "removing "$each" from file"
      sed -i ''$i'd' $links
      echo ""
   }
function connection_test()
   {
      ifconfig | grep Bcast
      result=$?
   }      
#
# --------------------
# MAIN BODY
# --------------------
#
# check if there is any connection present
connection_test
if [ $result -eq 1 ] # if there's no connection
   then
      echo "No connection to Internet...not downloading tonight!"
      exit
fi
# for each line from file...
for each in $(cat $links)
   do
      # call the function to download each line
      download_links
      # if wget doesn't return true then don't delete the link from the file
      if [ $result -ne 0 ] # if download didn't work
         then                           
            # there was a problem with the first try...
            echo "There was a problem with the download"
            echo "Trying download again..."
            download_links
            if [ $result -ne 0 ] # if download didn't work
               then
                  # there was a problem the second time...
                  echo "Download still not working"
                  echo "Skipping link #$i"
                  i=`expr $i + 1`
               else
                  echo "Download successful this time"
                  delete_lines
            fi
         else
            # download worked - removing links from file
            delete_lines
      fi
      echo "Items still on the list:"
      cat $links
      echo ""
done
echo "--------------------"
echo ""
# results dialogue
if [ 0 -eq `cat $links | wc -l` ]
   then
      echo "All files were downloaded successfully."
   else
      echo "Script finished, however, these files were not downloaded:"
      cat $links
fi


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: Google [Bot] and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP