Register
It is currently Thu Dec 18, 2014 5:43 am

Help with WGET


All times are UTC - 6 hours


Post new topic Reply to topic  [ 5 posts ] 
Author Message
 PostPosted: Sat Aug 21, 2010 4:59 am   

Joined: Sat Aug 21, 2010 3:36 am
Posts: 14
I am running a WGET comand manually and I would like to automate it.

this is the command:
Code:
wget -c --load-cookies .cookies/thecookie -P ~/downloads/ URL


I would like to automate this using a script.

Note:
I need to loop and load this command with every file I want to download:

Code:
wget -c --load-cookies .cookies/thecookie


I am aware that I can add -i to get the list of files (url) I want to download from a text file.
The reason I want to loop / load the cookie every time I want to download a file is that website requires the cookie else download will fail.
Another thing if download fails it should proceed to the next file.

Thanks in advance...


Top
 Profile  
 PostPosted: Sat Aug 21, 2010 5:21 am   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
assuming you already have an url.txt file, you could use a simple while loop:
Code:
while read URL
do wget -c --load-cookies .cookies/thecookie -P ~/downloads/ $URL
done < url.txt


Top
 Profile  
 PostPosted: Sat Aug 21, 2010 5:32 am   

Joined: Sat Aug 21, 2010 3:36 am
Posts: 14
Watael wrote:
assuming you already have an url.txt file, you could use a simple while loop:
Code:
while read URL
do wget -c --load-cookies .cookies/thecookie -P ~/downloads/ $URL
done < url.txt


Wow thanks a lot man I think I will be very active in this forum.

Follow up question though. There are times when download fails wherein it hangs and stays on a specific url or file.
How do we skip that and proceed to the next file from the URL list (url.txt)

Thanks again...


Top
 Profile  
 PostPosted: Sat Aug 21, 2010 6:31 am   

Joined: Mon Mar 02, 2009 3:03 am
Posts: 579
set --timeout to a reasonable amount of seconds, and --tries to a number of attempts


Top
 Profile  
 PostPosted: Sat Aug 21, 2010 12:06 pm   

Joined: Sat Aug 21, 2010 3:36 am
Posts: 14
Watael wrote:
set --timeout to a reasonable amount of seconds, and --tries to a number of attempts


Dude thanks a million it work perfectly I added:

Code:
-t 1 -T 30


and it did just what I needed and wanted :)

Cheers,
r0cks0ul


Top
 Profile  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 5 posts ] 

All times are UTC - 6 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Jump to:  


BashScripts | Promote Your Page Too
Powered by phpBB © 2011 phpBB Group
© 2003 - 2011 USA LINUX USERS GROUP