BashScripts.org
http://bashscripts.org/forum/

Help with WGET
http://bashscripts.org/forum/viewtopic.php?f=8&t=1171
Page 1 of 1

Author:  r0cks0ul [ Sat Aug 21, 2010 4:59 am ]
Post subject:  Help with WGET

I am running a WGET comand manually and I would like to automate it.

this is the command:
Code:
wget -c --load-cookies .cookies/thecookie -P ~/downloads/ URL


I would like to automate this using a script.

Note:
I need to loop and load this command with every file I want to download:

Code:
wget -c --load-cookies .cookies/thecookie


I am aware that I can add -i to get the list of files (url) I want to download from a text file.
The reason I want to loop / load the cookie every time I want to download a file is that website requires the cookie else download will fail.
Another thing if download fails it should proceed to the next file.

Thanks in advance...

Author:  Watael [ Sat Aug 21, 2010 5:21 am ]
Post subject:  Re: Help with WGET

assuming you already have an url.txt file, you could use a simple while loop:
Code:
while read URL
do wget -c --load-cookies .cookies/thecookie -P ~/downloads/ $URL
done < url.txt

Author:  r0cks0ul [ Sat Aug 21, 2010 5:32 am ]
Post subject:  Re: Help with WGET

Watael wrote:
assuming you already have an url.txt file, you could use a simple while loop:
Code:
while read URL
do wget -c --load-cookies .cookies/thecookie -P ~/downloads/ $URL
done < url.txt


Wow thanks a lot man I think I will be very active in this forum.

Follow up question though. There are times when download fails wherein it hangs and stays on a specific url or file.
How do we skip that and proceed to the next file from the URL list (url.txt)

Thanks again...

Author:  Watael [ Sat Aug 21, 2010 6:31 am ]
Post subject:  Re: Help with WGET

set --timeout to a reasonable amount of seconds, and --tries to a number of attempts

Author:  r0cks0ul [ Sat Aug 21, 2010 12:06 pm ]
Post subject:  Re: Help with WGET

Watael wrote:
set --timeout to a reasonable amount of seconds, and --tries to a number of attempts


Dude thanks a million it work perfectly I added:

Code:
-t 1 -T 30


and it did just what I needed and wanted :)

Cheers,
r0cks0ul

Page 1 of 1 All times are UTC - 6 hours
© 2000, 2002, 2005, 2007 phpBB Group • http://www.phpbb.com