BashScripts.org
http://bashscripts.org/forum/

website map
http://bashscripts.org/forum/viewtopic.php?f=8&t=1670
Page 1 of 1

Author:  p13 [ Thu Nov 22, 2012 4:06 am ]
Post subject:  website map

I have to do script in bash who do in file website map.

For example:
I use this script:

Code:
./website.sh www.adress.com


as te result I see in new file all links in pages in this domain.
for example:

Code:
www.adress.com/index.htm

www.adress.com/page1.htm
.
.
.
etc.


I don't now how do it.

Author:  cfajohnson [ Wed Jan 02, 2013 1:04 pm ]
Post subject:  Re: website map

Use lynx:

Code:
lynx -dump "$url" |
   sed  -e '1,/^References$/d' -e 's/^ *[0-9]*. //'

Author:  crouse [ Sun May 05, 2013 9:39 pm ]
Post subject:  Re: website map

wget -r --spider -b -o /var/tmp/wget.log http://target.site.com/

-r - Recursive download
--spider - Check downloaded content for new links, then discard
-b - Run in the background
-o /var/tmp/wget.log - Target log file
http://target.site.com/ - Replace with the site you want to check

Page 1 of 1 All times are UTC - 6 hours
© 2000, 2002, 2005, 2007 phpBB Group • http://www.phpbb.com