Comments on: Getting very large files from flaky hosts using wget http://www.ghidinelli.com/2010/11/26/getting-very-large-files-from-flaky-hosts-using-wget Thu, 01 Jun 2017 18:51:00 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Brian http://www.ghidinelli.com/2010/11/26/getting-very-large-files-from-flaky-hosts-using-wget/comment-page-1#comment-61252 Brian Mon, 06 Dec 2010 23:16:05 +0000 http://www.ghidinelli.com/?p=1186#comment-61252 Here's one more big of (unrelated) wget kung-fu; this one spiders a website and turns it into a directory full of html, graphics and css: <code>wget --random-wait -p -r -e robots=off -U mozilla http://foo.com/</code> Here’s one more big of (unrelated) wget kung-fu; this one spiders a website and turns it into a directory full of html, graphics and css:

wget --random-wait -p -r -e robots=off -U mozilla http://foo.com/

]]> By: Brian http://www.ghidinelli.com/2010/11/26/getting-very-large-files-from-flaky-hosts-using-wget/comment-page-1#comment-61200 Brian Tue, 30 Nov 2010 17:24:27 +0000 http://www.ghidinelli.com/?p=1186#comment-61200 It took some 6 days but I got all 13gb! I need to run by the colo so I can copy it to a USB drive rather than try and download it but the file looks complete. It took some 6 days but I got all 13gb! I need to run by the colo so I can copy it to a USB drive rather than try and download it but the file looks complete.

]]>