it's kinds of uncomfortable for me if i can't download all contents for article im reading because they're chapter-base or section-based. by doing that, the authors tend to forget to create one single html/pdf/or_whatever file. *sigh.don't get me wrong, i'm here wasn't to complain about the authors nor the articles. It's really nice if we only need to save/download a single file rather then a bounch of files with a weird name like "chap01_sec1_titleX_beginner.html".huhh.:D. anyways, back to the post title,yeahh downloader. lucky for us to had downloaders which are can download a whole directory to us. speaking of which CLI downloaders are fit for me, i picked wget and aget. why aget?, simply because wget didn't support simultaneous downloading. here is wget command to download all files inside a directory. We can control to lets wget recursively crawling to only a directory. when you're enable wget to recursively downloading files without controling how far it can crawl, your're in big trouble because wget will get all files inside the website which is we didn't want todo that.
shell>wget -r -k -l 1 http://www.example.com/articles/
-r for recursive
-k convert all links to local files
-l level of depth wget can crawl.
shell >wget -r -k -l 1 http://www.freeos.com/guides/lsst/
---------------------------output------------------------------------
00:37:59 (1.23 KB/s) - `www.freeos.com/guides/lsst/aboutauth.html' saved [3267/3267]
--00:37:59-- http://www.freeos.com/guides/lsst/index.html
=> `www.freeos.com/guides/lsst/index.html'
Connecting to www.freeos.com|66.98.242.53|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12,438 (12K) [text/html]
--------------------------end-----------------------------------------
btw, if you want to download bash-scripting tutorial, link i provided is good resource for that. have a nice reading days ahead.
opps. im using aget when i want a simultaneous download. you can use aget with this example
shell> aget -n 10 http://mirror.oscc.org.my/fedora/core/4/i386/iso/FC4-i386-disc1.iso
-n how many thread you want to open
-happy downloading with open source tools.:p
3 comments:
wget roxx especially the -m (mirror) flag.
if you're a developer and considering some cli help for downloading , i'd suggest CURL. it's more like a framework than it is as a cli downloader, while it can be used as it.
it would be nice to see axel/aget/prozilla/lftp ported for windows(no cygwin SH!7 - like cURL) command line also :(
http://www.mynitor.com/2010/09/30/command-line-alternatives-to-wget-and-so-much-better/
Post a Comment