I've noticed a user agent called web downloader appearing in my logs recently. Is this some user initiated thing like scrap book in firefox or is it something malicious? Should I use htaccess to block it? I did a few searched but didn't really come up with anything given how generic the wording is.

web downloader is an offline browser, only reason to want to block it is it can suck up your bandwidth. i believe i have it blocked

Isn't that part of the new Google internet speed up tool that pre-downloads the sites before you click on the links?

I suspected something like that, it was just that when I searched for most of the results that came up were to do with it being in peoples htaccess file as being blocked, so I feared the worse.

What you said may be true Joe, it seemed strange that it was the Web Downloader user agent with the same ip address but it came as a link from an external site that links to mine. If it is through Google its a bit of a worry as the damn thing spidered 16 pages all through one link! I would imagine that would be even worse if the user using it was on the homepage which has even more links.

My web downloader allows me to spoof the useragent string...
It even ignores robots.txt and allows me to download perl and php scripts as well as password files and other things people try to hide/protect...


BTW: It sounds as if someone was ripping/reaping someone elses site that links to your site and the ripper/reaper was set to crawl too deep... I can set mine to crawl as deep as I want and I have to be careful that I don't start downloading the entire internet...

let me know when you download google's entire index

Don't laugh - I nearly did it this morning along with epartner.onetravel.com when I ripped/reaped Jesse's site to test my rollover/highlight script on his pages...


.....Willy