[Hpr] Slow downloads again
dosman at packetsniffers.org
Wed Sep 6 08:10:26 PDT 2017
While Google tends to behave better in regards to user-agent shenanigans, I’ve even had to drop-route them before when they started raping some of my sites. It seems to be better these days, but search spiders tend to be oblivious to how much traffic they can drive, especially when they malfunction and decide to download a large file repeatedly for days on end.
> On Sep 6, 2017, at 10:14 AM, Joshua Knapp <jknapp85 at gmail.com> wrote:
> They mean that some times it didn't identify as yandex, sometimes it would fake as a browser. I saw the same thing in the logs after I started trying to block the bot by user agent.
> On Sep 6, 2017 7:09 AM, "Claes Wallin (韋嘉誠)" <hackerpublicradio at clacke.user.lysator.liu.se <mailto:hackerpublicradio at clacke.user.lysator.liu.se>> wrote:
> On Sep 6, 2017 4:09 PM, "Ken Fallon" <ken at fallon.ie <mailto:ken at fallon.ie>> wrote:
> http://www.webhostingtalk.com/showthread.php?t=924727 <http://www.webhostingtalk.com/showthread.php?t=924727>
> That thread is so funny. People really don't bother to read the full tread before they post, do they?
> - Using robots.txt didn't help, so here's how I detect and redirect that bot.
> - Dude use robots.txt
> - IP block!
> - I redirected instead.
> - Dude you really should be using robots txt.
> - Here's how I IP block!
> Some people seem to believe that text file has magical powers.
> I didn't get this part:
> > One more thing: it identifies as client, not as bot !! Google and Yahoo identified as bots !
> What are they talking about? Is there anything aside from the user agent that indicates what the client is?
> Hpr mailing list
> Hpr at hackerpublicradio.org <mailto:Hpr at hackerpublicradio.org>
> http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org <http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org>
> Hpr mailing list
> Hpr at hackerpublicradio.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Hpr