<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">While Google tends to behave better in regards to user-agent shenanigans, I’ve even had to drop-route them before when they started raping some of my sites. It seems to be better these days, but search spiders tend to be oblivious to how much traffic they can drive, especially when they malfunction and decide to download a large file repeatedly for days on end.<div class=""><br class=""></div><div class="">-dosman</div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Sep 6, 2017, at 10:14 AM, Joshua Knapp <<a href="mailto:jknapp85@gmail.com" class="">jknapp85@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="auto" class="">They mean that some times it didn't identify as yandex, sometimes it would fake as a browser.  I saw the same thing in the logs after I started trying to block the bot by user agent. </div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Sep 6, 2017 7:09 AM, "Claes Wallin (韋嘉誠)" <<a href="mailto:hackerpublicradio@clacke.user.lysator.liu.se" class="">hackerpublicradio@clacke.user.lysator.liu.se</a>> wrote:<br type="attribution" class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto" class=""><div class="gmail_extra" dir="auto"><div class="gmail_quote"><div class="gmail_quote" dir="auto">On Sep 6, 2017 4:09 PM, "Ken Fallon" <<a href="mailto:ken@fallon.ie" target="_blank" class="">ken@fallon.ie</a>> wrote:</div><div class="gmail_quote" dir="auto"><a href="http://www.webhostingtalk.com/showthread.php?t=924727" target="_blank" class="">http://www.webhostingtalk.com/<wbr class="">showthread.php?t=924727</a></div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">That thread is so funny. People really don't bother to read the full tread before they post, do they?</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto"> - Using robots.txt didn't help, so here's how I detect and redirect that bot.</div><div class="gmail_quote" dir="auto"> - Dude use robots.txt</div><div class="gmail_quote" dir="auto"> - IP block!</div><div class="gmail_quote" dir="auto"> - I redirected instead.</div><div class="gmail_quote" dir="auto"> - Dude you really should be using robots txt.</div><div class="gmail_quote" dir="auto"> - Here's how I IP block!</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">:-D</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">Some people seem to believe that text file has magical powers.</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">I didn't get this part:</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">> One more thing: it identifies as client, not as bot !! Google and Yahoo identified as bots !</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">What are they talking about? Is there anything aside from the user agent that indicates what the client is?</div><div class="gmail_quote" dir="auto"><br class=""></div><div class="gmail_quote" dir="auto">-- </div><div class="gmail_quote" dir="auto">   /c</div></div></div></div>
<br class="">______________________________<wbr class="">_________________<br class="">
Hpr mailing list<br class="">
<a href="mailto:Hpr@hackerpublicradio.org" class="">Hpr@hackerpublicradio.org</a><br class="">
<a href="http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org" rel="noreferrer" target="_blank" class="">http://hackerpublicradio.org/<wbr class="">mailman/listinfo/hpr_<wbr class="">hackerpublicradio.org</a><br class="">
<br class=""></blockquote></div></div>
_______________________________________________<br class="">Hpr mailing list<br class=""><a href="mailto:Hpr@hackerpublicradio.org" class="">Hpr@hackerpublicradio.org</a><br class="">http://hackerpublicradio.org/mailman/listinfo/hpr_hackerpublicradio.org<br class=""></div></blockquote></div><br class=""></div></body></html>