Take a Hint, AllResearch – Go Away
It was a couple of months ago when I had to ban a bot for the first time. At the time, I noticed one IP address (My Most Frequent Visitor) outshining all others in my web stats. A little research showed MMFV to be an outfit called AllResearch.com, who appears to specialize in hoovering down other sites in order to provide such services as trademark tracking, webclipping, and “law enforcement” services.
As bad as that sounds, the only reason I took even took note was the bandwidth comsumption. Every 60 minutes, they hit my RSS feed, and then pulled down every item listed in the feed. What a horrible perversion of intent. At the time, I banned the IP address, thusly (in .htaccess):
RewriteCond %{REMOTE_ADDR} “^38.144.36.16$” RewriteRule .* – [F,L]
I then watched, ammused, for several days as the 403 errors stacked up, once an hour, from their IP address. Worst. Bot. Ever.
Well, I got around to checking my stats again today, and what do you know? My Most Frequent Visitor just can’t take a hint. He’s back, using IP address 38.144.36.19. And I didn’t find him due to vigilance; he’s just stupid. The exact same usage pattern. While expanding my ban, I even corrected the unescaped dots in my original version:
RewriteCond %{REMOTE_ADDR} “^38.144.36.” RewriteRule .* – [F,L]
I’m not the only one seeing this. If you run a site, take a minute and check your stats or your logs for addresses beginning with 38.144.36.. If you see abuse like I’ve seen, take a minute and ban them. Maybe they will eventually take a hint.
Both comments and pings are currently closed.
Using mod_rewrite will keep them away but you’re still using bandwidth to serve them a 403 error.
Why not use the firewall? Assuming you’re using a Linux server, try something like:
If you’re not hosting on your own machine then that’s not possible. 2589 403 hits are nothing compared to 2359 hits at about 25KB each. That’s what I’ve experienced since 12-3-2004 when AllResearch.com changed their IP address from 38.144.36.16 to 38.144.36.19. I experienced 723 hits on the *19 IP address before I banned them. They should at least obey robots.txt but they don’t even check…
Our little friends allresearch.com
Well Got a response…
For the last few weeks I have enormous amount of hits (20,000 per day) from askjeeves from ip 65.214.38.241 Also, Gigablast is huge traffic eater. Eventually, have to block them.
Being evil
BEING EVIL:
OK, I have a solution of sorts to referrer spam – I bounce it back to them. I have almost eliminated it, except for one from h*hproftclub who refuses to go away. To be really EVIL, I’ve redirected that spammers output alone to the signup page at http://www.webclipping.com/signup.html. Probably not entirely ethical but then we’re dealing with two unethical orgs – why not play them off against each other? (Yeah I know, two wrongs…)
It does mean more crap in my log file (the way I have it set up) but it’s fun. Now, my log file sees an attempt to access a page on my site, which is redirected. However, because of the way it is redirected the bot still thinks it is on MY site – so I also get to see the cgi request. For example:
207.248.240.119 – – [04/Mar/2005:06:19:37 -0600] “GET /cgi-bin/signup.cgi?cname=web+poker&caddress=+%3Ch1%3EYou+can+also+check+out+the+sites+about%3CA …
So now webclippings are being auto-spammed by a spammer! We hate them both – now they can hate each other. And I think I am effectively hidden from this loop, thanks to how these redirects work. It just looks like h**hprofit is spamming them direct!
Ha ha ha! Death to the spammer…