Antispam Bandwidth

[isaac]

Ok, so this has been really a popular topic on these forums lately, and I've been investigating a few possibilities.

.htaccess rules based on the blacklist are simply unreasonable (with my host, at least) since you have several thousands of antispam strings, and it would be crazy to make Apache chug through all of those each time there's a referral.

Checking the referring page for a link to your site doesn't always work, since you might have links to your site from mail messages or other hidden things, and also that means that you're delaying loading your page until some other page loads into the get_file_contents call, thus doubling your load time, and triggering a back-button frenzy.

RewriteMap doesn't work, since I can't access my httpd.conf.

Stick this in your conf/hacks.php file, and you're good to go. Very low bandwidth, prevents referrer spiders from getting past the front gate, and generally friendly.

[personman]

I've applied the hack. Thanks. I laughed out loud when I read it.

[ralphy]

I haven't done any speed tests, but I wonder how Apache would be slower using an .htaccess file than a PHP script accessing database's tables...

Doesn't using b2evolution's blacklist into an .htaccess file make the b2evolution tests useless?

(Nice message. )

[BenFranske]

[ralphy]

Here are some preliminary page loading speed test results:

[BenFranske]

What that tells me is that a lot more time is spent redering the php than reading a full htaccess blacklist so it's probably not a concern as long as it's not too processor intensive. Also, I'd be interesting in finding out how my SetEnvIf... htaccess method is faster or slower than the rewrite method.

[ralphy]

[roadies]

if only we could block at the switch

[personman]

How often do you all report spammers to their ISPs and/or web hosts?

[isaac]

I added another check in there so that it doesn't keep you from clicking from your own antispam page (quite annoying!)

[ralphy]

Two very interesting articles about referrer spam:
Proposal on referrer spam: Background and blacklists
Referrer and Comment Spam: A Primer

The MT-Blacklist plugin spammer blacklist:
MT-Blacklist Master Copy

A PHP script intended to update a .htaccess file with referrer spammers after Apache logs analysis:
Referrer Spam Fucker 3000

Using DNSBL appears to be a nice idea:
MT-DSBL
New anti-spam trick using DSBL

The DNSBL can be used directly from Apache, as explained there:
referrer-b-gone

Finally, a page to test the referrer spam rules (it allows setting a referrer before loading a given page):
WANNABrowser

Also worth to read that discussion about spam fight (especially blocking HTTP_USER_AGENT's):
A Close to perfect .htaccess ban list
Don't miss andreasfriedrich .htaccess and httpd.conf single and multiple RewriteCond benchmarks:
A Close to perfect .htaccess ban list (Page #8)

[BenFranske]

I've read several of those articles because referer spam has become a really big problem for me, even so far as to close my site for the time being. Most of the commentators go on about how using htaccess is an unwinnable fight (maybe so if you're doing it by hand, hence my automatic "Refer This!" script, yet they really have no better solution than what I'm doing. That is to block based on a blacklist (in my case the b2 antispam blacklist generates the htaccess file). My qwest is to find the least processor intensive way of dealing with the problem because that's whay i had to pull my site, it was using too many processor cycles when the spambots accessed the page which caused a database lookup. As far as I can tell the best we can do now is to generate a htaccess blacklist based on the database. Of course I will make sure I'm not publishing referer information either. If anyone else has suggestions I'm all ears.

[BenFranske]

[isaac]

I'm not sure, Ben. In fact, it seems to me that you should be right - .htaccess should be faster.

However, when I put more than about 500 RewriteCond directives in my .htaccess, my site suddenly gets slow as molasses.

There's a lot of reasons why that might be, and I'm not enough of an Apache expert to say why. I know that relational database lookups are really not that slow, depending on a few factors, because of all sorts of indexing and whatnot that goes on in the backend. Then again, I've got more experience with that side of things on MSSQL than on MySQL, so I can't really come up with figures or very sound explanations. On the other hand, Apache must actually read, process, and cache all the RewriteConds until it gets to a RewriteRule. Perhaps packing the rules, and having several Cond/Rule bunches could optimize this somewhat. I'm not sure.

Without any doubt, more research is definitely needed. And, like birth control, I suspect that multiple methods of prevention is the way to go.

[ralphy]

I used andreasfriedrich's .htaccess vs httpd.conf HTTP_USER_AGENT Blocking Benchmarks figures to display them more readable (click on the image to enlarge):



It appears "Multiple RewriteCond" .htaccess (several conditions packed on the same line using regular expression's OR) appear pretty fine, even if slower than "Simple RewriteCond" httpd.conf (one condition per line).