Recent Topics

The Backwards Approach To Referrer Spam Control

Started by on Mar 04, 2006 – Contents updated: Mar 04, 2006

Mar 04, 2006 15:42    

I made some minor hacks to my b2e install so that one of my blog skins displays the status of my custom referrer spam solution and the number of entries in the stock b2e spam table (currently over 4000). You can see the output at the bottom of the left hand nav column on http://www.synysys.com/YTP

Currently my firewall is blocking 3532 unique IPs from which the bot/user was doing antisocial things on my server. This includes url fishing attacks on my httpd, brute force attacks on sshd and an overwhelming number of referrer spam attempts. My solution returns a custom 403 error page which includes a link to a tool for unblocking a real user who stumbles into the honey pot by "accident". In the event of a bot, their apparent IP goes into my firewall table and doesn't get flushed until after a number of days.

The magnitude of these numbers is somewhat mind numbing. It occurs to me that we are all approaching referrer spam from the wrong end. What we really need to do is to work with the good folks at the major search engines (Google, Yahoo and friends). It is the current practices of the search engines and their current ranking systems which makes referrer spam a profitable activity, thus bringing down a plague on the entire blogging community. Until the powers that control the search tools make it unprofitable to pound on our sites with these junk hits, we're going to be chasing our tails through endless iterations and variations on the same theme - all the while consuming our valuable time, network and server resources.

I would encourage the dev team here at b2e to work with the dev teams from the other major blog packages and make a combined representation to Google and friends. We aren't going to solve referrer spam on our own and I'd far rather see the collective wasted effort which has gone into this problem to date be spent on inventing the next big thing, adding core features to the blogging packages or even (gasp!) spend the time blogging.

In the mean time, I'd be more than happy to discuss the tools and the methods we're using on the Synysys site with anyone who needs a leg up protecting their site from these fools.

--
Dr. Bob
Synysys Technologies

Mar 04, 2006 16:45

Nice linkback to b2evolution you got there!

Dr. Bob wrote:

It occurs to me that we are all approaching referrer spam from the wrong end. What we really need to do is to work with the good folks at the major search engines (Google, Yahoo and friends). It is the current practices of the search engines and their current ranking systems which makes referrer spam a profitable activity, thus bringing down a plague on the entire blogging community.

Couple of things come to mind. First off, b2evolution doesn't display referer traffic anymore, so they're getting nothing from faking referer traffic. Second, try searching these forums for "google nofollow".

Dr. Bob wrote:

In the mean time, I'd be more than happy to discuss the tools and the methods we're using on the Synysys site with anyone who needs a leg up protecting their site from these fools.

So what are you waiting for? Zip it up and give it away. You know - like how the dev team zipped up b2evolution and gave it to you. I doubt I'll use it since anything that blocks IPs doesn't block ONLY offensive traffic, but there are people who feel it's an acceptable way to go.

Really do like the linkback though. I'm sure the dev team loves to see how much credit they get for the product they work on then give away.

Mar 04, 2006 18:52

EdB wrote:

Couple of things come to mind. First off, b2evolution doesn't display referer traffic anymore, so they're getting nothing from faking referer traffic. Second, try searching these forums for "google nofollow".

Unfortunately the problem isn't restricted to b2e users, but it's to the credit of the b2e devs that they responded to this problem as effectively as they did. That said, there are still many sites running older versions of b2e or other blogging software which isn't patched. The spammers are using a shotgun approach, even hitting sites which aren't running any blogging software at all. The problem has spilled over into general sites that may, through some obscure link, still display site traffic information. They'll pound away on thousands/millions of sites until they find the ones with vulnerabilities. We have to look for ways to stop the pounding. My suggestion to the dev team here stands. Patching away an endless stream of attacks is not as effective as trying to address the root of the problem by working with other developers. The Antispam database currently has over 4000 entries. At what point to we begin to bend under the weight of this "protection"?

So what are you waiting for? Zip it up and give it away. You know - like how the dev team zipped up b2evolution and gave it to you. I doubt I'll use it since anything that blocks IPs doesn't block ONLY offensive traffic, but there are people who feel it's an acceptable way to go.

Real time IP blocks are an extremely effective way to protect your entire site and are not limited to one package such as b2e or even just to your httpd. By fending off the user at the first sign of antisocial behaviour, you have immediately shut down what is probably a stream of probes/attacks. You can cycle the IP out of the blocked pool on whatever schedule seems appropriate for your site. I would agree that adding permanent long lists of IP addresses to your block is a tireless, fruitless and largely pointless exercise. By doing it dynamically, you can be responsive, blocking the attack as at happens and if you do it properly, don't end up blocking legitimate users from your site. I have found that trying to address WHO the attacker is, is not as effective as blocking them instantly when they do something antisocial on your site. The trick is in defining "antisocial" and linking the tools to respond appropriately.

Unfortunately there is no "it" to zip up and just hand off. What we did on Synysys was deploy numerous tools, customized them where appropriate and wrote additional hacks/tools to tie them together. It's 50% freely available tools, 25% customization and 25% best practices.

My offer was to assist and suggest techniques for folks who were still having specific problems. I'm not selling anything here, just offering to share the fruits of 15 years of sysadmin experience with fellow b2e users and trying to give back a bit to the blogging community. Obviously you found that offensive given the sarcastic tone of your post.

Really do like the linkback though. I'm sure the dev team loves to see how much credit they get for the product they work on then give away.

The site you are so quick to criticize is targeted at children. We were specifically asked to remove all external links on the main site pages that did not point to specific scientific educational resources with K12 content. We "negotiated" leaving the login page intact and did not remove the references to b2e or the good work of Francois. Stripping the links wasn't my choice. I ran b2e for over a year as part of the pilot project for their site. During the pilot, all link backs were left in place. When the site went live a few weeks ago, the customer asked for logos and links to be removed. I checked the license agreement and the terms of service agreement and the only requirement for linkback was for the use of the b2evolution.net ping back service, so the links were taken off. I should probably go in and remove the b2e pingback from the core b2e code so the blog owners aren't trying to ping that service on new posts. I suspect the teachers haven't read the terms of service document.

The nature of contributing to open source is such that some times you see credit for your work, some times you don't. What is more important is that we assist each other openly, without feeling the need to make sarcastic comments over someone's good intentions.

--
Dr. Bob
Synysys Technologies


Form is loading...

Build your own site! – This forum is powered by b2evolution CMS, a complete engine for your website.