Recent Topics

Antispam Deluxe

Started by on Feb 17, 2005 – Contents updated: Feb 17, 2005

Feb 17, 2005 14:04    

After being hammered by some w@^ker I came up with an automated anti-spam system.

This system is vicious, make sure you add your own site to the authorised list or you will get black-listed by it !

Before you start back up your database and the following files :-

conf/_advanced.php
b2evocore/_functions_antispam.php
b2evocore/_functions_hitlog.php
admin/_menutop.php

This has been tested on a fresh install of version 0.9.0.11

Database :-
Add new table evo_potential (replace evo with your install prefix)

create table evo_potential (ID integer(4) auto_increment,theUrl varchar(250),primary key(ID));

Add new table evo_authorised

create table evo_authorised (ID integer(4) auto_increment,theUrl varchar(250),logvisits char(1),primary key(ID));

open conf/_advanced.php and add the red bits :-

/**#@+
* database tables' names
*
* (change them if you want to have multiple b2's in a single database)
*/
$tableposts = 'evo_posts';
$tableusers = 'evo_users';
$tablesettings = 'evo_settings';
$tablecategories = 'evo_categories';
$tablecomments = 'evo_comments';
$tableblogs = 'evo_blogs';
$tablepostcats = 'evo_postcats';
$tablehitlog = 'evo_hitlog';
$tableantispam = 'evo_antispam';
$tablegroups = 'evo_groups';
$tableblogusers = 'evo_blogusers';
$tablelocales = 'evo_locales';
$tablepotential = 'evo_potential';
$tableauthorised = 'evo_authorised';

/**#@-*/

/**
* Aliases for class DB:
*/
$db_aliases = array(
'T_posts' => $tableposts ,
'T_users' => $tableusers ,
'T_settings' => $tablesettings ,
'T_categories' => $tablecategories ,
'T_comments' => $tablecomments ,
'T_blogs' => $tableblogs ,
'T_postcats' => $tablepostcats ,
'T_hitlog' => $tablehitlog ,
'T_antispam' => $tableantispam ,
'T_groups' => $tablegroups ,
'T_blogusers' => $tableblogusers ,
'T_locales' => $tablelocales ,
'T_potential' => $tablepotential ,
'T_authorised' => $tableauthorised ,

);

open b2evocore/_functions_antispam.php and modify the red bit :-

function antispam_create( $abuse_string, $aspm_source = 'local' )
{
global $DB, $tableantispam, $querycount, $cache_antispam,$Settings;

// Cut the crap if the string is empty:
$abuse_string = trim( $abuse_string );
if( empty( $abuse_string ) ) return false;

// Check if the string already is in the blacklist:
//Temporarily disable the antispamdeluxe;
$spamTemp=$Settings->get('spamEnabled');
$Settings->set('spamEnabled','');
$Settings->updatedb();
$known=
antispam_url($abuse_string);
//Now return antispamdeluxe to its previous state
$Settings->set('spamEnabled',$spamTemp);
$Settings->updatedb();
if ($known ) return false;

// Insert new string into DB:
$sql = "INSERT INTO $tableantispam( aspm_string, aspm_source )
VALUES( '".$DB->escape($abuse_string)."', '$aspm_source' )";
$DB->query( $sql );

// Insert into cache:
$cache_antispam[] = $abuse_string;

return true;
}

..............
..........
.....

/**
* Check if an URL contains abusive substrings
*
* antispam_url(-)
*/
function antispam_url( $url )
{

global $DB, $cache_antispam, $debug;
if( !isset($cache_antispam))
{ // Cache not loaded, load now:
$cache_antispam = $DB->get_col( 'SELECT aspm_string FROM T_antispam' );
}

// Check URL for abuse:
foreach( $cache_antispam as $block )
{
if( stristr($url, $block) !== false)
{
if( $debug )
debug_log( 'Spam block: '.$block );
return $block; // SPAM detected!
}
}

return spamDeluxe($url); // call SpamDeluxe protection

}

Then add this code to the end of the file (before ?> obviously :p)

Code

/* SpamDeluxe hack
* This function checks the url against it's "potential" table
* If $spamLevel = true then the url is also compared to the content of previous comments
* The url is then added to the "potential" table, for future posts
* If the url has appeared greater than $spamThreshold times the desired action is taken
* based on the value in $spamAuto
*
* $spamAuto = 0 reject the url, no other action
* $spamAuto = 1 reject the url,url is added to the ban list, the comments and hit logs are cleaned
*
* Happy anti-spamming
* Yabba
*/
 
 
/* function spamDeluxe($url) - extends function antispam_url($url) */
 
function spamDeluxe($theSuspect)
{
 
global $DB,$Settings;
 
// first lets check that we're turned on
if (!$Settings->get('spamEnabled')) return false;
 
// reduce the url to just the domain bit
if(strpos(substr($theSuspect,7,strlen($theSuspect)),'/'))
 
$theSuspect=substr($theSuspect,7,strpos(substr($theSuspect,7,strlen($theSuspect)),'/'));
 
// check if it's on the allowed list
$sql="select theUrl from T_authorised where theUrl like '%$theSuspect%'";
if ($DB->query($sql)) return false;
 
// first check that the parameters are at least set
 
$spamThreshold=$Settings->get("spamThreshold");
$spamLevel=$Settings->get("spamLevel");
$spamAuto=$Settings->get("spamAuto");
if (isset($spamThreshold) && isset($spamLevel) ){
 
$sql="SELECT theUrl FROM T_potential WHERE theUrl LIKE '%$theSuspect%'";
$theResult = $DB->get_results( $sql, ARRAY_A );
$potential = $DB->num_rows;
 
if ($potential < $spamThreshold && $spamLevel){ //ok, they haven't hit us enough, but we're paranoid so lets check the comments
$sql="SELECT comment_content FROM T_comments WHERE comment_content LIKE '%$theSuspect%'";
$theResult = $DB->get_results( $sql, ARRAY_A );
$potential = $potential + $DB->num_rows;
}
if ($potential >= $spamThreshold){
if ($spamAuto) nuke_em($theSuspect); // ok, they've hit us once to often, lets nuke 'em
 
// and now we have the pleasure of telling them to sod off
return true;
}
}
// ok, so far they're clean, now lets add them to the potential table for future reference
$sql = "INSERT INTO T_potential (theUrl) VALUES('".$DB->escape($theSuspect)."')";
$DB->query($sql);
// and go home
return false;
}
 
 
 
 
/* function nuke_em($url) - removes spammers from the records */
 
function nuke_em($theSuspect){
global $DB,$Settings;
 
// ok time to clear up their mess
// first we'll tidy up the potential table
 
$sql="DELETE FROM T_potential WHERE theUrl like '%".$theSuspect."%'";
$DB->query($sql);
 
// next we'll add them to our local table and remove any occurences from hitlogs and comments
// Insert new string into DB:
$sql = "INSERT INTO T_antispam( aspm_string, aspm_source ) VALUES( '".$DB->escape($theSuspect)."', 'local' )";
$DB->query( $sql );
 
$sql = "DELETE FROM T_hitlog WHERE referingURL LIKE '%$theSuspect'";
$DB->query($sql);
$sql = "DELETE FROM T_comments WHERE comment_author_url LIKE '%$theSuspect%' OR comment_content LIKE '%$theSuspect%'";
$DB->query($sql);
}

open b2evocore/_functions_hitlog.php and add the red bit

{ // Lookup robots
foreach ($user_agents as $user_agent)
{
if( ($user_agent[0] == 'robot') && (strstr($UserAgent, $user_agent[1])) )
{
$ignore = "robot";
debug_log( 'Hit Log: '. T_('referer ignored'). ' ('. T_('robot'). ')');
break;
}
}
}

if ($ignore=='no'){
$sql="select theUrl,logvisits from T_authorised where theUrl like '%$ref%' and logvisits='1'";
$yTemp=$DB->get_row($sql);
if (!$yTemp->logvisits) return;
}

if( $ignore == 'no' )
{
if( strlen($ref) < 13 )
{ // minimum http://az.fr/ , this will be considered direct access (although it could be https:)
$ignore = 'invalid';
debug_log( 'Hit Log: '. T_('referer ignored'). ' ('. T_('invalid'). ')' );
}
}

open admin/_menutop.php add the red bit

if( $current_User->check_perm( 'spamblacklist', 'view' ) )
{
if( $admin_tab == 'antispam' )
echo '<li class="current">';
else
echo '<li>';
echo '<a href="b2antispam.php" >', T_('Antispam'), '</a></li>';

if ($admin_tab=='deluxespam')
echo '<li class="current">';
else
echo '<li>';
echo '<a href="spamdeluxe.php">Deluxe Antispam</a></li>';

}

if( $current_User->check_perm( 'templates', 'any' ) )
{
if( $admin_tab == 'templates' )
echo '<li class="current">';
else
echo '<li>';
echo '<a href="b2template.php">', T_('Templates'), '</a></li>';
}

Create admin/spamdeluxe.php

Code

<?php
/**
* Antispam blacklist handling
*
* b2evolution - {@link http://b2evolution.net/}
* Released under GNU GPL License - {@link http://b2evolution.net/about/license.html}
* @copyright (c)2003-2004 by Francois PLANQUE - {@link http://fplanque.net/}
*
* @package admin
*/
 
/**
* Includes:
*/
require_once (dirname(__FILE__).'/_header.php');
require_once (dirname(__FILE__).'/'.$admin_dirout.'/'.$core_subdir.'/_functions_antispam.php');
 
$admin_tab = 'deluxespam';
$admin_pagetitle = 'Deluxe Anti-Spam';
 
param( 'action', 'string' );
param( 'enab', 'string' );
param( 'auto', 'string' );
param( 'threshold', 'string' );
param( 'level', 'string' );
param( 'potID','ARRAY');
param( 'ban','array');
require(dirname(__FILE__).'/_menutop.php');
require(dirname(__FILE__).'/_menutop_end.php');
 
// Check permission:
$current_User->check_perm( 'spamblacklist', 'view', true );
switch( $action )
{
  case 'update':
    // Check permission:
    $current_User->check_perm( 'spamblacklist', 'edit', true );
 
    $Settings->set('spamEnabled',$enab);
    $Settings->set('spamAuto',$auto);
    $Settings->set('spamLevel',$level);
    $Settings->set('spamThreshold',$threshold);
    $Settings->updatedb();
    $logged='';
    $banned='';
    $authorised='';
    $delete='';
    foreach ($_POST as $Field=>$Value){
       switch (substr($Field,0,3))
      {
        case 'all';
          $authorised.=" or theUrl like'%".substr($Field,3)."%'";
          break;
        case 'ban';
          $banned.=" or theUrl like '%".substr($Field,3)."%'";
          break;
        case 'log';
          $logged.=" and ID!='".substr($Field,3)."'";
          break;
        case 'del';
          $delete.=" or ID='".substr($Field,3)."'";
          break;
      }
    }
 
    if ($logged) $logged=" where ".substr($logged,5);
    $logged="update T_authorised set logvisits='0'".$logged;
    $DB->query("update T_authorised set logvisits='1'");
    $DB->query($logged);
 
  if ($authorised) {
    $DB->query('insert into T_authorised (theUrl) select theUrl from T_potential where '.substr($authorised,4));
    $DB->query('delete from T_potential where '.substr($authorised,4));
  }
 
 
  if ($delete) $DB->query('delete from T_authorised where '.substr($delete,4));
  if ($banned) {
    $theResults=$DB->get_results('select distinct theUrl from T_potential where '.substr($banned,4),'ARRAY_A');
    foreach ($theResults as $aPain){
      nuke_em($aPain['theUrl']);
         b2evonet_report_abuse($aPain['theUrl']);
    }
  }
}
 
$sql="select distinct theUrl from T_potential order by ID desc";
 
$theResults=$DB->get_results($sql,ARRAY_A);
echo '<form action="spamdeluxe.php" method="post">';
echo '<table class="grouped" style="float:left;margin-left:5%"><tr><th
colspan="2">Settings</th></tr><tr><td>Enabled</td><td style="text-align:center"><input type="checkbox" name="enab"';
if ($Settings->get('spamEnabled')) echo ' checked="checked"';
echo ' /></td></tr><tr class="odd"><tr class="odd"><td>Auto clean</td><td style="text-align:center"><input type="checkbox" name="auto"';
if ($Settings->get('spamAuto')) echo ' checked="checked"';
echo ' /></td></tr><tr><td>Check comments</td><td style="text-align:center"><input type="checkbox" name="level"';
if ($Settings->get('spamLevel')) echo ' checked="checked"';
echo ' /></td></tr><tr class="odd"><td>Threshold</td><td><input type="text" size="1" name="threshold"
value="'.$Settings->get('spamThreshold').'" /></td></tr></table><br />';
echo '<input type="hidden" name="action" value="update" /><br style="clear:both" /><br /><table style="width:90%;margin:0 5%"><tr><td style="width:50%;vertical-align:top">';
echo '<table class="grouped" style="width:95%;margin-right:5%"><tr><th>Potential  spammers</th><th>Ban</th><th>Auth</th></tr>';
if($DB->num_rows==0){
  echo '<tr><td colspan="3">No potential spammers</td></tr>';
} else {
$myCount=0;
  foreach ($theResults as $theRow){
    echo '<tr';
    if ($myCount%2==1) echo ' class="odd"';
    echo '><td><a href="'.$theRow['theUrl'].'">'.$theRow['theUrl'].'</a></td><td
style="text-align:center"><input type="checkbox" name="ban'.$theRow['theUrl'].'" /></td><td style="text-align:center"><input type="checkbox" name="all'.$theRow['theUrl'].'" /></td></tr>';
    $myCount++;
  }
}
echo '</table></td><td style="width:50%;vertical-align:top">';
$sql="select ID,theUrl,logvisits from T_authorised order by ID asc";
$theResults=$DB->get_results($sql,ARRAY_A);
echo '<table class="grouped" style="width:95%;margin-right:5%"><tr><th>Authorised Urls</th><th>Log visits</th><th>Delete</th></tr>';
if($DB->num_rows==0){
echo "<tr><td>No Authorised urls</td></tr>";
} else {
$myCount=0;
  foreach ($theResults as $theRow){
    echo '<tr';
    if ($myCount%2==1) echo ' class="odd"';
    echo '><td><a href="'.$theRow['theUrl'].'">'.$theRow['theUrl'].'</a></td><td
style="text-align:center"><input type="checkbox"';
    if ($theRow['logvisits']==1) echo ' checked="checked"';
    echo ' name="log'.$theRow['ID'].'" /></td><td style="text-align:center"><input type="checkbox" name="del'.$theRow['ID'].'"></td></tr>';
    $myCount++;
  }
}
echo '</table></td></tr></table><br /><input type="submit" value="Update records" style="margin-left:5%" /></form>';
require( dirname(__FILE__).'/_footer.php' );
?>

Save all files

Log into the admin area, you will find a new tab named Deluxe Antispam....click it ;)
You should now be presented with 3 tables, 1 for settings, 1 for potential spammers and 1 for authorised urls.

[u]The settings[/u]
Enabled - tick this to turn it on
Auto Clean - Tick this to automatically ban and clean potential spammers who cross the threshold
Check comments - Tick this to also check previous posts comments with the same url
Threshold - This is the number of times to allow a url before taking any action

[u]Potential spammers[/u]
This is a list of all potential spammers that have hit your blogs (ie every url)
you have two options, you can either tick ban or auth
They're pretty self explanatory.

[u]Authorised[/u]
This is a list of safe urls (make sure your own url is on this list !), you can also choose whether to count hits from these urls or just ignore them completely.

The easiest way to get your url on the safe list is, enable the system, set a threshold of about 5, click update. Exit to blogs, go back into admin > Deluxe Antispam, your url will now be on the potential list, tick auth, click update

At the moment, about the only idiosyncracy that the code has is, if you authorise a url from the potential table, it creates a new record in the authorised table for every time it appears in the potential table..... until I sort this, you'll just have to delete the extra ones manually :-S

*rips off Edb's standard disclaimer* If I understood php I'd be twisted, there's probably a dozen ways this can be improved, but my knowledge of php is only matched by my knowledge of quantum physics......ie NIL

*edit* To include "Report to b2evo central on ban"

Feb 18, 2005 04:23

I didn't read the whole code so forgive me if it should be obvious, but how does this relate to the current "antispam central" thing? When you ban a suspect are they still reported to central? Also what about when you get the update from central?

Feb 18, 2005 14:39

Hi EdB,

Ok, this isn't a replacement for the antispam it's kind of an enhancement, it's meant to stop the tossers who spam your refer list & comments before you can get to the red button.

It works on the principle that, if a url has appeared more than n times, then the chances are this is a spammer we haven't caught yet, so it blocks all further attempts.

If you have it in auto mode, it also cleans up the hit list & comments & potential table (similar to the way that adding them to your blacklist does) and adds them to your local blacklist without you having to do anything else.

At the moment it doesn't report the abuse when you ban them, it just bans them localy (it's only a matter of adding one line of code though).

The main reason is to save b2evo from being flooded by innocent victims if people have it in automatic mode (I have auto mode turned off).

As the code comes after the normal blacklist checking it shouldn't interfere with the normal operation at all.

I would strongly advise that anyone who trys it out :-
a) BACK UP YOUR DATABASE
b) DONT run it in auto mode until you've been using it a while and have added all the safe url's to your authorised list, otherwise you'll find chunks of comments going missing...... This is such a strong possibility that I'm thinking of amending the admin page to list all the url's that it considers a potential spammer, and ones that it will blacklist with your current threshold setting.

If you'd like to have a play with it, then I'd be happy to give you access to a fresh install and access to the database so you can play with your own sites data with no risk, just let me know.

?

Feb 18, 2005 15:32

Thanks. I was just curious because you didn't mention how it interacts with the current system, so I wondered '...how deep does this go...'. It looks like a super-cool hack and, now that I know it does not shut down or exclude the current group-grope system, I'll probably be adding it to my blog blog after this weekend.

Personally I'll probably go with non-auto because if you get into a conversation in comments with someone they hit you quite a bit over a short period of time. You're right about not overloading b2evo central with automatically banned keywords, but I think adding reporting if ASPM_DELUXE is not in auto mode is a good idea. "The human made the decision, so let the code do all appropriate follow up work" seems right to me.

I still get a kick out of being the first one to turn in a spammer ;)

Feb 18, 2005 16:51

Hi EdB,
Gotta agree with you about the pleasure using the big red button gives you :D

To get the code to report to b2evo central in manual mode you just need to add the following line to admin\spamdeluxe.php (I've added the line in the original post).

if ($banned) {
$theResults=$DB->get_results('select distinct theUrl from T_potential where '.substr($banned,4),'ARRAY_A');
foreach ($theResults as $aPain){
nuke_em($aPain['theUrl']);
b2evonet_report_abuse($aPain['theUrl']);
}

I will reiterate my warning to anyone who tries this BACKUP YOUR DATABASE, this system is vicious and will happily kill friends as well as enemys.

Personally I'll probably go with non-auto because if you get into a conversation in comments with someone they hit you quite a bit over a short period of time.

You're spot on there, it may even be a good idea to change "auto mode" to "email notification". ie/ you get an email warning you of a potential spammer, and you have to ban them manuaully instead. The url would still be barred from any further action, but if it's a friend their previous comments etc will still be alive.

?

Mar 03, 2005 18:03

I've just installed Antispam Deluxe ( someone spam my comment
during the night every day ) but something strange happened .

After installation Antispam Deluxe it's ok but when I ' ve tried
to post a new messagge on my blog contain a link return a message
that advice me the URL ( I've tried with many different URL )
can't be accetted ... so I must write it without HTML TAG

( escuse me for bad english , I' m italian )

Ex . I want to write <a href="http://www.pippo.com">Pippo</a>
, return message that this URL is inacceptable , so I've to write

http://www.pippo.it without HTML formatting

Sorry for english I only want to know If it's possibile to dipend
from something I've changed installing ANtispam Deluxe ...

Thanks a lot and compliments for the hacks :D

Barbara

Mar 03, 2005 18:26

Hi Barbara,
This hack shouldn't stop you using links the way you do, just to check I tried posting <a href="http://www.pippo.com">Pippo</a> on my test blog with no problems.

The only reasons I can think of for it stopping you is if you have your threshold set to a very low number (ie 1) or if you have the same url in your previous comments/referrers/posts as this is what it checks against.

If you click on the anti-spam deluxe tab in your admin area you should see http://www.pippo.com (and all the other urls that you've tried) on the potential list. If you tick auth and then click update records it should move the url to the authorised list and allow you to post as normal.

If that doesn't cure your problem, let me know.

?

Mar 04, 2005 11:30

Thanks a lot ????? :D

I'll try it in the week end ...

Barbara

Jul 10, 2005 10:05

I have installed Anti-Spam deluxe on my [url=http://www.hhollick.com/blog]blog[/url] and it certainly stops all unwanted referrers.

In fact it stops ALL referrers, legitimate or otherwise. I do not even have it "enabled" (no check in the "enabled" box) and I have received no further referrals in the last week.

Before Anti-Spam Deluxe, I would get a few legitimate referrals a day from the posts and links I have around the internet. Now these are gone.

Does this sound right? Is that how Deluxe is supposed to work?

Many thanks,
Heather

Jul 11, 2005 08:10

Hi Heather,

I'm not sure what's causing your problem, if you don't have this hack enabled then it has no affect on your blog.

¥

Aug 10, 2005 00:34

I installed Deluxe Antispam, then I tried to post to my [url=http://blog.pesola.us]blog[/url]. I included four links. When I tried to preview I was told an invalid URL was included.

I found that one link [url=http://www.mobiuslife.com]mobiuslife.com[/url] was not showing up in the 'potential' list. The others showed up and I was able to add them.

Is this the expected behavior? I tried to disable 'deluxe antispam' and submit the post and experienced the same behavior.

Any suggestions?

Thanks,
Troy

Aug 10, 2005 09:18

Hi Troyp,

I just tried posting the same link on my own blog, and found out that it's caused by life.com being in the central blacklist @ record 2239

If you remove it from the blacklist you'll be able to post the link and it will show up on the potential list.

¥

Sep 29, 2005 06:18

I dont really know what you all are talking about when you mention Spam, I know that it isn't the normal E-mail spam I get all the time. I am new to this and would like to run a safe site. But in your instructions on how to do this hack your talking about stuff I have no idea how to do, I dont know hot to creat a table or do stuff like that. I am really trying to learn this stuff as quick as I can.

Sep 29, 2005 20:49

Hi Diablo,
The spam this fights is called "referer spam" or "comment spam", it's kind of similar to email spam in the fact that it's some tosser trying to promote a site that just trys to sell you cheap rolexes or printer inks etc.

To creat tables you need to have access to phpmyadmin or some similar databse interface or you need to write a page that does it for you.

However, this hack was written for version ....urm, well an older one than the latest one, and I haven't actually tried it on a later install to see if it works (I still run the ancient version). So, if you do try to install it backup your database and all the files involved first as I can't guarantee it'll work.

But what the hell, all of life's just russian roulette right?;)

¥

Dec 03, 2005 10:00

Hi,

I'm happy to use spamDeluxe with 0.9.1 "Dawn". Because your code is against older version, so I need some modifications to port.

In 0.9.1, every urls, emails and comments is checked with antispam_check() function. So antispam_deluxe() need to recognize these.

http://blogs.da-cha.jp/media/antispamdeluxe.diff.gz is a patch for 0.9.1.

The points of modification are in _function_antispam.php:spamDeluxe().

PHP

if (! preg_match_all('|https?://[a-zA-Z0-9@:%_~#?&=.,/;-]+|'$theSuspect$matches))
                return false;
 
foreach ($matches[0] as $url)  {
            .....

This extract URLs from comment and check every URLs.

Thanks.

Hiroshi

Dec 03, 2005 10:39

Hi momokuri,

Nice, I still haven't installed Dawn, thanks for converting it :)

¥

Mar 06, 2006 23:06

Can someone explain how to use this "patch" file?

I have 0.9.1 and badly need this (I'm already using a rotation htsrv directory name, DNSBL's, .htaccess traps for the comment post having to have the correct referrer etc) and they're still getting through... :-(

I've tried doing it by hand, but it doesn't seem to be working...

When a comment post includes a URL, I get this error -

Fatal error: Call to a member function on a non-object in /home/tvotwd2/public_html/blog/htsrv.BYr1ES/comment_post.php on line 82

However I've not edited comment_post.php!

Any help gratefully recieved!

Mar 11, 2006 10:11

I'm afraid I still haven't installed 9.1 so I really can't help you much with this.

The patch file shows you all the lines that you need to add/remove/amend to convert it to 9.1.

¥

Mar 11, 2006 19:56

As I said, I tried doing the patch manaully, but couldn't get it working - so I've backed it out for now.


Form is loading...

CMS software – This forum is powered by b2evolution CMS, a complete engine for your website.