- b2evolution CMS Support Forums
- b2evolution Support
- General Support
- How to keep crawlers/bots from killing the SQL server
1 jaymzh Nov 28, 2005 01:52
I'm having a lot of problems with one of my blogs that I host on shared hosting.
People crawling (mostly the main page, it seems) are generating enough SQL queries to cause the SQL server serious problems - and then my account gets suspended.
Now, as a few points of reference:
- This particular blog installation has < 10 valid users
- I nightly (via cron) parse the logs and block the top requestors, some of which are often upwards of 70,000 hits.
- I have another b2evo installation in the same shared-hosting account that has a lot more users, but less crawlers/bots causing issues - in fact it's never caused an issue at all
- I update the b2evo Spam list every few days on both b2evo installs
- The troublesome blog was using "BlueSky" which had a ton of unneeded junk in the sidebar that was doing lots of SQL queries, so I chucked a lot of stuff from skins/bluesky/_main.php - I included a diff below.
I'm using version 0.9.0.11.
Since b2evo uses SQL on every page, some sort of rate-limiting-per-IP would be really great. But any other hints on how to stop this would be great.
Thanks,
- Phil
PS, here's my _main.php diff:
--- _main.php.bk Wed Jul 6 12:13:50 2005
+++ _main.php Wed Oct 12 11:19:28 2005
@@ -141,10 +141,6 @@
<h4><?php $Blog->disp( 'name', 'htmlbody' ) ?> :</h4>
<?php $Blog->disp('longdesc', 'htmlbody') ?>
<ul>
- <li><a href="<?php $Blog->disp( 'staticurl', 'raw' ) ?>"><strong><?php echo T_('Recently') ?></strong></a><?php echo T_('(cached)') ?></li>
- <li><a href="<?php $Blog->disp( 'dynurl', 'raw' ) ?>"><strong><?php echo T_('Recently') ?></strong></a><?php echo T_('(no cache)') ?></li>
-</ul>
-<ul>
<li><a href="<?php $Blog->disp( 'lastcommentsurl', 'raw' ) ?>"><strong><?php echo T_('Last comments') ?></strong></a></li>
<li><a href="<?php $Blog->disp( 'blogstatsurl', 'raw' ) ?>"><strong>Stats</strong></a></li>
</ul>
@@ -172,53 +168,11 @@
</form>
-<?php form_formstart( $Blog->dget( 'blogurl', 'raw' ), '', '', 'get', 'switcher' ) ?>
- <fieldset><label for=set><h4><?php echo T_('Choose a skin') ?> :</h4></label>
- <select id=set name="skin">
- <?php // ---------------------------------- START OF SKIN LIST ----------------------------------
- for( skin_list_start(); skin_list_next(); )
- {
- echo '<option value="';
- skin_list_iteminfo( 'name' );
- echo '"';
- if( skin_list_iteminfo( 'name',false ) == $skin ) echo ' selected="selected" ';
- echo '>';
- skin_list_iteminfo( 'name' );
- echo "</option>\n";
- } // --------------------------------- END OF SKIN LIST --------------------------------- ?>
- </select>
- <input type="submit" value="Ok" />
- </fieldset>
-</form>
-
-
<h4><?php echo T_('Archives') ?> :</h4>
<ul>
<?php require( dirname(__FILE__).'/_archives.php' ); ?>
</ul>
-<?php if ($disp != 'stats') { ?>
-
- <h4><?php echo T_('Recent Referers') ?></h4>
- <?php refererList(5, 'global', 0, 0, 'no', '', ($blog > 1) ? $blog : ''); ?>
- <ul>
- <?php if( count( $res_stats ) ) foreach( $res_stats as $row_stats ) { ?>
- <li><a href="<?php stats_referer() ?>"><?php stats_basedomain() ?></a></li>
- <?php } // End stat loop ?>
- <li><a href="<?php $Blog->disp( 'blogstatsurl', 'raw' ) ?>"><?php echo T_('more...') ?></a></li>
- </ul>
- <br />
- <h4><?php echo T_('Top Referers') ?></h4>
- <?php refererList(5, 'global', 0, 0, 'no', 'baseDomain', ($blog > 1) ? $blog : ''); ?>
- <ul>
- <?php if( count( $res_stats ) ) foreach( $res_stats as $row_stats ) { ?>
- <li><a href="<?php stats_referer() ?>"><?php stats_basedomain() ?></a></li>
- <?php } // End stat loop ?>
- <li><a href="<?php $Blog->disp( 'blogstatsurl', 'raw' ) ?>"><?php echo T_('more...') ?></a></li>
- </ul>
-
-<?php } ?>
-
<h4><?php echo T_('Misc') ?></h4>
<ul>
<?php
Upgrade to 0.9.1 (dawn). It will reduce the load on your server.