Hello

As I am not able/cannot manage to reply to this post
http://forums.b2evolution.net/abnormally-high-server-load-from-b2evo
I am starting a new thread;

I seem to have a similar issue since Dec 19th, when my hosting company disabled the script related to my blog due to high CPU load.
After re-enabling it on following day, the hoster requested me to do a few things to minimize indexing traffic, which I did (see emails below), however this proved insufficient and on Dec 30th, my blog script was disabled again until further notice.

Could anybody help me please. I am using a recent b2evolution version: 5.0.6

I am attaching some emails between the hoster and me to describe the situation and what happened. It is not clear to me whether the high CPU load is indeed due to robot indexing traffic

Thanks

==============================================================================
Last email from Dec 30th disabling my blog until further notice

From: [...]
Subject: Re: [Lunarpages Online HelpDesk] [EB84OE68MXK4] Script disabled due to high load
Date: 30 décembre 2013 17:17:06 UTC+1
To:

Note: Replying more than once may delay our response time, because your ticket will be placed at the bottom of our ticket queue.

Dear Son,
A technician responded to your ticket with:
Hello,

Your script /[...]/blogs/index.php was causing extremely high processing on server again:

Stats for 30 Dec 2013:
---------------------------------
CPU Usage - %32.09
MEM Usage - %3.38
Number of MySQL procs (average) - 1.60
Top Process %CPU 101.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 82.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 75.00 /usr/bin/php /[...]/blogs/index.php

Processes:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
12542 [...] 17 0 54188 25m 7148 S 13.4 0.6 0:00.70 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12614 [...] 18 0 48464 18m 7016 S 11.1 0.5 0:00.40 /usr/bin/php /[...]/blogs/htsrv/comment_post.php
12558 [...] 16 0 51912 22m 7156 R 10.0 0.6 0:00.56 /usr/bin/php /[...]/blogs/index.php
12512 [...] 17 0 47912 18m 7012 S 9.2 0.5 0:00.39 /usr/bin/php /[...]/blogs/index.php
12432 [...] 16 0 53144 23m 7104 R 8.6 0.6 0:00.63 /usr/bin/php /[...]/blogs/index.php
12504 [...] 17 0 55696 25m 7172 R 8.4 0.6 0:00.70 /usr/bin/php /[...]/blogs/index.php
12505 [...] 18 0 51180 21m 7116 R 8.4 0.5 0:00.50 /usr/bin/php /[...]/blogs/index.php
12534 [...] 21 0 51264 21m 7100 R 8.4 0.5 0:00.50 /usr/bin/php /[...]/blogs/index.php
12143 [...] 18 0 59328 29m 7196 S 8.1 0.7 0:00.98 /usr/bin/php /[...]/blogs/index.php
12412 [...] 17 0 54448 24m 7124 R 8.1 0.6 0:00.68 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12209 [...] 19 0 58476 29m 7224 S 7.8 0.7 0:00.99 /usr/bin/php /[...]/blogs/index.php
12297 [...] 17 0 53316 23m 7156 R 7.5 0.6 0:00.58 /usr/bin/php /[...]/blogs/index.php
12524 [...] 15 0 47728 18m 7016 S 7.5 0.5 0:00.39 /usr/bin/php /[...]/blogs/index.php
12577 [...] 17 0 43904 14m 6856 S 7.0 0.4 0:00.25 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12536 [...] 18 0 46112 16m 7004 R 6.7 0.4 0:00.30 /usr/bin/php /[...]/blogs/index.php
12626 [...] 17 0 43940 14m 6856 S 6.7 0.4 0:00.24 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12562 [...] 17 0 44444 13m 6656 R 6.1 0.3 0:00.22 /usr/bin/php /[...]/blogs/index.php
12548 [...] 15 0 47808 18m 7012 S 5.9 0.5 0:00.36 /usr/bin/php /[...]/blogs/index.php
12557 [...] 16 0 47712 18m 7040 S 5.9 0.5 0:00.40 /usr/bin/php /[...]/blogs/index.php
12291 [...] 18 0 55072 25m 7148 S 5.6 0.6 0:00.73 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12312 [...] 16 0 58576 28m 7192 S 5.6 0.7 0:00.97 /usr/bin/php /[...]/blogs/index.php
12357 [...] 18 0 53668 24m 7124 R 5.6 0.6 0:00.71 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12358 [...] 18 0 52012 22m 7108 R 5.6 0.6 0:00.59 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12395 [...] 19 0 53848 23m 7140 R 5.6 0.6 0:00.61 /usr/bin/php /[...]/blogs/index.php
12460 [...] 16 0 47240 18m 7020 S 5.6 0.4 0:00.38 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12461 [...] 19 0 47252 18m 7000 R 5.6 0.4 0:00.39 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12479 [...] 18 0 53676 23m 7128 R 5.6 0.6 0:00.60 /usr/bin/php /[...]/blogs/index.php
12520 [...] 22 0 53012 23m 7132 R 5.6 0.6 0:00.58 /usr/bin/php /[...]/blogs/index.php
12593 [...] 18 0 42832 13m 6616 R 5.6 0.3 0:00.20 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12625 [...] 17 0 43760 13m 6616 R 5.6 0.3 0:00.20 /usr/bin/php /[...]/blogs/htsrv/anon_async.php
12224 [...] 19 0 57992 28m 7172 R 5.3 0.7 0:00.88 /usr/bin/php /[...]/blogs/index.php
12259 [...] 15 0 59308 29m 7192 S 5.3 0.7 0:00.99 /usr/bin/php /[...]/blogs/index.php
12462 [...] 19 0 48232 18m 7064 R 5.3 0.4 0:00.39 /usr/bin/php /[...]/blogs/htsrv/comment_post.php
12468 [...] 17 0 48232 18m 7040 S 5.3 0.5 0:00.39 /usr/bin/php /[...]/blogs/index.php
12516 [...] 18 0 43332 14m 6884 S 5.3 0.4 0:00.24 /usr/bin/php /[...]/blogs/index.php
12476 [...] 17 0 47816 18m 7012 S 5.0 0.5 0:00.38 /usr/bin/php /[...]/blogs/index.php

As such, the script has been chowned to root with 000 file permissions. This means that the script is no longer accessible and cannot use up the resources on the server. You may not use this script again as [%] top processing is an unacceptable level on a shared server, affects load balancing and all other accounts on the server. Please let us know, and we can remove this script for you.

Alternatively, if this script is critical to your website, you might wish to consider our dedicated plans as it isn?t suitable to run on a shared server. Our dedicated plan details are available at:

http://www.lpwebhosting.com/hosting/enterprise/dedicated

We appreciate your attention to this matter. We have scheduled a phone call from one of our system administrators for you. Should you have any questions prior to this call, you may reply to this ticket, or call us directly at 877-586-2772 ext. 4225 Monday-Friday 7:00am-7:00pm Pacific Time. If you choose to contact us by phone, please have your account username and this ticket number for reference.

Thank you.

Kind Regards,

Additionally, you may also view the entire ticket by visiting: http://support.lunarpages.com/tickets/view/3081802
Your ticket ID is 3081802.

Lunarpages Internet Solutions Helpdesk 
https://support.lunarpages.com

=================================================================================
My last email to the hoster on Dec 30th describing my actions
From:
Subject: Re: [Lunarpages Online HelpDesk] [EB84OE68MXK4] Script disabled due to high load
Date: 30 décembre 2013 11:59:09 UTC+1
To: [...]

Hello

I have edited the .htaccess file located in my /public_html folder to add the section disallowing bots that you recommended below, can you pls check it is correctly done.

I had already banned a few ip addresses previously through the cpanel and it seems those ip addresses were already set in my .htaccess file as well.

How sure are we that the high CPU load is due to the robot indexing traffic?

Thanks

regards

Son

On 30 déc. 2013, at 06:30, [...] wrote:
Note: Replying more than once may delay our response time, because your ticket will be placed at the bottom of our ticket queue.

Dear Son Dophan,
A technician responded to your ticket with:
Hello,

Here are the latest updates on your resource usage:

Stats for 29 Dec 2013:
---------------------------------
CPU Usage - %4.30
MEM Usage - %0.49
Number of MySQL procs (average) - 0.09
Top Process %CPU 89.00 [php]
Top Process %CPU 79.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 70.00 /usr/bin/php /[...]/blogs/index.php

Please note that some bad bots simple ignore robots.txt file and keep hitting domains. Under such cases, we suggest to disallow bots through .htaccess file. You can add following code to your .htaccess file and check if it helps:

SetEnvIfNoCase User-Agent "^Yandex" search_bot
SetEnvIfNoCase User-Agent "^BingBot " search_bot
SetEnvIfNoCase User-Agent "^Yahoo" search_bot
SetEnvIfNoCase User-Agent "^igdeSpyder" search_bot
SetEnvIfNoCase User-Agent "^Robot" search_bot
SetEnvIfNoCase User-Agent "^msnbot" search_bot
SetEnvIfNoCase User-Agent "^Aport" search_bot
SetEnvIfNoCase User-Agent "^Mail" search_bot
SetEnvIfNoCase User-Agent "^bot" search_bot
SetEnvIfNoCase User-Agent "^spider" search_bot
SetEnvIfNoCase User-Agent "^php" search_bot
SetEnvIfNoCase User-Agent "^Parser" search_bot
SetEnvIfNoCase User-Agent "^Baidu" search_bot

<Limit GET POST HEAD>
Order Allow,Deny
Allow from all
Deny from env=search_bot
</Limit>

Also you can block the unwanted IPs and domain in htaccess file. I will provide more information on how you can to refuse access to certain robots or human visitors to your web site using the .htaccess file:

1. Basic .htaccess file

order allow,deny
deny from 127.0.0.1
allow from all

This will refuse all GET and POST requests made by IP address 127.0.0.1, an error message is shown instead.

2. More options

To block multiple IP addresses, list them one per line.

order allow,deny
deny from 127.0.0.1
deny from 127.0.0.2
deny from 127.0.0.3
allow from all

You can also block an entire IP block/range. Here we will not specify the last octet in the .htaccess file.

deny from 127.0.0

This will refuse access for any user with an address in the 127.0.0.0 to 127.0.0.255 range.

Instead of using numeric addresses, domain names (and subdomain names) can be used to ban users.

deny from isp_name.com

It bans users with a remote hostname ending in isp_name.com. This would stop all users connected to the internet via isp_name.com from viewing your site. Please continue making adjustments to your account as needed and we will update you with new resource usage statistics in 24 hours. If you have any questions, please don't hesitate to ask us, we will be happy to answer them. Please feel free to contact us for further help.

Thank you.
--------------------------------------------------
Additionally, you may also view the entire ticket by visiting: http://support.lunarpages.com/tickets/view/3081802
Your ticket ID is 3081802.

Lunarpages Internet Solutions Helpdesk &#8232;https://support.lunarpages.com

====================================================================================
My email to the hoster on Dec 27th describing my first actions

From:
Subject: Re: [Lunarpages Online HelpDesk] [EB84OE68MXK4] Script disabled due to high load
Date: 27 décembre 2013 14:27:09 UTC+1
To: [...]

Hello

thanks for all your helpful notes.

I have uploaded a robot.txt file, blacklisted some ips listed below, pls let me know how this affects the load going forward.

Thanks

regards

Son

On 25 déc. 2013, at 17:39, [...] wrote:
Note: Replying more than once may delay our response time, because your ticket will be placed at the bottom of our ticket queue.

Dear Son,
A technician responded to your ticket with:
Hello,

Here are the last updates on your resource usage:
=================
Stats for 25 Dec 2013:
---------------------------------
CPU Usage - %14.89 [HIGH]
MEM Usage - %1.71
Number of MySQL procs (average) - 0.21
Top Process %CPU 79.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 76.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 75.00 /usr/bin/php /[...]/blogs/index.php
====================

Please note that your site dophan.com is receiving unnecessary hits from different bots/Ips/spam, more information below:
==========================================
# Hits Files KBytes Visits Hostname
==========================================
1 15505 5.76% 15099 7.98% 532784 10.26% 393 2.10% ks4001798.ip-198-100-144.net
2 12871 4.79% 12270 6.49% 443206 8.54% 317 1.69% ks4006744.ip-198-27-83.net
3 10745 3.99% 9596 5.07% 197387 3.80% 177 0.95% crawl-66-249-73-217.googlebot.com
4 9266 3.44% 4346 2.30% 85236 1.64% 226 1.21% 14.150.159.100
5 7642 2.84% 3572 1.89% 68401 1.32% 137 0.73% 14.147.76.224
6 7541 2.80% 3523 1.86% 67463 1.30% 348 1.86% 14.147.75.80
7 5766 2.14% 5693 3.01% 207800 4.00% 117 0.63% ns7000146.ip-198-27-85.net
8 5575 2.07% 2660 1.41% 52295 1.01% 525 2.81% 14.151.177.232
9 5394 2.01% 2557 1.35% 49177 0.95% 94 0.50% 183.5.191.30
10 4878 1.81% 4454 2.35% 165462 3.19% 259 1.38% spider-199-21-99-114.yandex.com
11 4589 1.71% 2206 1.17% 42418 0.82% 120 0.64% 218.19.206.210
12 4490 1.67% 1884 1.00% 39087 0.75% 403 2.15% 113.66.187.97
13 3934 1.46% 3658 1.93% 70921 1.37% 374 2.00% 94.228.34.203
14 3878 1.44% 1845 0.98% 37947 0.73% 185 0.99% 14.151.179.48
15 3427 1.27% 2260 1.19% 38336 0.74% 9 0.05% majestic12.co.uk
16 3402 1.26% 2346 1.24% 42681 0.82% 6 0.03% node2.mobiflock.com
17 3145 1.17% 1726 0.91% 47780 0.92% 171 0.91% 14.147.82.251
18 2881 1.07% 2039 1.08% 42435 0.82% 3 0.02% root2.naefmarco.ch
19 2617 0.97% 109 0.06% 803 0.02% 21 0.11% img-spider-37-140-141-21.yandex.com
20 2409 0.90% 1553 0.82% 30502 0.59% 14 0.07% hosted-by.leaseweb.com
21 2325 0.86% 2325 1.23% 66447 1.28% 0 0.00% danzig.lunarpages.com
22 2232 0.83% 1116 0.59% 25257 0.49% 64 0.34% 113.66.187.109
23 1959 0.73% 1957 1.03% 84544 1.63% 9 0.05% 31.41.218.158
24 1942 0.72% 318 0.17% 6091 0.12% 11 0.06% 77.240.206.171
25 1738 0.65% 1581 0.84% 42013 0.81% 244 1.30% crawl-66-249-73-207.googlebot.com
26 1673 0.62% 1273 0.67% 29853 0.58% 4 0.02% 208.110.80.210
27 1492 0.55% 741 0.39% 17275 0.33% 100 0.53% 14.147.76.200
28 1482 0.55% 1072 0.57% 19914 0.38% 16 0.09% 198.74.231.14
29 1449 0.54% 776 0.41% 21523 0.41% 2 0.01% 173.208.150.250
30 1439 0.54% 1218 0.64% 39153 0.75% 3 0.02% 74.112.202.19
==========================================

Please use the robots.txt file to control access of the bots on your account. Robots are indexing your website, you may want to use a robots.txt file in order to manage this process. More information about it at: http://www.robotstxt.org/. Also you should block these IP addresses if you do not recognize them via the IP Deny Manager from your cPanel.

More than that you can use Google Webmaster Central Tool at https://www.google.com/webmasters/sitemaps/ to reduce the rate of crawling a domain or you can use the following link http://www.google.com/support/webmasters/bin/answer.py?an... to change the Google crawl rate.

Not all of the above are bad bots, because of that we advise you to check them with this online tool and block only the unnecessary:

http://centralops.net/co/

If you have any questions, please don't hesitate to ask us, we will be happy to answer them. Please feel free to contact us for further help.

Thank you.

----
Best Regards,

Additionally, you may also view the entire ticket by visiting: http://support.lunarpages.com/tickets/view/3081802
Your ticket ID is 3081802.

Lunarpages Internet Solutions Helpdesk &#8232;https://support.lunarpages.com

=====================================================================================
Hoster email on Dec 20th describing why blog script had to be disabled and re-activating it giving me a chance to fix the issue
From: [...]
Subject: Re: [Lunarpages Online HelpDesk] [EB84OE68MXK4] Script disabled due to high load
Date: 20 décembre 2013 22:34:31 UTC+1
To:

Note: Replying more than once may delay our response time, because your ticket will be placed at the bottom of our ticket queue.

Dear Son,
A technician responded to your ticket with:
Hello,

Thank you for contacting us back regarding this issue.

Please note that I re-enabled the following script and you can check further into this:

/[...]/blogs/index.php

We were forced to disable the script because the high usage on the account destabilized the server functionality. Please note that you are hosted on a shared server and you need to share resources with other customers. We need to provide our customers a fair share of the resource usage on the server where you are hosted too. Server resources often refer to the CPU,MySQL and RAM usage on a web hosting server. To put it in simple terms, this means your web hosting account is overloading the server to an extent where no other accounts on the same hosting server can function.

We appreciate your cooperation in this matter. Please understand that we are just trying to preserve the quality of our services, we are very sorry for any inconveniences.

We will continue to monitor your account usage for the next 24 hours and we will provide you new stats when available.

If you have any questions, please don't hesitate to ask us, we will be happy to answer them. Please feel free to contact us for further help.

Thank you.

----
Best Regards,

From: [...]
Subject: Re: [Lunarpages Online HelpDesk] [EB84OE68MXK4] Script disabled due to high load
Date: 19 décembre 2013 11:22:29 UTC+1
To:

Note: Replying more than once may delay our response time, because your ticket will be placed at the bottom of our ticket queue.

Dear Son Dophan,
A technician responded to your ticket with:
Hello,

Your script /[...]/blogs/index.php was causing extremely high processing on DANZIG :

[[...]]
Stats for 19 Dec 2013:
---------------------------------
CPU Usage - %20.76
MEM Usage - %1.14
Number of MySQL procs (average) - 0.42
Top Process %CPU 76.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 68.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 67.00 /usr/bin/php /[...]/blogs/htsrv/comment_post.php

Stats for 18 Dec 2013:
---------------------------------
CPU Usage - %3.56
MEM Usage - %0.52
Number of MySQL procs (average) - 0.05
Top Process %CPU 77.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 75.00 /usr/bin/php /[...]/blogs/index.php
Top Process %CPU 70.00 /usr/bin/php /[...]/blogs/xmlsrv/rdf.php

As such, the script has been chowned to root with 000 file permissions. This means that the script is no longer accessible and cannot use up the resources on the server. Please let us know what changes you will be making to correct this issue and we can re-provide ownership of the script to you.

Here are the most common causes for high usage and our recommendations:

- Your scripts are not upgraded: you may have not upgraded your scripts yet, as older versions of the scripts can have bugs in them that would cause high CPU and Memory usage.

- Your scripts are configured incorrectly: you might need to check your scripts' configuration for any tweaks of settings available that might influence the usage, as a single wrong setting might do a whole lot of harm.

- Your add-ons are misbehaving: some add-ons might cause high load because they were badly written. You might consider disabling them or finding alternatives for some add-ons. In the usage mitigation process you may want to disable all your add-ons that do not affect the very basic functionality of your website (scheduled back-ups, fancy statistics, additional functions etc). After your account?s usage was stabilized we may begin enabling them one by one and identify the problematic one.

- You might need caching: a caching plugin will help in most cases as it will serve pre-generated html pages instead of using the resource intensive PHP scripts to generate the same page over and over again for different visitors of the site (for WP ? WP super cache).

- Bots are flooding your site: your pages might have been found by spam bots that try continuously to send messages via your contact form, or add various comments to your items and such, depending on your site's configuration. An implementation of a CAPTCHA system would be useful to keep bots out of your pages, while checking for the highly accessed webpages using the tools provided by cPanel.

- Robots are indexing your website, you may want to use a robots.txt file in order to manage this process. More information about it at: http://www.robotstxt.org/ .

- You should, also, audit your crons if any and optimize / (re)set them accordingly to reduce the usage

- Your account might be exploited: using a badly written extension / plug-in or an old / not recently update CMS (Content Management System) (like Joomla, WP etc), the hackers or the script-kiddies can find their way to your files (and system?s resources) endangering the entire shared environment.

We appreciate your attention to this matter. We have scheduled a phone call from one of our system administrators for you. Should you have any questions prior to this call, you may reply to this ticket, or call us directly at 877-586-2772 ext. 4225 Monday-Friday 7:00am-7:00pm Pacific Time. If you choose to contact us by phone, please have your account username and this ticket number for reference.

Thank you.

--
Additionally, you may also view the entire ticket by visiting: http://support.lunarpages.com/tickets/view/3081802
Your ticket ID is 3081802.

Lunarpages Internet Solutions Helpdesk &#8232;https://support.lunarpages.com