eZ is DOSing server, need help

eZ is DOSing server, need help

Wednesday 12 January 2005 12:48:56 pm - 19 replies

Author Message

Björn Dieding@xrow.de

Wednesday 12 January 2005 1:45:05 pm

Does your server swap at all? Try o reduce the swap used ...

The optimum is that you do not swap at all to reduce load from the disk IO

other steps

- install a accelerator e.g phpa
- clear cache

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Marek W

Wednesday 12 January 2005 2:16:28 pm

Nope, server still has 300MB of free memory, cache is cleared, PHP accelerator is enabled.

Michal Dyro

Wednesday 12 January 2005 3:10:38 pm

Hi!
Must say that Marek is a very modest person.

We need help with the website of Polish Mozilla Community - http://mozillapl.org While testing - it worked quite well.. but it only needed a official blast-off, couple of hours and traffic at ~375 000 page views/ day to make our server go offline :(

We're one of the Biggest Open Source communities in Poland. We tried everything we could but there's not even a SINGLE person within us that knows eZ moze than Marek. So if he can't help all our HOPE is in YOU.

PS
My english is not as good as I supposed :( Sorry

Bruce Morrison

Wednesday 12 January 2005 3:45:35 pm

Debugging is always hard when you don't have access to a system. So how about some more information?

php version ?
apache version ?
ez publish version ?

The systems seems to have worked in testing, so what has changed since going live?

> CPU usage went 100%
So what process is taking up all the CPU?

> Apache "Max Servers" limit was also acquired.

This could be an apache issue. It could be caused by the system getting many hits and grinding to a halt. The hits continue coming but the apache processes are unable to finish serving the requests due to high server load.

Can you access the site at all? How long does it take to become unusable?

Cheers
Bruce

My Blog: http://www.stuffandcontent.com/
Follow me on twitter: http://twitter.com/brucemorrison
Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish

Marek W

Wednesday 12 January 2005 4:18:57 pm

More information:
PHP 4.3.10
MySQL 4.1
Apache 1.3
eZ 3.5

Since testing only eZ directory has changed, eZ was moved to root directory.

Adminstrator told that apache processes are eating CPU, eZ couldn't send a page and all requests were hold with "sending reply" status... When DOSed I can't access the site, apache must be restarted...

Björn Dieding@xrow.de

Wednesday 12 January 2005 4:33:16 pm

You told us you moved the server. Does that mean you actually had a non eZ website running there before?

If this is the case try to redirect all requests that belong to a non eZ url to an error page ( e.g everything *.html )

In those cases eZ will give a module not found error those error have a high impact on the system they ea a lot of cpu time.

Besides that you may have reached a critcal limit in request per server. It might be just the case that his maschine can't handle it with eZ running on it. I have never been on a SINGLE server that can serve more then 15 eZ pages per second.

In this case block all networking traffic and do a benchmark with no traffic and compare it with your usual traffic.

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Björn Dieding@xrow.de

Wednesday 12 January 2005 4:39:11 pm

An other idea...

have you checked the mysql? When mysql has problems it might be that mysql requests are set to a hold state... during that time apache just hold it till it timeouts...

Usually in this senario apach and mysql will eat up all the ram and swap... i guess this is not the case

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Björn Dieding@xrow.de

Wednesday 12 January 2005 4:41:04 pm

Is debugging disabled? It should... You want as less IO as possible

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Gabriel Ambuehl

Thursday 13 January 2005 12:08:49 am

If you don't already use it, get turckmm cache. It greatly speeds up large PHP Apps like eZPublish by caching the bytecode.

I just saw you use PHPaccellerator. That should do the job, too.

Visit http://triligon.org

Łukasz Serwatka

Thursday 13 January 2005 6:17:46 am

have you checked the mysql? W
hen mysql has problems it might be that mysql requests are set to a hold state... 
during that time apache just hold it till it timeouts...

This maybe a reason. We had problem with server DOSing by eZ publish when mysql crashes. CPU usage went to 100%. Accessing main page takes very very long time ...

Personal website -> http://serwatka.net
Blog (about eZ Publish) -> http://serwatka.net/blog

Marek W

Thursday 13 January 2005 7:50:39 am

1. Yes, main page (website) was using Postnuke CMS
2. I agree that some non existing URL could be redirected to eZ, but this doesn’t explain that eZ is also DOSing when moved to other directory and tested with ab utility
3. Site is not running on dedicated server, but I assume that if it could handle Postnuke with 1%-2% load it would handle eZ
4. Site is using eAccelerator (new mmcache)
5. I don’t think that this is MySQL issue (will ask administrator), only one request is made when page is cached

Some more info on my settings:

site.ini override:

[Session]
SessionNameHandler=custom

[DatabaseSettings]
DatabaseImplementation=ezmysql
Charset=utf-8
Socket=disabled
SQLOutput=disabled

[ContentSettings]
ViewCaching=enabled
PreViewCache=disabled
PreCacheSiteaccessArray[]=mozillapl

[TemplateSettings]
TemplateCache=enabled
TemplateCompile=enabled
TemplateCompression=enabled
Debug=disabled

[DebugSettings]
DebugOutput=disabled
DebugRedirection=disabled

site.ini for siteaccess:

[SiteSettings]
ErrorHandler=defaultpage

[SiteAccessSettings]
RequireUserLogin=false
ShowHiddenNodes=false

[SiteAccessRules]
Rules[]=Access;disable
Rules[]=Module;user/register

[RoleSettings]
EnableCaching=true
PolicyOmitList[]
PolicyOmitList[]=content/pdf
PolicyOmitList[]=content/read
PolicyOmitList[]=form
PolicyOmitList[]=rss/feed

Note: Some settings were removed

Björn Dieding@xrow.de

Thursday 13 January 2005 8:13:22 am

I think you should let a professional look at your setup and test it. I like said before it is very hard to resolve from the outside. Root access is required too.

Please check also the disk IO, is this also running at 100% or is it just the CPU?

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Björn Dieding@xrow.de

Thursday 13 January 2005 8:17:05 am

Waht happens if you disable those 2 options?

[ContentSettings]
ViewCaching=enabled

[TemplateSettings]
TemplateCache=enabled
TemplateCompile=disabled
TemplateCompression=disabled

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Marek W

Thursday 13 January 2005 9:02:29 am

Made some more test, seems that this is caching issue with disk access, with following settings:

[ContentSettings]
ViewCaching=enabled
PreViewCache=disabled

[TemplateSettings]
TemplateCache=disabled
TemplateCompile=enabled
TemplateCompression=disabled
Debug=disabled

I can run ab with -c 50, of course reply times are not so good (longest reply 11 sec) but it isn't DOSing the server... Do you have any ideas on improving eZ caching system? Any info about internal aspects of TemplateCache?

Bruce Morrison

Thursday 13 January 2005 11:29:04 pm

Are you able to actually get a page from the server?

Have you tried pre compiling the templates?
http://ez.no/community/forum/install_configuration/zz_systems_or_cpu_100

Have you tried disabling template compiling?

Cheers
Bruce

My Blog: http://www.stuffandcontent.com/
Follow me on twitter: http://twitter.com/brucemorrison
Consolidated eZ Publish Feed : http://friendfeed.com/rooms/ez-publish

Björn Dieding@xrow.de

Friday 14 January 2005 12:25:26 am

There is not much more you can do to improve the caching.

An option is to make static copy of the whole site. So that you don't hit mod_php4.

What disk is actually in the server? I think providing a better and faster disk will help you a lot.

Looking for a new job? http://www.xrow.com/xrow-GmbH/Jobs
Looking for hosting? http://hostingezpublish.com
-----------------------------------------------------------------------------
GMT +01:00 Hannover, Germany
Web: http://www.xrow.com/

Russ Crandall

Friday 14 January 2005 5:26:39 am

I had a similar issue with 3.5 and an upgraded php 4.3.10 with apache2. (we run apache1 on the production server, never tried to see if the problem existed there)

Having run out of things to try, I decided to go the 'old school programmer route'.. randomly commenting out bits of code. After disabling phpa, and the load problems went away. I suspect it had something to do with an older version of phpa not liking the newer php version and apache2.

Marek W

Friday 14 January 2005 9:12:50 am

Server is running only IDE disks, even not in RAID... But managed to make some static cache system... Mayby we will improve that and send it to contributions :) Thanks for help!

Arran Price

Sunday 16 January 2005 12:40:10 pm

From a unix sysadmin point of view it would be interesting to see the output of the unix command "top" which should should cpu state and which processes specfically are using the processor/memory as well as state of those processes.
In addition, I would check the error/message logs for apache and mysql respectively and ensure thats its not an issue being reported by either one.
Another thing to check is any email to users running crontab entries for ezpublish (eg runcronjobs.php etc).

ie just determine that it is actually ezpublish causing the problem by asking questions like, if no users log in to ezpublish does the load increase anyway?

hope thats of some help
Arran

You must be logged in to post messages in this topic!

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.