Reduce memory usage on users import (120k +)

Reduce memory usage on users import (120k +)

Tuesday 01 March 2011 12:56:43 am - 8 replies

Modified on Tuesday 01 March 2011 12:59:07 am by Remi B

Author Message

Marko Žmak

Tuesday 01 March 2011 3:02:23 pm

I believe you should use:

$object->resetDataMap();
eZContentObject::clearCache($object->attribute('id'));

after import of every user. Maybe also using some unset() for used variables.

If you want some more details, I had a question about this here:

http://share.ez.no/forums/developer/resetdatamap-and-ezcontentobject-clearcache/

--
Nothing is impossible. Not if you can imagine it!

Hubert Farnsworth

Marko Žmak

Tuesday 01 March 2011 3:06:02 pm

Also, for faster import, you could disable some module operations, look at this discussion for more info:

  • http://share.ez.no/forums/developer/publish-objects-from-script-without-cache-clear

--
Nothing is impossible. Not if you can imagine it!

Hubert Farnsworth

Remi B

Wednesday 02 March 2011 12:30:17 am

Thanks for your second post with the disableModuleOperation, the memory peak does not increase at each iteration.

I was already do the reset stuff, memory usage is about 30Mo between each publication.
But the publication take 500Mo :)

Btw, i've just relaunched my import with your trick, i'll post here to debrief about the memory usage.

Marko Žmak

Wednesday 02 March 2011 2:56:26 am

From your case I can conclude that executing module operations operations upon publishing increases the memory usage.

That would suggest that some of the module operation has a memory leakage...

Anyone from the eZ crew wishes to comment on this one?

--
Nothing is impossible. Not if you can imagine it!

Hubert Farnsworth

Remi B

Wednesday 02 March 2011 4:59:40 am

ATM, the script memory usage peak is 547 Mo
Since this morning, 2980 objects have been imported

I've got 31375 objects in the destination node.

Script is still runing...

Jean-Yves Zinsou

Wednesday 02 March 2011 5:20:05 am

Hi Remi B,

I have used those lines inside an import loop, with significant results :

eZContentObject::clearCache();
unset( $GLOBALS['eZContentObjectContentObjectCache'] );
unset( $GLOBALS['eZContentObjectDataMapCache'] );
unset( $GLOBALS['eZContentObjectVersionCache'] );
unset( $GLOBALS['eZContentClassAttributeCache'] );
//unset( $GLOBALS["eZUserObject_".$node->ContentObjectID] );

Give it a try !

Jey

Do Androids Dream of Electric Sheep?
I dream of eZpubliSheep....
------------------------------------------------------------------------
http://www.alma.fr

Remi B

Wednesday 02 March 2011 6:09:37 am

Hi Jey,

Thanks for your post, but, as i said, I already make this stuff :-)

The memory peak coming during content publish action.

Bertrand Dunogier

Wednesday 02 March 2011 6:39:53 am

One thing I can say for sure is that using the Asynchronous Publishing that comes with eZ Publish 4.5 will significantly reduce the memory usage that comes from the publishing process (make it close from 0), and will fasten the process a LOT.

Bertrand Dunogier
eZ Systems Engineering, Lyon
http://twitter.com/bdunogier
http://gplus.to/BertrandDunogier

You must be logged in to post messages in this topic!

Powered by eZ Publish™ CMS Open Source Web Content Management. Copyright © 1999-2014 eZ Systems AS (except where otherwise noted). All rights reserved.