I can`t reporduce your error. I tried both 3.6.0rc2 SVN: 11801 and 3.6.0rc1 SVN: 11754 installed in VH mode and Contet Object Export works fine with many sub-trees.
Install of 3.6.0rc1 went OK and the only thing missing in finetune was mbstring and postgresql, but I'm still getting the same result everytime.
On one attempt I got a memory exceeded error and that was with 60MB limit and 60 secs execution.
I'm running on a Firebright VDS and I've yet to be able to process a package that is anything larger than about 1 or 2 subtrees totalling about 20/30 nodes.
Just tried to create a package with a 30/40 images from the media library (about 50k each) and I got this error
Fatal error: Allowed memory size of 62914560 bytes exhausted (tried to allocate 35 bytes) in /home/webadmin/wildaboutbritain.co.uk/html/cms/lib/ezi18n/classes/ezcodepage.php on line 82 Fatal error: eZ publish did not finish its request
The execution of eZ publish was abruptly ended, the debug output is present below.
60MBs for memeory limit and it's still having problems.
I would try telling one of the FireBright crew about the problem, as we're more than capable of fixing this for you (it looks like a Virtuozzo problem, and has nothing what-so-ever to do with eZ Publish).
In a Virtualized environment, you have to account for resource allocations that are kernel level (as each VDS runs it's own Linux kernel), which is most likely the problem here, and not memory (there is 6 GB on the machine you're on).
Do put in a helpdesk ticket, and we'll get it squared away for you (ticket at firebright dot com).