You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by jo...@verona.se on 2007/06/12 13:03:30 UTC
cocoon 2.2 application memory consumption problems
Hello list,
I've made a small cocoon 2.2 application which browses images
contained i zip files.
I started out with the cocoon project archetype.
Basically I have a flowscript which figures out the content of a zip
file using this:
var imgarc=new Packages.java.util.zip.ZipFile(basepath);
gets the Enumeration of the files, converts it to a list, sorts it,
and then does sendpage to a view page for each file.
The image file in the zip is viewed with a cocoon jar protocol reader:
<map:match pattern="flib/**.cbz!**">
<map:read src="jar:file:///flib/{1}.cbz!/{2}"/>
</map:match>
This application consumes about 1.5Gb virtual, 170 Mb resident,
and I run out of heapspace after a while.
The application is very simple. How can I debug this memory problem?
I also use the Directory Generator to browse the files, and an xslt
with some xpath 2.0 so I've configured saxxon.
--
Joakim Verona
Re: cocoon 2.2 application memory consumption problems
Posted by Vadim Gritsenko <va...@reverycodes.com>.
joakim@verona.se wrote:
> Vadim Gritsenko <va...@reverycodes.com> writes:
>
>> joakim@verona.se wrote:
>>> I did some more studying with jhat, and it appears that the jpegs
>>> inside the zips are being held in memory by the cocoon cache, in the
>>> end leading to the out of memory condition.
>> ...
>>> However, I find it odd the cache doesnt release the objects when
>>> memory grows scarce.
>> It does, when configured properly. Check your store janitor settings.
>
> Sincce I use only default settings, maybe these needs review?
Hell, yeah!
Vadim
Re: cocoon 2.2 application memory consumption problems
Posted by jo...@verona.se.
Vadim Gritsenko <va...@reverycodes.com> writes:
> joakim@verona.se wrote:
>> I did some more studying with jhat, and it appears that the jpegs
>> inside the zips are being held in memory by the cocoon cache, in the
>> end leading to the out of memory condition.
> ...
>> However, I find it odd the cache doesnt release the objects when
>> memory grows scarce.
>
> It does, when configured properly. Check your store janitor settings.
Sincce I use only default settings, maybe these needs review?
>
> Vadim
--
Joakim Verona
Re: cocoon 2.2 application memory consumption problems
Posted by Vadim Gritsenko <va...@reverycodes.com>.
joakim@verona.se wrote:
> I did some more studying with jhat, and it appears that the jpegs
> inside the zips are being held in memory by the cocoon cache, in the
> end leading to the out of memory condition.
...
> However, I find it odd the cache doesnt release the objects when
> memory grows scarce.
It does, when configured properly. Check your store janitor settings.
Vadim
Re: cocoon 2.2 application memory consumption problems
Posted by jo...@verona.se.
I did some more studying with jhat, and it appears that the jpegs
inside the zips are being held in memory by the cocoon cache, in the
end leading to the out of memory condition.
I used a "noncaching" pipeline instead, and the problem seems to go
away. However, I find it odd the cache doesnt release the objects when
memory grows scarce.
joakim@verona.se writes:
> Grzegorz Kossakowski <gk...@apache.org> writes:
>
>> joakim@verona.se pisze:
>>> Hello list,
>>>
>>> I've made a small cocoon 2.2 application which browses images
>>> contained i zip files.
>>>
>>> I started out with the cocoon project archetype.
>>>
>>> Basically I have a flowscript which figures out the content of a zip
>>> file using this:
>>>
>>> var imgarc=new Packages.java.util.zip.ZipFile(basepath);
>>> gets the Enumeration of the files, converts it to a list, sorts
>>> it,
>>> and then does sendpage to a view page for each file.
>>>
>>> The image file in the zip is viewed with a cocoon jar protocol reader:
>>>
>>> <map:match pattern="flib/**.cbz!**">
>>> <map:read src="jar:file:///flib/{1}.cbz!/{2}"/>
>>> </map:match>
>>>
>>> This application consumes about 1.5Gb virtual, 170 Mb resident,
>>> and I run out of heapspace after a while.
>>>
>>> The application is very simple. How can I debug this memory problem?
>>>
>>> I also use the Directory Generator to browse the files, and an xslt
>>> with some xpath 2.0 so I've configured saxxon.
>>
>> Cocoon has a status generator that can give you information about
>> objects stored in cache. You can see it in action by running
>> cocoon-webapp and accessing
>> http://localhost:8888/blocks/cocoon-core-main-sample/system/ (I
>> discovered that styling is little broken so I suggest to view it as
>> XML to not miss any important bits).
>>
>> For more general memory measurements I suggest to use standard Java monitoring and profiling tools.
>
> The status block didnt give me much.
>
> I tried "jhat" on a hep dump of when cocoon runs out of heap space.
> I couldnt immediately grasp what is wrong, but it appears that most
> heap space is occupied by large byte arrays, and it also appears that
> lots of these byte arrays are held by zip classes.
>
> I dont currently know if these zip clases are leftovers from when i
> open the zips in floswcript(but I dont think so), or they are left
> from the cooon zip reader.
>
> heres the flowsript that reads filenames from zips. since it just
> copies the names to a list and returns those, the zip should be garbed:
>
> function zipimagelist(basepath){
> //figure out the images sequence from basepath to view
> var imgarc=new Packages.java.util.zip.ZipFile(basepath);
> var imgarcentries=imgarc.entries();
>
> var keyList = new Packages.java.util.ArrayList();
> re=/.jpg|.png|.gif/i;//add all browser compatible image types here
> while(imgarcentries.hasMoreElements()){
> var imagename=imgarcentries.nextElement().toString();
> if(imagename.match(re)) //only add images to the list
> keyList.add("/myBlock1//"+basepath+"!"+imagename);
> }
> imgarc=null; //at this point, delete zipfile from mem
> imgarcentries=null;
>
> Packages.java.util.Collections.sort(keyList);
> print(keyList);
>
> return keyList;
> }
>
> and heres the very simple viewer pipeline:
>
> <map:match pattern="flib/**.cbz!**">
> <map:read src="zip:file:///flib/{1}.cbz!/{2}"/>
> </map:match>
>
> My current suspicion is that the zip protocol reader leaks, but im
> unsure how to verify this further.
>
>> --
>> Grzegorz Kossakowski
>> http://reflectingonthevicissitudes.wordpress.com/
>
> --
> Joakim Verona
--
Joakim Verona
Re: cocoon 2.2 application memory consumption problems
Posted by jo...@verona.se.
Grzegorz Kossakowski <gk...@apache.org> writes:
> joakim@verona.se pisze:
>> Hello list,
>>
>> I've made a small cocoon 2.2 application which browses images
>> contained i zip files.
>>
>> I started out with the cocoon project archetype.
>>
>> Basically I have a flowscript which figures out the content of a zip
>> file using this:
>>
>> var imgarc=new Packages.java.util.zip.ZipFile(basepath);
>> gets the Enumeration of the files, converts it to a list, sorts
>> it,
>> and then does sendpage to a view page for each file.
>>
>> The image file in the zip is viewed with a cocoon jar protocol reader:
>>
>> <map:match pattern="flib/**.cbz!**">
>> <map:read src="jar:file:///flib/{1}.cbz!/{2}"/>
>> </map:match>
>>
>> This application consumes about 1.5Gb virtual, 170 Mb resident,
>> and I run out of heapspace after a while.
>>
>> The application is very simple. How can I debug this memory problem?
>>
>> I also use the Directory Generator to browse the files, and an xslt
>> with some xpath 2.0 so I've configured saxxon.
>
> Cocoon has a status generator that can give you information about
> objects stored in cache. You can see it in action by running
> cocoon-webapp and accessing
> http://localhost:8888/blocks/cocoon-core-main-sample/system/ (I
> discovered that styling is little broken so I suggest to view it as
> XML to not miss any important bits).
>
> For more general memory measurements I suggest to use standard Java monitoring and profiling tools.
The status block didnt give me much.
I tried "jhat" on a hep dump of when cocoon runs out of heap space.
I couldnt immediately grasp what is wrong, but it appears that most
heap space is occupied by large byte arrays, and it also appears that
lots of these byte arrays are held by zip classes.
I dont currently know if these zip clases are leftovers from when i
open the zips in floswcript(but I dont think so), or they are left
from the cooon zip reader.
heres the flowsript that reads filenames from zips. since it just
copies the names to a list and returns those, the zip should be garbed:
function zipimagelist(basepath){
//figure out the images sequence from basepath to view
var imgarc=new Packages.java.util.zip.ZipFile(basepath);
var imgarcentries=imgarc.entries();
var keyList = new Packages.java.util.ArrayList();
re=/.jpg|.png|.gif/i;//add all browser compatible image types here
while(imgarcentries.hasMoreElements()){
var imagename=imgarcentries.nextElement().toString();
if(imagename.match(re)) //only add images to the list
keyList.add("/myBlock1//"+basepath+"!"+imagename);
}
imgarc=null; //at this point, delete zipfile from mem
imgarcentries=null;
Packages.java.util.Collections.sort(keyList);
print(keyList);
return keyList;
}
and heres the very simple viewer pipeline:
<map:match pattern="flib/**.cbz!**">
<map:read src="zip:file:///flib/{1}.cbz!/{2}"/>
</map:match>
My current suspicion is that the zip protocol reader leaks, but im
unsure how to verify this further.
> --
> Grzegorz Kossakowski
> http://reflectingonthevicissitudes.wordpress.com/
--
Joakim Verona
Re: cocoon 2.2 application memory consumption problems
Posted by Ralph Goers <Ra...@dslextreme.com>.
Actually, It would be kind of nice if someone could volunteer to profile
Cocoon. I'd do it but I've been underwater at work for months now.
YourKit is free to Cocoon committers.
Ralph
Grzegorz Kossakowski wrote:
> joakim@verona.se pisze:
>> Hello list,
>>
>> I've made a small cocoon 2.2 application which browses images
>> contained i zip files.
>>
>> I started out with the cocoon project archetype.
>>
>> Basically I have a flowscript which figures out the content of a zip
>> file using this:
>>
>> var imgarc=new Packages.java.util.zip.ZipFile(basepath);
>> gets the Enumeration of the files, converts it to a list, sorts it,
>> and then does sendpage to a view page for each file.
>>
>> The image file in the zip is viewed with a cocoon jar protocol reader:
>>
>> <map:match pattern="flib/**.cbz!**">
>> <map:read src="jar:file:///flib/{1}.cbz!/{2}"/>
>> </map:match>
>>
>> This application consumes about 1.5Gb virtual, 170 Mb resident,
>> and I run out of heapspace after a while.
>>
>> The application is very simple. How can I debug this memory problem?
>>
>> I also use the Directory Generator to browse the files, and an xslt
>> with some xpath 2.0 so I've configured saxxon.
>
> Cocoon has a status generator that can give you information about
> objects stored in cache. You can see it in action by running
> cocoon-webapp and accessing
> http://localhost:8888/blocks/cocoon-core-main-sample/system/ (I
> discovered that styling is little broken so I suggest to view it as
> XML to not miss any important bits).
>
> For more general memory measurements I suggest to use standard Java
> monitoring and profiling tools.
>
Re: cocoon 2.2 application memory consumption problems
Posted by Grzegorz Kossakowski <gk...@apache.org>.
joakim@verona.se pisze:
> Hello list,
>
> I've made a small cocoon 2.2 application which browses images
> contained i zip files.
>
> I started out with the cocoon project archetype.
>
> Basically I have a flowscript which figures out the content of a zip
> file using this:
>
> var imgarc=new Packages.java.util.zip.ZipFile(basepath);
>
> gets the Enumeration of the files, converts it to a list, sorts it,
> and then does sendpage to a view page for each file.
>
> The image file in the zip is viewed with a cocoon jar protocol reader:
>
> <map:match pattern="flib/**.cbz!**">
> <map:read src="jar:file:///flib/{1}.cbz!/{2}"/>
> </map:match>
>
> This application consumes about 1.5Gb virtual, 170 Mb resident,
> and I run out of heapspace after a while.
>
> The application is very simple. How can I debug this memory problem?
>
> I also use the Directory Generator to browse the files, and an xslt
> with some xpath 2.0 so I've configured saxxon.
Cocoon has a status generator that can give you information about objects stored in cache. You can see it in action by running cocoon-webapp
and accessing http://localhost:8888/blocks/cocoon-core-main-sample/system/ (I discovered that styling is little broken so I suggest to view
it as XML to not miss any important bits).
For more general memory measurements I suggest to use standard Java monitoring and profiling tools.
--
Grzegorz Kossakowski
http://reflectingonthevicissitudes.wordpress.com/