You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Felix Knecht <fe...@apache.org> on 2008/03/05 17:19:08 UTC
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Robert
To get an answer to such questions the cocoon developer list would be
the right place (i'll do it for you this time as cc)! That's what the
lists are for!
See also [1] for cocoons mailing lists
[1] http://cocoon.apache.org/1275_1_1.html
>
> We are encountering this same problem in Cocoon 2.1.11. What is the
> status of this issue? Is there a fix for it? It doesn't seem to
> exist in Cocoon 2.0.4.
Obviously net yet, otherwise you could see it in the issue tracker.
Maybe BufferedOutputStream was introduced after cocoon 2.0.4 so that
version doesn't suffers this issue.
Ciao Felix
>
> https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568953#action_12568953
>
>
> Sincerely,
> Robert La Ferla
> OMS
>
>
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 08.03.2008 15:28, Felix Knecht wrote:
>> Felix, have you tried outputBufferSize?
>
> Not yet. I didn't even knew how to configure it :-(
Both map:pipe and map:pipeline should work as mentioned here:
http://marc.info/?l=xml-cocoon-dev&m=120477640924395&w=4
Joerg
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Felix Knecht <fe...@apache.org>.
Joerg Heinicke schrieb:
> On 05.03.2008 23:06, Joerg Heinicke wrote:
>
>> From what I see from the code (AbstractProcessingPipeline) it is
>> possible to configure and setup/parameterize a pipeline with
>> "outputBufferSize". This means on both map:pipe and map:pipeline it
>> should be possible to set an actual buffer size. Only if none is set
>> (and it defaults to -1) the non-flushing BufferedOutputStream is used.
>
> I found a thread on the users list claiming outputBufferSize does not
> work [1] at the moment (or since some time). Carsten wrote he sees a
> potential problem but does not outline them.
>
> Felix, have you tried outputBufferSize?
Not yet. I didn't even knew how to configure it :-(
Felix
>
> Carsten, can you tell us what might be wrong?
>
> Joerg
>
> [1] http://marc.info/?t=120111212300011&r=1&w=4
>
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Felix Knecht wrote:
> Joerg Heinicke schrieb:
>> On 05.03.2008 23:06, Joerg Heinicke wrote:
>>
>>> From what I see from the code (AbstractProcessingPipeline) it is
>>> possible to configure and setup/parameterize a pipeline with
>>> "outputBufferSize". This means on both map:pipe and map:pipeline it
>>> should be possible to set an actual buffer size. Only if none is set
>>> (and it defaults to -1) the non-flushing BufferedOutputStream is used.
>>
>> I found a thread on the users list claiming outputBufferSize does not
>> work [1] at the moment (or since some time). Carsten wrote he sees a
>> potential problem but does not outline them.
>
> ATM we are always talking about Buffered... I'm believe this has to do
> with caching the pipelines, right?
> So I wonder really why my the issue also occurs in noncaching pipelines?
>
Sorry for the late response. I thought that the output stream handling
implementation in AbstractEnvironment is wrong. But looking at the code
again I'm not sure anymore :( For whatever reason I had the thought the
multiple calls to getOutputStream(int) - perhaps with different buffer
sizes - could reveal a wrong output stream to be used.
Don't know if this is the actual problem, but I think we should throw an
exception if getOutputStream is called twice. It should only be called
once during a request.
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 08.03.2008 15:34, Felix Knecht wrote:
> ATM we are always talking about Buffered... I'm believe this has to do
> with caching the pipelines, right?
> So I wonder really why my the issue also occurs in noncaching pipelines?
No, it has to do with the capability of resetting the OutputStream to
send an error response in case of an error. If your buffer is too small,
a first part of the actual content might already have been flushed when
the error occurs and the error content is appended. Looks weird at the
client's browser ;)
Joerg
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Felix Knecht <fe...@apache.org>.
Joerg Heinicke schrieb:
> On 05.03.2008 23:06, Joerg Heinicke wrote:
>
>> From what I see from the code (AbstractProcessingPipeline) it is
>> possible to configure and setup/parameterize a pipeline with
>> "outputBufferSize". This means on both map:pipe and map:pipeline it
>> should be possible to set an actual buffer size. Only if none is set
>> (and it defaults to -1) the non-flushing BufferedOutputStream is used.
>
> I found a thread on the users list claiming outputBufferSize does not
> work [1] at the moment (or since some time). Carsten wrote he sees a
> potential problem but does not outline them.
ATM we are always talking about Buffered... I'm believe this has to do with caching the pipelines, right?
So I wonder really why my the issue also occurs in noncaching pipelines?
Felix
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 05.03.2008 23:06, Joerg Heinicke wrote:
> From what I see from the code (AbstractProcessingPipeline) it is
> possible to configure and setup/parameterize a pipeline with
> "outputBufferSize". This means on both map:pipe and map:pipeline it
> should be possible to set an actual buffer size. Only if none is set
> (and it defaults to -1) the non-flushing BufferedOutputStream is used.
I found a thread on the users list claiming outputBufferSize does not
work [1] at the moment (or since some time). Carsten wrote he sees a
potential problem but does not outline them.
Felix, have you tried outputBufferSize?
Carsten, can you tell us what might be wrong?
Joerg
[1] http://marc.info/?t=120111212300011&r=1&w=4
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Grzegorz Kossakowski <gr...@tuffmail.com>.
Antonio Gallardo pisze:
> 1MB Makes sense.
+1
--
Best regards,
Grzegorz Kossakowski
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Antonio Gallardo <ag...@agssa.net>.
1MB Makes sense.
Best Regards,
Antonio Gallardo.
Joerg Heinicke escribió:
> On 05.03.2008 23:06, Joerg Heinicke wrote:
>
>> We could argue about another default value than -1 though. Something
>> like 1024^2.
>
> What do others think? Shall we change the default value from "buffer
> everything" (which lead to the OutOfMemoryError [1]) to something more
> "secure" in the sense of avoiding a potential source of error? Besides
> mentioning it on the changes page we have to set it to a value that's
> unlikely to be hit with a normal web application to change user
> application's behavior only in extreme cases. That's why I suggested 1MB.
>
> Joerg
>
> [1] https://issues.apache.org/jira/browse/COCOON-2168
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Antonio Gallardo <ag...@agssa.net>.
Carsten Ziegeler escribió:
> Hmm, ok, we could change this in the main sitemap as a default
> configuration while leaving it in the java code untouched.
> However, I still think that this is not needed, if people want to
> stream huge responses, they should think about what they are doing and
> configure everything accordingly. I totally agree that we lack
> documentation here.
+1
Best Regards,
Antonio Gallardo.
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Joerg Heinicke wrote:
> On 11.03.2008 08:54, Carsten Ziegeler wrote:
>
>>>>>> We could argue about another default value than -1 though.
>>>>>> Something like 1024^2.
>>>>>
>>>> Hmm, not sure if we should change the default value. The idea of
>>>> this default was to be sure that error handling works out of the
>>>> box. If you have special cases like mentioned in the bug, it makes
>>>> imho more sense to fine tune these special cases (for instance by
>>>> not buffering).
>>>
>>>> The output buffer value is one of the settings which is "optimized"
>>>> for development and it should be tweaked for production usage.
>>>
>> Hmm, ok, we could change this in the main sitemap as a default
>> configuration while leaving it in the java code untouched.
>> However, I still think that this is not needed, if people want to
>> stream huge responses, they should think about what they are doing and
>> configure everything accordingly. I totally agree that we lack
>> documentation here.
>
> I really don't agree with this reasoning but I don't want to stress it
> too much to not get on your nerves, so it's my last try :-)
Hehe :) Ok, and I try hard that this is my last response :)
>
> 1. This IS already documented in the main sitemap and I think though
> hardly anybody is aware of it: Nobody had reacted on Felix' issue entry
> for 3 weeks. I can also see why: our main sitemap is the first thing I
> would get rid of when creating my own Cocoon application ;-)
>
> 2. Actually I don't care about the huge resources that much. I rather
> have big resources in mind - and concurrent users. What's worse as a
> default behavior for an incautious user: a screwed error page or a
> server brought down with an OOME? I also wanted to put the default as
> big as 1 MB so that it hardly affects anybody.
>
> I really don't see an additional value of endless buffering over such a
> big buffer. But I can prevent a fatal user error. And for everything
> that's beyond the default buffer size, here I completely agree with you,
> the user should think about potential issues in terms of buffering or
> caching.
>
> 3. Another, I admit minor point after so long time might be the change
> in behavior compared to former Cocoon versions. According to Robert [1]
> in Cocoon 2.0.4 it used to work out of the box.
>
Yes, we discussed this for 2.1.0 and decided that it's better to have a
correct error response per default.
Ok, I agree that the difference is really very subtle, so lets change
it; I guess one mb should be enough. I guess you want to change it for
2.1.x as well, right?
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 11.03.2008 08:54, Carsten Ziegeler wrote:
>>>>> We could argue about another default value than -1 though.
>>>>> Something like 1024^2.
>>>>
>>> Hmm, not sure if we should change the default value. The idea of this
>>> default was to be sure that error handling works out of the box. If
>>> you have special cases like mentioned in the bug, it makes imho more
>>> sense to fine tune these special cases (for instance by not buffering).
>>
>>> The output buffer value is one of the settings which is "optimized"
>>> for development and it should be tweaked for production usage.
>>
> Hmm, ok, we could change this in the main sitemap as a default
> configuration while leaving it in the java code untouched.
> However, I still think that this is not needed, if people want to stream
> huge responses, they should think about what they are doing and
> configure everything accordingly. I totally agree that we lack
> documentation here.
I really don't agree with this reasoning but I don't want to stress it
too much to not get on your nerves, so it's my last try :-)
1. This IS already documented in the main sitemap and I think though
hardly anybody is aware of it: Nobody had reacted on Felix' issue entry
for 3 weeks. I can also see why: our main sitemap is the first thing I
would get rid of when creating my own Cocoon application ;-)
2. Actually I don't care about the huge resources that much. I rather
have big resources in mind - and concurrent users. What's worse as a
default behavior for an incautious user: a screwed error page or a
server brought down with an OOME? I also wanted to put the default as
big as 1 MB so that it hardly affects anybody.
I really don't see an additional value of endless buffering over such a
big buffer. But I can prevent a fatal user error. And for everything
that's beyond the default buffer size, here I completely agree with you,
the user should think about potential issues in terms of buffering or
caching.
3. Another, I admit minor point after so long time might be the change
in behavior compared to former Cocoon versions. According to Robert [1]
in Cocoon 2.0.4 it used to work out of the box.
Joerg
[1] http://marc.info/?l=xml-cocoon-dev&m=120473398413381&w=4
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Joerg Heinicke wrote:
> On 11.03.2008 04:48, Carsten Ziegeler wrote:
>
>>>> We could argue about another default value than -1 though. Something
>>>> like 1024^2.
>>>
>>> What do others think? Shall we change the default value from "buffer
>>> everything" (which lead to the OutOfMemoryError [1]) to something
>>> more "secure" in the sense of avoiding a potential source of error?
>>> Besides mentioning it on the changes page we have to set it to a
>>> value that's unlikely to be hit with a normal web application to
>>> change user application's behavior only in extreme cases. That's why
>>> I suggested 1MB.
>>>
>> Hmm, not sure if we should change the default value. The idea of this
>> default was to be sure that error handling works out of the box. If
>> you have special cases like mentioned in the bug, it makes imho more
>> sense to fine tune these special cases (for instance by not buffering).
>
> But I fear hardly anybody is aware or even uses the feature.
>
Yes, that's possible.
>> The output buffer value is one of the settings which is "optimized"
>> for development and it should be tweaked for production usage.
>
> It's not really development, is it? I mean even if you can not reset
> output buffer completely, you will still get the error markup appended
> and I would not care for development about how this looks :)
Hmm, I would never rely on the default error handling for these cases in
production environments. If something is already written to the output
stream, it's too late anyway. But I see your point.
>
> Being aware of the potential change in behavior I also chose a quite
> huge buffer of 1 MB so that hardly anybody should be affected. We can
> also discuss about making it even bigger like 10 MB. But I consider a
> buffer that's flushed too early once in a while better than an OOME in
> the default setup. And people can still change it to -1 and endless
> buffering if they really need. But at least they are aware of the
> effects then.
Hmm, ok, we could change this in the main sitemap as a default
configuration while leaving it in the java code untouched.
However, I still think that this is not needed, if people want to stream
huge responses, they should think about what they are doing and
configure everything accordingly. I totally agree that we lack
documentation here.
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 11.03.2008 04:48, Carsten Ziegeler wrote:
>>> We could argue about another default value than -1 though. Something
>>> like 1024^2.
>>
>> What do others think? Shall we change the default value from "buffer
>> everything" (which lead to the OutOfMemoryError [1]) to something more
>> "secure" in the sense of avoiding a potential source of error? Besides
>> mentioning it on the changes page we have to set it to a value that's
>> unlikely to be hit with a normal web application to change user
>> application's behavior only in extreme cases. That's why I suggested 1MB.
>>
> Hmm, not sure if we should change the default value. The idea of this
> default was to be sure that error handling works out of the box. If you
> have special cases like mentioned in the bug, it makes imho more sense
> to fine tune these special cases (for instance by not buffering).
But I fear hardly anybody is aware or even uses the feature.
> The output buffer value is one of the settings which is "optimized" for
> development and it should be tweaked for production usage.
It's not really development, is it? I mean even if you can not reset
output buffer completely, you will still get the error markup appended
and I would not care for development about how this looks :)
Being aware of the potential change in behavior I also chose a quite
huge buffer of 1 MB so that hardly anybody should be affected. We can
also discuss about making it even bigger like 10 MB. But I consider a
buffer that's flushed too early once in a while better than an OOME in
the default setup. And people can still change it to -1 and endless
buffering if they really need. But at least they are aware of the
effects then.
Joerg
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Felix Knecht wrote:
>
>> You need to turn off the buffering of the pipeline as well. I don't
>> have the parameter name at hand, I assume it's "buffer-size" as well
>> but could be different:
>>
>> <map:pipeline id="test-nocache" type="noncaching">
>> <map:parameter name="buffer-size" value="0" />
>> <map:match pattern="nocache">
>> <map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso"
>> <map:parameter name="buffer-size" value="8192" />
>> </map:read>
>> </map:match>
>> </map:pipeline>
>>
>> This is usually the way I would define reader pipelines. If the above
>> still produces an OOMError than we have a bug :)
> Thanks Carsten, it works. Parameter is "outputBufferSize"
Ah, yes :) Great!
> I may ask at this point if the general configuration of noncaching
> pipelines is correct
> (cocoon-core/cocoon-core/src/main/resources/META-INF/cocoon/avalon/cocoon-core-sitemapcomponents.xconf):
>
> <map:pipe name="noncaching"
> src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline">
>
> <!-- parameter name="outputBufferSize" value="8192"/ -->
> </map:pipe>
>
> ==> As no parameter is specified '-1' is used what in fact leads to the
> same configuration as for caching pipelines?!
>
Yes, this is correct :) The output buffer only specifies the size of the
buffer for writing the response. This is not directly related to
caching. You might increase performance by buffering.
The buffer is in both case unlimited because of allowing proper error
handling.
If we don't have it yet, we should add these things to a "tuning cocoon"
page. I would turn off infinite buffering in production in all case and
set a fixed buffer size (like 8192). For reader pipelines I would turn
off buffering completly.
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Felix Knecht <fe...@apache.org>.
> You need to turn off the buffering of the pipeline as well. I don't
> have the parameter name at hand, I assume it's "buffer-size" as well
> but could be different:
>
> <map:pipeline id="test-nocache" type="noncaching">
> <map:parameter name="buffer-size" value="0" />
> <map:match pattern="nocache">
> <map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso"
> <map:parameter name="buffer-size" value="8192" />
> </map:read>
> </map:match>
> </map:pipeline>
>
> This is usually the way I would define reader pipelines. If the above
> still produces an OOMError than we have a bug :)
Thanks Carsten, it works. Parameter is "outputBufferSize"
<map:pipeline id="test-nocache" type="noncaching">
<map:parameter name="outputBufferSize" value="0" />
<map:match pattern="nocache">
<map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >
<map:parameter name="buffer-size" value="8192" />
</map:read>
</map:match>
</map:pipeline>
I may ask at this point if the general configuration of noncaching
pipelines is correct
(cocoon-core/cocoon-core/src/main/resources/META-INF/cocoon/avalon/cocoon-core-sitemapcomponents.xconf):
<map:pipe name="noncaching"
src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline">
<!-- parameter name="outputBufferSize" value="8192"/ -->
</map:pipe>
==> As no parameter is specified '-1' is used what in fact leads to the
same configuration as for caching pipelines?!
Felix
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Felix Knecht schrieb:
> Carsten Ziegeler schrieb:
>> Joerg Heinicke wrote:
>>> On 05.03.2008 23:06, Joerg Heinicke wrote:
>>>
>>>> We could argue about another default value than -1 though. Something
>>>> like 1024^2.
>>>
>>> What do others think? Shall we change the default value from "buffer
>>> everything" (which lead to the OutOfMemoryError [1]) to something
>>> more "secure" in the sense of avoiding a potential source of error?
>>> Besides mentioning it on the changes page we have to set it to a
>>> value that's unlikely to be hit with a normal web application to
>>> change user application's behavior only in extreme cases. That's why
>>> I suggested 1MB.
>>>
>> Hmm, not sure if we should change the default value. The idea of this
>> default was to be sure that error handling works out of the box. If
>> you have special cases like mentioned in the bug, it makes imho more
>> sense to fine tune these special cases (for instance by not buffering).
>>
>> The output buffer value is one of the settings which is "optimized"
>> for development and it should be tweaked for production usage. I think
>> also if you're using a reader in your pipeline it is more likely that
>> you don't want to let the pipeline buffer your output.
>
> IIUC this should work (no caching pipeline, set buffer-size to 8192):
>
> <map:pipeline id="test-nocache" type="noncaching">
> <map:match pattern="nocache">
> <map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >
> <map:parameter name="buffer-size" value="8192" />
> </map:read>
> </map:match>
> </map:pipeline>
>
> but it doesn't
>
> java.lang.OutOfMemoryError: Java heap space
You need to turn off the buffering of the pipeline as well. I don't have
the parameter name at hand, I assume it's "buffer-size" as well but
could be different:
<map:pipeline id="test-nocache" type="noncaching">
<map:parameter name="buffer-size" value="0" />
<map:match pattern="nocache">
<map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso"
<map:parameter name="buffer-size" value="8192" />
</map:read>
</map:match>
</map:pipeline>
This is usually the way I would define reader pipelines. If the above
still produces an OOMError than we have a bug :)
Carsten
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Felix Knecht <fe...@apache.org>.
Carsten Ziegeler schrieb:
> Joerg Heinicke wrote:
>> On 05.03.2008 23:06, Joerg Heinicke wrote:
>>
>>> We could argue about another default value than -1 though. Something
>>> like 1024^2.
>>
>> What do others think? Shall we change the default value from "buffer
>> everything" (which lead to the OutOfMemoryError [1]) to something
>> more "secure" in the sense of avoiding a potential source of error?
>> Besides mentioning it on the changes page we have to set it to a
>> value that's unlikely to be hit with a normal web application to
>> change user application's behavior only in extreme cases. That's why
>> I suggested 1MB.
>>
> Hmm, not sure if we should change the default value. The idea of this
> default was to be sure that error handling works out of the box. If
> you have special cases like mentioned in the bug, it makes imho more
> sense to fine tune these special cases (for instance by not buffering).
>
> The output buffer value is one of the settings which is "optimized"
> for development and it should be tweaked for production usage. I think
> also if you're using a reader in your pipeline it is more likely that
> you don't want to let the pipeline buffer your output.
IIUC this should work (no caching pipeline, set buffer-size to 8192):
<map:pipeline id="test-nocache" type="noncaching">
<map:match pattern="nocache">
<map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >
<map:parameter name="buffer-size" value="8192" />
</map:read>
</map:match>
</map:pipeline>
but it doesn't
java.lang.OutOfMemoryError: Java heap space
at
org.apache.cocoon.util.BufferedOutputStream.incBuffer(BufferedOutputStream.java:148)
at
org.apache.cocoon.util.BufferedOutputStream.write(BufferedOutputStream.java:96)
at
org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:355)
at
org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:386)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processReader(AbstractProcessingPipeline.java:656)
at
org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)
at $Proxy8.process(Unknown Source)
at
org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
at
org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)
at
org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at
org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
at
org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
at
org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
at
org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
at
org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
at
org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at
org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:501)
at
org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:473)
at
org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:230)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy5.service(Unknown Source)
Felix
>
> Carsten
>
>> Joerg
>>
>> [1] https://issues.apache.org/jira/browse/COCOON-2168
>>
>
>
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Carsten Ziegeler <cz...@apache.org>.
Joerg Heinicke wrote:
> On 05.03.2008 23:06, Joerg Heinicke wrote:
>
>> We could argue about another default value than -1 though. Something
>> like 1024^2.
>
> What do others think? Shall we change the default value from "buffer
> everything" (which lead to the OutOfMemoryError [1]) to something more
> "secure" in the sense of avoiding a potential source of error? Besides
> mentioning it on the changes page we have to set it to a value that's
> unlikely to be hit with a normal web application to change user
> application's behavior only in extreme cases. That's why I suggested 1MB.
>
Hmm, not sure if we should change the default value. The idea of this
default was to be sure that error handling works out of the box. If you
have special cases like mentioned in the bug, it makes imho more sense
to fine tune these special cases (for instance by not buffering).
The output buffer value is one of the settings which is "optimized" for
development and it should be tweaked for production usage. I think also
if you're using a reader in your pipeline it is more likely that you
don't want to let the pipeline buffer your output.
Carsten
> Joerg
>
> [1] https://issues.apache.org/jira/browse/COCOON-2168
>
--
Carsten Ziegeler
cziegeler@apache.org
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 05.03.2008 23:06, Joerg Heinicke wrote:
> We could argue about another default value than -1 though. Something
> like 1024^2.
What do others think? Shall we change the default value from "buffer
everything" (which lead to the OutOfMemoryError [1]) to something more
"secure" in the sense of avoiding a potential source of error? Besides
mentioning it on the changes page we have to set it to a value that's
unlikely to be hit with a normal web application to change user
application's behavior only in extreme cases. That's why I suggested 1MB.
Joerg
[1] https://issues.apache.org/jira/browse/COCOON-2168
Re: [#COCOON-2168] ResourceReader produces Java Heap Overflow when
reading a huge resource - ASF JIRA
Posted by Joerg Heinicke <jo...@gmx.de>.
On 05.03.2008 11:19, Felix Knecht wrote:
>> We are encountering this same problem in Cocoon 2.1.11. What is the
>> status of this issue? Is there a fix for it? It doesn't seem to
>> exist in Cocoon 2.0.4.
>
> Obviously net yet, otherwise you could see it in the issue tracker.
> Maybe BufferedOutputStream was introduced after cocoon 2.0.4 so that
> version doesn't suffers this issue.
I've looked into this issue and I'm against Felix' fix. It stands
completely against the idea of buffering the whole pipeline content
which is done for error handlers. It should be possible for them to
reset the OutputStream. Or asked in other words? Why using the
non-flushing BufferedOutputStream at all?
Now how to handle the issue with huge resources? Should be quite easy.
From what I see from the code (AbstractProcessingPipeline) it is
possible to configure and setup/parameterize a pipeline with
"outputBufferSize". This means on both map:pipe and map:pipeline it
should be possible to set an actual buffer size. Only if none is set
(and it defaults to -1) the non-flushing BufferedOutputStream is used.
Root sitemap has this example (in map:components/map:pipes section):
<map:pipe name="noncaching"
src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline"
pool-max="${noncaching-pipeline.pool-max}">
<!-- parameter name="outputBufferSize" value="8192"/ -->
</map:pipe>
The following should also work (in map:pipelines section):
<map:pipeline>
<map:parameter name="outputBufferSize" value="8192"/>
...
</map:pipeline>
We could argue about another default value than -1 though. Something
like 1024^2.
WDYT?
Joerg