You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by "Felix Knecht (JIRA)" <ji...@apache.org> on 2008/02/14 13:47:08 UTC
[jira] Created: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
ResourceReader produces Java Heap Overflow when reading a huge resource
-----------------------------------------------------------------------
Key: COCOON-2168
URL: https://issues.apache.org/jira/browse/COCOON-2168
Project: Cocoon
Issue Type: Bug
Components: * Cocoon Core
Affects Versions: 2.2-dev (Current SVN)
Reporter: Felix Knecht
Assignee: Felix Knecht
When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12577373#action_12577373 ]
Felix Knecht commented on COCOON-2168:
--------------------------------------
After all it turned out that it's a problem of configuration. Thanks to Carsten.
Last snippet of the mail thread thread http://marc.info/?t=120473411300003&r=1&w=2
<map:pipeline id="test-nocache" type="noncaching">
<map:parameter name="outputBufferSize" value="0" />
<map:match pattern="nocache">
<map:read src="/home/felix/tmp/livecd-i686-installer-2007.0.iso" >
<map:parameter name="buffer-size" value="8192" />
</map:read>
</map:match>
</map:pipeline>
The output buffer only specifies the size of the buffer for writing the response. This is not directly related to caching. You might increase performance by buffering.
The buffer is in both case unlimited because of allowing proper error handling.
If we don't have it yet, we should add these things to a "tuning cocoon" page. I would turn off infinite buffering in production in all case and set a fixed buffer size (like 8192). For reader pipelines I would turn off buffering completly.
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Knecht updated COCOON-2168:
---------------------------------
Attachment: test-case.tar.gz
ResourceReader.diff
Patch for ResourceReader
Patch with test block
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Assignee: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12568953#action_12568953 ]
Felix Knecht commented on COCOON-2168:
--------------------------------------
It seems that the problem is situated somewhere deeper. After fixing the ResourceReader I'm getting next errors, depending on the pipeline I use (caching/noncaching).
The errors I'm getting now (after applying the attached patch for the ResourceReader with the test block):
Please replace in the test block the file which is read with a huge file you have on your machine (no need to attache a 700MB file here).
Caching:
2008-02-14 15:35:20.955:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'.
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2786)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:71)
at org.apache.cocoon.servletservice.ServletServiceContext$StatusRetrievableBufferedWrappedResponse$1.write(ServletServiceContext.java:593)
at java.io.OutputStream.write(OutputStream.java:99)
at org.apache.cocoon.util.BufferedOutputStream.writeBuffer(BufferedOutputStream.java:137)
at org.apache.cocoon.util.BufferedOutputStream.realFlush(BufferedOutputStream.java:127)
at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:391)
at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)
at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.processReader(AbstractProcessingPipeline.java:656)
at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)
at $Proxy8.process(Unknown Source)
at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)
at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479
Noncaching:
2008-02-14 15:34:55.179:/:INFO: Apache Cocoon Spring Configurator v1.0.2-SNAPSHOT is running in mode 'dev'.
java.lang.OutOfMemoryError: Java heap space
at org.apache.cocoon.util.BufferedOutputStream.incBuffer(BufferedOutputStream.java:148)
at org.apache.cocoon.util.BufferedOutputStream.write(BufferedOutputStream.java:96)
at org.apache.cocoon.caching.CachingOutputStream.write(CachingOutputStream.java:69)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at org.apache.cocoon.reading.ResourceReader.processStream(ResourceReader.java:386)
at org.apache.cocoon.reading.ResourceReader.generate(ResourceReader.java:420)
at org.apache.cocoon.components.pipeline.impl.AbstractCachingProcessingPipeline.processReader(AbstractCachingProcessingPipeline.java:866)
at org.apache.cocoon.components.pipeline.AbstractProcessingPipeline.process(AbstractProcessingPipeline.java:431)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.cocoon.core.container.spring.avalon.PoolableProxyHandler.invoke(PoolableProxyHandler.java:72)
at $Proxy8.process(Unknown Source)
at org.apache.cocoon.components.treeprocessor.sitemap.ReadNode.invoke(ReadNode.java:94)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:55)
at org.apache.cocoon.components.treeprocessor.sitemap.MatchNode.invoke(MatchNode.java:87)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at org.apache.cocoon.components.treeprocessor.sitemap.PipelineNode.invoke(PipelineNode.java:144)
at org.apache.cocoon.components.treeprocessor.AbstractParentProcessingNode.invokeNodes(AbstractParentProcessingNode.java:78)
at org.apache.cocoon.components.treeprocessor.sitemap.PipelinesNode.invoke(PipelinesNode.java:81)
at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:239)
at org.apache.cocoon.components.treeprocessor.ConcreteTreeProcessor.process(ConcreteTreeProcessor.java:171)
at org.apache.cocoon.components.treeprocessor.TreeProcessor.process(TreeProcessor.java:247)
at org.apache.cocoon.servlet.RequestProcessor.process(RequestProcessor.java:351)
at org.apache.cocoon.servlet.RequestProcessor.service(RequestProcessor.java:169)
at org.apache.cocoon.sitemap.SitemapServlet.service(SitemapServlet.java:84)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:512)
at org.apache.cocoon.servletservice.ServletServiceContext$PathDispatcher.forward(ServletServiceContext.java:479)
at org.apache.cocoon.servletservice.spring.ServletFactoryBean$ServiceInterceptor.invoke(ServletFactoryBean.java:230)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Assignee: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575626#action_12575626 ]
Felix Knecht commented on COCOON-2168:
--------------------------------------
I agree with Joerg that my patch is against the idea of buffering the whole pipeline. OTH (looking at security issues) I think it can be dangerous to accept Overflows by using a default configuration - I don't know how Java behaves in this case.
That's why I'd prefer another default value than -1 if none is configured (no matter if it's 8192 or 1024^2). But I think it should be a value which doesn't causes the application to fail when run with default settings and using large resource files.
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Jörg Heinicke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jörg Heinicke updated COCOON-2168:
----------------------------------
Affects version (Component): Parent values: Components: Pipeline(10157). Level 1 values: 1.0.0-RC1(10224).
Fix version (Component): Parent values: Components: Pipeline(10228).
Other Info: (was: [Patch available])
Affects Version/s: 2.1.11
Fix Version/s: 2.2-dev (Current SVN)
2.1.12-dev (Current SVN)
Assignee: Jörg Heinicke
As discussed in http://marc.info/?t=120473411300003&r=1&w=4 I changed the default buffer size used in Cocoon to 1 MB. This will prevent OOMEs with overly large resources in most circumstances when the user has not configured the buffer size. On the other hand it will hardly affect any "normal" Cocoon pipeline rendering since 1 MB for markup is rather large. The buffer size can still be configured to complete buffering by setting it to -1 explicitly (which used to be the default).
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.1.11, 2.2
> Reporter: Felix Knecht
> Assignee: Jörg Heinicke
> Fix For: 2.1.12-dev (Current SVN), 2.2-dev (Current SVN)
>
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Knecht updated COCOON-2168:
---------------------------------
Assignee: (was: Felix Knecht)
Other Info: [Patch available]
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Felix Knecht (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Felix Knecht closed COCOON-2168.
--------------------------------
Resolution: Fixed
It's a configuration problem.
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Jörg Heinicke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603768#action_12603768 ]
Jörg Heinicke commented on COCOON-2168:
---------------------------------------
The allocated memory for the buffer will now be increasing starting from an initial buffer size of 8KB rather than allocating the whole memory for the potentially big buffer (like 1MB) beforehand.
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.1.11, 2.2
> Reporter: Felix Knecht
> Assignee: Jörg Heinicke
> Fix For: 2.1.12-dev (Current SVN), 2.2-dev (Current SVN)
>
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (COCOON-2168) ResourceReader produces Java Heap
Overflow when reading a huge resource
Posted by "Jörg Heinicke (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/COCOON-2168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12575550#action_12575550 ]
Jörg Heinicke commented on COCOON-2168:
---------------------------------------
I've looked into this issue and I'm against your fix. It stands completely against the idea of buffering the whole pipeline content which is done for error handlers. It should be possible for them to reset the OutputStream. Or asked in other words? Why using the non-flushing BufferedOutputStream at all?
Now how to handle the issue with huge resources? Should be quite easy. From what I see from the code (AbstractProcessingPipeline) it is possible to configure and setup/parameterize a pipeline with "outputBufferSize". This means on both map:pipe and map:pipeline it should be possible to set an actual buffer size. Only if none is set (and it defaults to -1) the non-flushing BufferedOutputStream is used.
Root sitemap has this example (in map:components/map:pipes section):
<map:pipe name="noncaching" src="org.apache.cocoon.components.pipeline.impl.NonCachingProcessingPipeline"
pool-max="${noncaching-pipeline.pool-max}">
<!-- parameter name="outputBufferSize" value="8192"/ -->
</map:pipe>
The following should also work (in map:pipelines section):
<map:pipeline>
<map:parameter name="outputBufferSize" value="8192"/>
...
</map:pipeline>
We could argue about another default value than -1 though. Something like 1024^2.
WDYT?
> ResourceReader produces Java Heap Overflow when reading a huge resource
> -----------------------------------------------------------------------
>
> Key: COCOON-2168
> URL: https://issues.apache.org/jira/browse/COCOON-2168
> Project: Cocoon
> Issue Type: Bug
> Components: * Cocoon Core
> Affects Versions: 2.2-dev (Current SVN)
> Reporter: Felix Knecht
> Attachments: ResourceReader.diff, test-case.tar.gz
>
>
> When reading a huge resource (i.e. 700MB file) the ResourceReader produces an overflow due to the BufferedOutputStream which is used (and forced to be used via AbstractReader). The BufferedOutputStream flushes only at the end (or when forced to), but overwrites the flush method to do nothing.
> As I don't know exactly where the BufferedOutputStream is used and what kind of impacts it will have to change it there I'm just going to fix the ResourceReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.