You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@maven.apache.org by "Andrew Gaul (JIRA)" <ji...@codehaus.org> on 2012/12/17 08:43:13 UTC
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Andrew Gaul created SUREFIRE-938:
------------------------------------
Summary: out of memory when logging more messages than heap size
Key: SUREFIRE-938
URL: https://jira.codehaus.org/browse/SUREFIRE-938
Project: Maven Surefire
Issue Type: Bug
Components: Maven Surefire Plugin
Affects Versions: 2.12.4
Reporter: Andrew Gaul
Observed OOM when logging more messages than heap size:
Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOfRange(Arrays.java:3209)
at java.lang.String.<init>(String.java:215)
at java.lang.StringBuilder.toString(StringBuilder.java:430)
at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315918#comment-315918 ]
Kristian Rosenvold edited comment on SUREFIRE-938 at 12/19/12 2:51 AM:
-----------------------------------------------------------------------
We could use something like http://commons.apache.org/io/api-release/org/apache/commons/io/output/DeferredFileOutputStream.html or FileBackedOutputStream for storing stderr
was (Author: krosenvold):
We could use something like FileBackedOutputStream for storing stderr
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315872#comment-315872 ]
Kristian Rosenvold commented on SUREFIRE-938:
---------------------------------------------
This problem has several facets to it; one is that we actually keep stdout/stderr from the process in memory on the plugin side. So technically increasing maven memory through MAVEN_OPTS should get you some way along the line (i.e. export MAVEN_OPTS=-Xmx1024m)
The reason "must" keep the output in memory is because we use Xpp3Dom to write the XML reports, which basically keeps it all in memory. So for this issue to be "truly" fixed we'd need to stream the content directly to disk for the xml reports. A second solution could be to simply not buffer in-memory if we don't write xml, which would remove the need for the buffered content.
But on the other hand, the stack-trace in question shows that we actually keep the entire output in memory twice, so we can most certainly improve on this situation by only storing it once ;)
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315919#comment-315919 ]
Kristian Rosenvold commented on SUREFIRE-938:
---------------------------------------------
There already is a unit test for the statelessxmlreporter, so it should mostly be an issue of improving that and changing the impl
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315918#comment-315918 ]
Kristian Rosenvold commented on SUREFIRE-938:
---------------------------------------------
We could use something like FileBackedOutputStream for storing stderr
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Kristian Rosenvold (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315901#comment-315901 ]
Kristian Rosenvold commented on SUREFIRE-938:
---------------------------------------------
I'm afraid the pull request doesn't entirely do it, since most of the memory related issues come from the bytBuffer.toString, which is where the nasty memory usage lies.
The real problem is making the StatelessXmlReporter use a streaming xml api so it can immediately stream the stdout to file and keep stderr in a tempfile.
It should be possible to make a simple unit test for the xml reporter while converting it to sax (it is tested extremely well through the IT's).
If you want to have a shot at doing that, I can handle all the other changes that would need to be done (if you want to make it use Sax and works with a unit test I can fix the rest).
The pull request has no real effect, please close it.
btw; 2.13 is about 36 hours from now if you want it in there ;)
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] (SUREFIRE-938) out of memory when logging more messages than
heap size
Posted by "Andrew Gaul (JIRA)" <ji...@codehaus.org>.
[ https://jira.codehaus.org/browse/SUREFIRE-938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=315893#comment-315893 ]
Andrew Gaul commented on SUREFIRE-938:
--------------------------------------
Kristian, I opened a pull request which avoids one of the copies:
https://github.com/apache/maven-surefire/pull/16
I do not know enough about Surefire to tackle using a file-backed structure at this point.
> out of memory when logging more messages than heap size
> -------------------------------------------------------
>
> Key: SUREFIRE-938
> URL: https://jira.codehaus.org/browse/SUREFIRE-938
> Project: Maven Surefire
> Issue Type: Bug
> Components: Maven Surefire Plugin
> Affects Versions: 2.12.4
> Reporter: Andrew Gaul
>
> Observed OOM when logging more messages than heap size:
> Exception in thread "ThreadedStreamConsumer" java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOfRange(Arrays.java:3209)
> at java.lang.String.<init>(String.java:215)
> at java.lang.StringBuilder.toString(StringBuilder.java:430)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.getAsString(TestSetRunListener.java:225)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.wrap(TestSetRunListener.java:230)
> at org.apache.maven.plugin.surefire.report.TestSetRunListener.testSucceeded(TestSetRunListener.java:161)
> at org.apache.maven.plugin.surefire.booterclient.output.ForkClient.consumeLine(ForkClient.java:100)
> at org.apache.maven.plugin.surefire.booterclient.output.ThreadedStreamConsumer$Pumper.run(ThreadedStreamConsumer.java:67)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://jira.codehaus.org/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira