You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fop-dev@xmlgraphics.apache.org by bu...@apache.org on 2008/02/05 19:45:01 UTC
DO NOT REPLY [Bug 44358] New: - OufOfMem
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
Summary: OufOfMem
Product: Fop
Version: 0.93
Platform: All
OS/Version: All
Status: NEW
Severity: major
Priority: P1
Component: pdf
AssignedTo: fop-dev@xmlgraphics.apache.org
ReportedBy: mihail@us.ibm.com
When testing fop.war application, we run out of memory while generating large
pdf - 560 pages. Environment - WAS 5.1.1.13 JRE 1.4.2 (SR9). After the first
run we observe that large portion of the memory does not get GC. The heap
analisys shows the following hirarchy holds 64% of the mem
PrimaryGridUnit
TableCellLayoutManager
TableLayoutManager
TableContentLayout
TableRowIterator
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From adelmelle@apache.org 2008-02-14 15:12 -------
Small update:
I've been browsing around, and may have found the possible cause of the arrays not being collected.
Theoretically, it is possible that an implementation of a tracing GC algorithm would still view the arrays
as reachable if their first element is still strongly reachable... Since this is the default absolute-position
property, and it is possibly referenced by a significant amount of FObjs.
Maybe you could try something like:
* add an empty protected cleanup() method to org.apache.fop.fo.PropertyList
* add an override for this method to StaticPropertyList, and explicitly null out the first element of the
member arrays
* in FOTreeBuilder.MainFOHandler.endElement(), inside the if-block right after
currentFObj.endOfNode(), add currentPropertyList.cleanup() as a first line
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From vincent.hennebert@anyware-tech.com 2008-02-07 03:23 -------
(In reply to comment #5)
> Thanks for the response.
>
> We are using preserve-whitespace property in our stylesheets. That could be
> causing the infinite loop. We don't want to go to production with Trunk code.
> But, if you could please point out to the classes from the Trunk code that fixed
> this problem, I can get those and include them in the 0.94 jar and run our
> stylesheets. Let us know.
Edit the src/java/org/apache/fop/layoutmgr/table/TableContentLayoutManager.java
file and delete lines 145 and 166, which correspond to the following lines of code:
ElementListUtils.removeLegalBreaks(this.headerList);
ElementListUtils.removeLegalBreaks(this.footerList);
They are causing the infinite loop. Removing them is totally safe and won't
affect the quality of the layout.
Vincent
>
> (In reply to comment #4)
> > 1) There was a bug in the Layout code that meant an infinite loop could be
> > entered for certain FO. IIRC, preserve-whitespace property had something to do
> > with it. This has been fixed in the Trunk code.
> > 2) long page-sequences are a problem for FOP because FOP uses a total-fit
> > algorithm to layout the FO. This has the advantage of achieving more elegant
> > layout but the disadvantage of requiring the whole page-sequence to be kept in
> > memory and it is only released when the whole page-sequence has been finished.
> > I think forward references via fo:page-number-cititation can also cause FOP to
> > hang onto more objects in memory until the forward reference is resolved.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From adelmelle@apache.org 2008-02-13 11:11 -------
(In reply to comment #7)
>
> We modified FOP 0.94 as Vincent suggested (thanks!). This resolved the infinite
> loop problem. However we still get the OutOfMemory exception.
OK, thanks a lot for trying this out. Can you judge whether the exception occurred sooner or later than
with FOP 0.93?
> I am wondering if the FOP framework holds large amount of memory in static class variables.
Not that I'm aware of. There are some static variables in the property classes, but those only serve to
reduce the footprint. (caches that are shared between different FOs in the same document, or even in
different documents if they are processed concurrently in the same JVM)
> Also I noticed that there are a very large number of fop.fo.StaticPropertyList
> objects created - 35000
That means your FO contains 35000 objects, which is not abnormal for larger documents. If those are
all inside the same page-sequence, there is only little you can do for the moment, apart from making
sure that such an FO document is never generated in the first place. This could be done by
restructuring the stylesheet to generate multiple page-sequences, but we realize that this is not always
possible.
In the old days, those PropertyLists were never released. Being attached to the corresponding FONode
(hard member reference), they were only released when that FONode was no longer referenced.
Currently, they are more like a window, from which the relevant properties are transferred to the
FONode during parsing. As soon as the endElement() event occurs for that FONode, the PropertyList
goes out of scope and should theoretically be 100% garbage-collectable (including the backing arrays)
> ; each of those objects holds two arrays - size 252.
Yep. 252 is the total number of possible properties.
> This correlates with the heapdump analyses we performed - large number of
> arrays without parent and the GC cannot dispose them. (I am under the
> impression that these objects get allocated in a recursive manner.)
Correct, although the number of StaticPropertyLists to which there exists a hard reference will be
determined not by the number of FOs, but by the nesting level. If you have a document with a
maximum nesting depth of 10 nodes, then there will be at most 10 StaticPropertyList instances alive at
any given point during the processing. I've literally seen this with my own eyes during a profiling
session.
Is it normal that the backing arrays are not collected? I'd think not, but I'm not 100% sure.
Which JVM is used? Are you using Sun's implementation, or an IBM JVM? Is there a way to rule out the
possibility of the GC algorithm being at fault here? Can you try other Java Runtimes?
> Would it be possible to reduce the number of arrays using static var or object
> cache?
Not really, I think... The properties themselves are already cached for a large part, i.e. a simple
FixedLength with a value of "10pt" will be the same instance for all occurrences in the document.
Initially, each FixedLength is a separate instance, but we check immediately whether we already have
one cached with the same value. If that is the case, then the separate instance exists purely on the
stack, and is substituted with the cached instance before it is attached/bound to the FONode.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From bowditch_chris@hotmail.com 2008-02-06 01:11 -------
1) There was a bug in the Layout code that meant an infinite loop could be
entered for certain FO. IIRC, preserve-whitespace property had something to do
with it. This has been fixed in the Trunk code.
2) long page-sequences are a problem for FOP because FOP uses a total-fit
algorithm to layout the FO. This has the advantage of achieving more elegant
layout but the disadvantage of requiring the whole page-sequence to be kept in
memory and it is only released when the whole page-sequence has been finished.
I think forward references via fo:page-number-cititation can also cause FOP to
hang onto more objects in memory until the forward reference is resolved.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From mihail@us.ibm.com 2008-02-05 11:11 -------
Thanks for the suggestion.
We tried fop 0.94 with the xml/xsl that we use to test 0.93 -- unfirtunatelly
we could not get it working. It appears to be sitting in an endless loop and
does not produce any output pdf.
Could you please advise what is the status of the FOP Trunk code. Thanks!
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From mihail@us.ibm.com 2008-02-05 12:28 -------
A few more comments with the hope they will clarify the situation. Any help
and/or suggestions will be highly appreciated.
1) We have tried FOP 0.94. The stylesheet that runs fine with 0.93 runs into an
infinite loop with 0.94. I'm not sure why this happens, but when I searched on
google, I found that other users have run into this problem as well. I haven't
been able to find a solution to this.
2) The FO stylesheet does have a page sequence with a table that can span a lot
of pages based on the data.
Could please let us know why this is a problem? If this is a known problem, can
you let us know if there are any work arounds or fixes to get by this? I would
appreciate if you can send us some examples with alternate implementations so
we can try that out.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From mihail@us.ibm.com 2008-02-06 09:18 -------
Thanks for the response.
We are using preserve-whitespace property in our stylesheets. That could be
causing the infinite loop. We don't want to go to production with Trunk code.
But, if you could please point out to the classes from the Trunk code that fixed
this problem, I can get those and include them in the 0.94 jar and run our
stylesheets. Let us know.
(In reply to comment #4)
> 1) There was a bug in the Layout code that meant an infinite loop could be
> entered for certain FO. IIRC, preserve-whitespace property had something to do
> with it. This has been fixed in the Trunk code.
> 2) long page-sequences are a problem for FOP because FOP uses a total-fit
> algorithm to layout the FO. This has the advantage of achieving more elegant
> layout but the disadvantage of requiring the whole page-sequence to be kept in
> memory and it is only released when the whole page-sequence has been finished.
> I think forward references via fo:page-number-cititation can also cause FOP to
> hang onto more objects in memory until the forward reference is resolved.
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From mihail@us.ibm.com 2008-02-13 07:19 -------
Hi all:
We modified FOP 0.94 as Vincent suggested (thanks!). This resolved the infinite
loop problem. However we still get the OutOfMemory exception. I am wondering
if the FOP framework holds large amount of memory in static class variables.
Also I noticed that there are a very large number of fop.fo.StaticPropertyList
objects created - 35000; each of those objects holds two arrays - size 252.
This correlates with the heapdump analyses we performed - large number of
arrays without parent and the GC cannot dispose them. (I am under the
impression that these objects get allocated in a recursive manner.)
Would it be possible to reduce the number of arrays using static var or object
cache?
Thank you all for the prompt response!
Regards
Mihail
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
DO NOT REPLY [Bug 44358] - OufOfMem
Posted by bu...@apache.org.
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG�
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
<http://issues.apache.org/bugzilla/show_bug.cgi?id=44358>.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND�
INSERTED IN THE BUG DATABASE.
http://issues.apache.org/bugzilla/show_bug.cgi?id=44358
------- Additional Comments From adelmelle@apache.org 2008-02-05 10:55 -------
Are you at liberty to try out more recent versions of FOP (0.94 or even FOP Trunk), and see if that gets you
any further? Some enhancements have been made, and the table-related code in FOP Trunk has changed
a lot. It would be much appreciated if you could report back on that.
Apart from that: Is it possible to alter the structure of the input FO? From what you're telling, it seems like
your FO document consists of one gigantic table (or at least contains a page-sequence with a table that
spans a lot of pages). Correct? If so, you may be out of luck, but that depends on whether you really need
to have the table completely in one page-sequence...
--
Configure bugmail: http://issues.apache.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.