JPEG2000 - RE: Fop Memory Use

["Marquart, Joshua D" <jo...@firstdata.com>]

While somewhat unrelated, I figure this is the best thread in which to
note that there is a known memory leak within JAI when writing handling
JPEG2000 images that a third party has fixed in March and documented it
here:

 

http://www.jpedal.org/PDFblog/2011/03/java-jai-image-io-jpeg2000-memory-
leak-fix/

http://www.jpedal.org/PDFblog/2011/04/jai-memory-leak-source-changes/

 

They have been unable to commit their change to java-net since the move
to Oracle.

 

-Josh

 

 

From: Eric Douglas [mailto:edouglas@blockhouse.com] 
Sent: Wednesday, May 18, 2011 1:43 PM
To: fop-dev@xmlgraphics.apache.org
Subject: RE: Fop Memory Use

 

This test run isn't using SVG at all.  The PDFRenderer is working, the
PNGRenderer runs out of memory, so it is using images but as output.

I already broke up the input to multiple FOs with multiple calls to the
transform to generate a large document by combining small documents
using the initial-page-number.

As the program runs it just keeps increasing memory use.

I tried running a profile with Java's VisualVM though I'm not sure what
exactly I'm looking at or what to do with it.

The number one item showing memory hog in the profiler, as of my last
snapshot was: class int[], live bytes 23,273,952 B, live objects 382,
generations 10

After the program crashed with the profiler running I had an additional
file opened, Java2DRenderer.class, so I'm assuming it's doing something
that breaks PNGRenderer.

My class doesn't have any int[] references.

After that first reference the sizes drop off sharply in the profiler.
The next class reference is char[], then
org.apache.fop.area.inline.SpaceArea.

 

 

________________________________

From: Peter Hancock [mailto:peter.hancock@gmail.com] 
Sent: Wednesday, May 18, 2011 12:05 PM
To: fop-dev@xmlgraphics.apache.org
Subject: Re: Fop Memory Use

Hi Eric,

Does your document contain many large SVG's?  If so take a look at
Bugzilla #46360.  This issue was resolved in rev 997602 of FOP trunk.

Pete





On Wed, May 18, 2011 at 5:10 PM, Adrian Cumiskey
<ad...@gmail.com> wrote:

Hi Eric,

 

Fop calculates layout in page sequence chunks, so try breaking up your
pages into chunks of page sequences.  Pages should be available for
garbage collection once the page sequence has been rendered.


Cheers, Adrian.


On May 18, 2011, at 7:24 AM, Michael Rubin <mr...@thunderhead.com>
wrote:

	Just a wild thought. But is there a way you could possibly get
the JVM to garbage collect between each run? Maybe that might free the
memory up?
	
	Thanks.
	
	-Mike
	
	On 18/05/11 13:20, Eric Douglas wrote: 

		I am using Fop 1.0. 
		I tried using Fop to transform a single document.  When
I got a little over 100 pages my FO file was over 5 MB.  The transform
crashed with a Java heap out of memory error.

		I managed to break the input down, as I'm using embedded
code generating the input programmatically, and the PDF output is a lot
smaller.

		So I'm currently transforming 10 pages at a time,
setting the initial-page-number to the next sequence (1, 11, 21, etc).

		Then I save all the generated PDFs in memory and merge
them using pdfbox.  So far this is working great. 

		I tried to do the same thing with the PNGRenderer, just
calling a method to transform 10 pages at a time and save the output
images in an array.

		The PNGRenderer is created locally in the method.  It
should be getting released when the method ends but the java process
never releases any memory.

		I tested a 90 page report and the memory use was over 1
GB.  I tested on another machine where the memory limit is apparently
lower and it crashed on page 24.

		Everything about the method to render to PNG is the same
as the method to render to PDF aside from the Renderer. 
		Is there a problem with this renderer or something I
could need to do different? 

	 

	 

				
Michael Rubin

Developer

Thunderhead Logo

Error! Filename not specified.

Error! Filename not specified.

 T

 F

 M

 E

 W

+44 20 8238 7400 

+44 20 8238 7401 

 

mrubin@Thunderhead.com 

www.thunderhead.com <http://www.thunderhead.com> 

 

Thunderhead featured in The Sunday Times Profit Track 100 league table
of companies with fastest-growing profits. Click here
<http://www.fasttrack.co.uk/fasttrack/press/pt11-lon.pdf>  to read more.


Error! Filename not specified.
<http://www.linkedin.com/companies/25033/Thunderhead> Error! Filename
not specified. <http://twitter.com/Thunderheadon> Error! Filename not
specified. <http://www.thunderhead.com/rss/rss.php> Error! Filename not
specified. <http://www.youtube.com/user/ThunderheadOn> Error! Filename
not specified. <http://thunderheadinnovate.wordpress.com/> Error!
Filename not specified. <http://thunderhead.com/about/careers.php> 

The contents of this e-mail are intended for the named addressee only.
It contains information that may be confidential. Unless you are the
named addressee or an authorized designee, you may not copy or use it,
or disclose it to anyone else. If you received it in error please notify
us immediately and then destroy it.

 




-----------------------------------------
The information in this message may be proprietary and/or
confidential, and protected from disclosure.  If the reader of this
message is not the intended recipient, or an employee or agent
responsible for delivering this message to the intended recipient,
you are hereby notified that any dissemination, distribution or
copying of this communication is strictly prohibited. If you have
received this communication in error, please notify First Data
immediately by replying to this message and deleting it from your
computer.