You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fop-users@xmlgraphics.apache.org by Hamed Mohammed <mo...@gmail.com> on 2010/08/25 21:39:34 UTC
Memory Leak issue -- FOP
Hello Users:
Any idea on how to solve the FOP memory leak issue. Below snap shots are
taken from Eclipse Memory Analyzer of a IBM portable heap dump.
**
**
**
**
**
**
--
Thanks,
Hamed Mohammed,
Email: mohdhamedmscse@gmail.com.
Re: AW: Memory Leak issue -- FOP
Posted by Jeremias Maerki <de...@jeremias-maerki.ch>.
On 10.09.2010 10:13:43 Craig Ringer wrote:
> On 09/10/2010 03:44 PM, Georg Datterl wrote:
> > Hi Hamed,
> >
> > I did some pretty large publications with lots of images. 1500 pages
> > took 2GB memory, after I put some effort in memory optimization. The
> > only FOP-related issue I found was image caching and that can be
> > disabled. I’m quite sure I would have found a memory leak in FOP,
> > especially one related to ordinary LayoutManagers. So either make your
> > page-sequences shorter or give fop more memory.
>
> I can't help but wonder if FOP needs to keep the whole page sequence in
> memory, at least for PDF output. Admittedly I haven't verified that it
> *is* keeping everything in RAM, but that's certainly a whole lot of RAM
> for a moderate-sized document.
I doesn't need to keep whole page-sequences (FO tree) in memory in
theory. In practice, it does currently. FOP is still waiting for someone
to tackle that problem which is going to be quite a job.
> I've been meaning to look at how fop is doing its PDF generation for a
> while, but I've been head-down trying to finish a web-based UI for work
> first. I do plan to look at it though as I've done a fair bit of work on
> PDF generation libraries and I'm curious about how Fop is doing it (and
> how much wheel-reinvention might be going on).
>
> Anyway, PDF is *designed* for streaming output, so huge PDFs can be
> produced using only very small amounts of memory with a bit of thought
> into how the output works. I've had no issues generating
> multi-hundred-megabyte PDF documents with very small amounts of RAM
> using PoDoFo, a C++ PDF library that supports direct-to-disk PDF generation.
FOP's PDF library is highly optimized for streaming output and uses very
little RAM. I can easily produce image-heavy photo books with a size of
2GB and as little as 64MB RAM. What's taking a lot of RAM is the FO tree
for which we currently have no means to release subtrees that have been
fully processed. Right now this happens for a page-sequence when it's
fully processed. The layout engine itself also takes some memory. And
finally the area tree, but this one can release single pages once they
are fully resolved and written out. The intermediate format is strictly
streaming and does not use a noticeable amount of memory.
> There are all sorts of tricks you can do. The most important is of
> course that you can make back- or forward- indirect references to almost
> any object, with no constraints on object order in the document. You can
> write whatever you generate out very aggressively. You can even split
> your content stream(s) for each page into multiple segments so you can
> write the content stream out when it gets too big. Or write the content
> stream to a tempfile, then merge it into the PDF after the other
> resources for the page have been written.
Right, these are the work-arounds that need to be made right now.
> There should be no need for image caching, because once you've written
> the image object to the PDF once, you can just reference it again in
> later pages. Not only does that save RAM but it makes your PDF smaller
> and faster. It works even if your image is used in different sizes,
> scales, etc in different parts of the document, because you can crop and
> scale using content-stream instructions.
For one document, an image is only loaded once and written once and
re-used even if there is no image cache. The image cache is there only
to cache images between rendering runs. Then, FOP is optimized to
"preload" images, i.e. it only extracts the intrinsic size of an image
without actually loading and processing it (where possible). Only the
output format will finally load the image fully and process it for
output.
> You don't even have to keep the page dictionaries in RAM. You can write
> them out when the page is done (or before). Because forward-indirect
> references are permitted, if you have content on the page that's yet to
> be generated you can reserve some object IDs for those content streams
> and output indirect references to the as-yet nonexistent content streams
> in the page dictionary.
That already happens to a certain degree.
> About the only time I can think of when you have to keep something in
> memory (or at least, in a tempfile) is when you have content in a page
> (like total page counts) that cannot be generated until later in the
> document - and may re-flow the rest of the page's content. If the
> late-generated content won't force a reflow it can just be put in a
> separate content stream with a forward-reference.
>
> Admittedly, I'm speaking only about the actual PDF generation. It may
> well be that generating the AT/IF is inherently demanding of resident
> RAM, or that the IF/AT don't contain enough information to generate
> pages progressively.
Not at all. It's all the FO tree and the layout engine that can still be
improved.
> The point, though, is that PDF output shouldn't use much RAM if the PDF
> output code is using PDF features to make it efficient. Sometimes it's a
> trade-off between how efficient the produced PDF is and how efficient
> its creation is, but you can always post-process (optimize) a PDF once
> it's generated if you want to do things like linearize it for fast web
> loading.
As you've seen you're suspecting the problems in the wrong part of FOP.
PDF linearization would indeed be impossible right now (without
post-processing) exactly because we are streaming PDF for low memory
consumption.
Jeremias Maerki
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
AW: AW: Memory Leak issue -- FOP
Posted by Georg Datterl <ge...@geneon.de>.
Hi Craig,
Before you start working on that, look at
https://issues.apache.org/bugzilla/show_bug.cgi?id=47314
Ben Wuest did some stuff to start rendering without a finished page sequence.
Regards,
Georg Datterl
------ Kontakt ------
Georg Datterl
Geneon media solutions gmbh
Gutenstetter Straße 8a
90449 Nürnberg
HRB Nürnberg: 17193
Geschäftsführer: Yong-Harry Steiert
Tel.: 0911/36 78 88 - 26
Fax: 0911/36 78 88 - 20
www.geneon.de
Weitere Mitglieder der Willmy MediaGroup:
IRS Integrated Realization Services GmbH: www.irs-nbg.de
Willmy PrintMedia GmbH: www.willmy.de
Willmy Consult & Content GmbH: www.willmycc.de
-----Ursprüngliche Nachricht-----
Von: Craig Ringer [mailto:craig@postnewspapers.com.au]
Gesendet: Freitag, 10. September 2010 10:14
An: fop-users@xmlgraphics.apache.org
Cc: Georg Datterl
Betreff: Re: AW: Memory Leak issue -- FOP
On 09/10/2010 03:44 PM, Georg Datterl wrote:
> Hi Hamed,
>
> I did some pretty large publications with lots of images. 1500 pages
> took 2GB memory, after I put some effort in memory optimization. The
> only FOP-related issue I found was image caching and that can be
> disabled. I'm quite sure I would have found a memory leak in FOP,
> especially one related to ordinary LayoutManagers. So either make your
> page-sequences shorter or give fop more memory.
I can't help but wonder if FOP needs to keep the whole page sequence in
memory, at least for PDF output. Admittedly I haven't verified that it
*is* keeping everything in RAM, but that's certainly a whole lot of RAM
for a moderate-sized document.
I've been meaning to look at how fop is doing its PDF generation for a
while, but I've been head-down trying to finish a web-based UI for work
first. I do plan to look at it though as I've done a fair bit of work on
PDF generation libraries and I'm curious about how Fop is doing it (and
how much wheel-reinvention might be going on).
Anyway, PDF is *designed* for streaming output, so huge PDFs can be
produced using only very small amounts of memory with a bit of thought
into how the output works. I've had no issues generating
multi-hundred-megabyte PDF documents with very small amounts of RAM
using PoDoFo, a C++ PDF library that supports direct-to-disk PDF generation.
There are all sorts of tricks you can do. The most important is of
course that you can make back- or forward- indirect references to almost
any object, with no constraints on object order in the document. You can
write whatever you generate out very aggressively. You can even split
your content stream(s) for each page into multiple segments so you can
write the content stream out when it gets too big. Or write the content
stream to a tempfile, then merge it into the PDF after the other
resources for the page have been written.
There should be no need for image caching, because once you've written
the image object to the PDF once, you can just reference it again in
later pages. Not only does that save RAM but it makes your PDF smaller
and faster. It works even if your image is used in different sizes,
scales, etc in different parts of the document, because you can crop and
scale using content-stream instructions.
You don't even have to keep the page dictionaries in RAM. You can write
them out when the page is done (or before). Because forward-indirect
references are permitted, if you have content on the page that's yet to
be generated you can reserve some object IDs for those content streams
and output indirect references to the as-yet nonexistent content streams
in the page dictionary.
About the only time I can think of when you have to keep something in
memory (or at least, in a tempfile) is when you have content in a page
(like total page counts) that cannot be generated until later in the
document - and may re-flow the rest of the page's content. If the
late-generated content won't force a reflow it can just be put in a
separate content stream with a forward-reference.
Admittedly, I'm speaking only about the actual PDF generation. It may
well be that generating the AT/IF is inherently demanding of resident
RAM, or that the IF/AT don't contain enough information to generate
pages progressively.
The point, though, is that PDF output shouldn't use much RAM if the PDF
output code is using PDF features to make it efficient. Sometimes it's a
trade-off between how efficient the produced PDF is and how efficient
its creation is, but you can always post-process (optimize) a PDF once
it's generated if you want to do things like linearize it for fast web
loading.
--
Craig Ringer
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Re: AW: Memory Leak issue -- FOP
Posted by Craig Ringer <cr...@postnewspapers.com.au>.
On 09/10/2010 03:44 PM, Georg Datterl wrote:
> Hi Hamed,
>
> I did some pretty large publications with lots of images. 1500 pages
> took 2GB memory, after I put some effort in memory optimization. The
> only FOP-related issue I found was image caching and that can be
> disabled. I’m quite sure I would have found a memory leak in FOP,
> especially one related to ordinary LayoutManagers. So either make your
> page-sequences shorter or give fop more memory.
I can't help but wonder if FOP needs to keep the whole page sequence in
memory, at least for PDF output. Admittedly I haven't verified that it
*is* keeping everything in RAM, but that's certainly a whole lot of RAM
for a moderate-sized document.
I've been meaning to look at how fop is doing its PDF generation for a
while, but I've been head-down trying to finish a web-based UI for work
first. I do plan to look at it though as I've done a fair bit of work on
PDF generation libraries and I'm curious about how Fop is doing it (and
how much wheel-reinvention might be going on).
Anyway, PDF is *designed* for streaming output, so huge PDFs can be
produced using only very small amounts of memory with a bit of thought
into how the output works. I've had no issues generating
multi-hundred-megabyte PDF documents with very small amounts of RAM
using PoDoFo, a C++ PDF library that supports direct-to-disk PDF generation.
There are all sorts of tricks you can do. The most important is of
course that you can make back- or forward- indirect references to almost
any object, with no constraints on object order in the document. You can
write whatever you generate out very aggressively. You can even split
your content stream(s) for each page into multiple segments so you can
write the content stream out when it gets too big. Or write the content
stream to a tempfile, then merge it into the PDF after the other
resources for the page have been written.
There should be no need for image caching, because once you've written
the image object to the PDF once, you can just reference it again in
later pages. Not only does that save RAM but it makes your PDF smaller
and faster. It works even if your image is used in different sizes,
scales, etc in different parts of the document, because you can crop and
scale using content-stream instructions.
You don't even have to keep the page dictionaries in RAM. You can write
them out when the page is done (or before). Because forward-indirect
references are permitted, if you have content on the page that's yet to
be generated you can reserve some object IDs for those content streams
and output indirect references to the as-yet nonexistent content streams
in the page dictionary.
About the only time I can think of when you have to keep something in
memory (or at least, in a tempfile) is when you have content in a page
(like total page counts) that cannot be generated until later in the
document - and may re-flow the rest of the page's content. If the
late-generated content won't force a reflow it can just be put in a
separate content stream with a forward-reference.
Admittedly, I'm speaking only about the actual PDF generation. It may
well be that generating the AT/IF is inherently demanding of resident
RAM, or that the IF/AT don't contain enough information to generate
pages progressively.
The point, though, is that PDF output shouldn't use much RAM if the PDF
output code is using PDF features to make it efficient. Sometimes it's a
trade-off between how efficient the produced PDF is and how efficient
its creation is, but you can always post-process (optimize) a PDF once
it's generated if you want to do things like linearize it for fast web
loading.
--
Craig Ringer
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
AW: Memory Leak issue -- FOP
Posted by Georg Datterl <ge...@geneon.de>.
Hi Hamed,
I did some pretty large publications with lots of images. 1500 pages took 2GB memory, after I put some effort in memory optimization. The only FOP-related issue I found was image caching and that can be disabled. I'm quite sure I would have found a memory leak in FOP, especially one related to ordinary LayoutManagers. So either make your page-sequences shorter or give fop more memory.
Regards,
Georg Datterl
------ Kontakt ------
Georg Datterl
Geneon media solutions gmbh
Gutenstetter Straße 8a
90449 Nürnberg
HRB Nürnberg: 17193
Geschäftsführer: Yong-Harry Steiert
Tel.: 0911/36 78 88 - 26
Fax: 0911/36 78 88 - 20
www.geneon.de<http://www.geneon.de>
Weitere Mitglieder der Willmy MediaGroup:
IRS Integrated Realization Services GmbH: www.irs-nbg.de<http://www.irs-nbg.de>
Willmy PrintMedia GmbH: www.willmy.de<http://www.willmy.de>
Willmy Consult & Content GmbH: www.willmycc.de<http://www.willmycc.de>
Von: Hamed Mohammed [mailto:mohdhamedmscse@gmail.com]
Gesendet: Donnerstag, 9. September 2010 19:55
An: fop-users@xmlgraphics.apache.org
Cc: Dominik Stadler; Craig Ringer
Betreff: Re: Memory Leak issue -- FOP
Hi,
Attached word file includes snap shots of exeption and heap dump analysis.
On Wed, Sep 8, 2010 at 7:10 PM, Craig Ringer <cr...@postnewspapers.com.au>> wrote:
On 9/09/2010 1:55 AM, Hamed Mohammed wrote:
Hi,
This the expection I get when I try to convert XSL:FO to PDF
.
Did you try to attach a screenshot or something?
Put the exception text in a text file attachment (to protect it from email client word wrapping) or paste it here:
http://pastebin.com/
... and include the link it give you in your email.
--
Craig Ringer
Tech-related writing at http://soapyfrogs.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org<ma...@xmlgraphics.apache.org>
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org<ma...@xmlgraphics.apache.org>
--
Thanks,
Hamed Mohammed,
Email: mohdhamedmscse@gmail.com<ma...@gmail.com>.
Re: Memory Leak issue -- FOP
Posted by Craig Ringer <cr...@postnewspapers.com.au>.
On 09/10/2010 01:55 AM, Hamed Mohammed wrote:
> Hi,
> Attached word file includes snap shots of exeption and heap dump analysis.
Just a tip for the future: screenshots* are much less useful than text.
Please copy and paste the text of the exception rather than taking
screenshots and pasting the screenshots. You should find that your
profiling tool lets you select the exception text and copy it. Once
you've done that, open Notepad (since you're on Windows) and paste the
text in there. Or just use the pastebin website link I sent you.
The easier you can make it for people to help you, the more effort
they'll put in and the more likely they'll be to try.
In this case, your memory profiling shows that fop is using lots of
memory - but that's pretty meaningless without knowing what you're
feeding to it. Can you supply the XSL-FO document involved? Or a
suitable test-case? If you save the XSL-FO to a plain text file and run
fop on it standalone, does it use the same amount of memory?
It's not necessarily surprising that the FlowLayoutManager is using lots
of memory if it's processing a really huge or complex XSL-FO document.
Seeing the input is necessary to be able to help you.
You also didn't answer the rest of the questions. I quote my earlier mail:
> What's PdfStorage? It doesn't seem to be in fop's packages...
>
> Any chance you can post a self-contained test program that demonstrates
> the problem you describe, along with sample input for it?
>> String xml = builder.getReport(); /// gets the XSL:FO file data approx 50mb
> If you're concerned about memory consumption, writing your XSL-FO to a
> tempfile rather than an in-memory string and passing that to FOP's SAX
> engine would be a big help. Or, for that matter, writing directly to the
> SAX handler's InputStream, progressively generating the document as it's
> required so it never has to exist entirely in memory or on disk.
* Why? If you supply a screenshot image, it can't be searched easily and
is generally annoying to use. It's made even more so by having it inside
a Microsoft Word document - not everyone uses Microsoft Word and besides
its an irritating format to work with. In this case, confusingly, it
looks like the exception trace might be composed of two different
exceptions or omit some content in the middle, which isn't helpful.
Also, Google can't index screenshots so others won't be able to able to
find your question and any answers to it as easily when looking for help
for the same problem later.
--
Craig Ringer
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Re: Memory Leak issue -- FOP
Posted by Hamed Mohammed <mo...@gmail.com>.
Hi,
Attached word file includes snap shots of exeption and heap dump analysis.
On Wed, Sep 8, 2010 at 7:10 PM, Craig Ringer <cr...@postnewspapers.com.au>wrote:
> On 9/09/2010 1:55 AM, Hamed Mohammed wrote:
>
>> Hi,
>>
>> This the expection I get when I try to convert XSL:FO to PDF
>>
>>
>> .
>>
>
> Did you try to attach a screenshot or something?
>
> Put the exception text in a text file attachment (to protect it from email
> client word wrapping) or paste it here:
>
> http://pastebin.com/
>
> ... and include the link it give you in your email.
>
> --
> Craig Ringer
>
> Tech-related writing at http://soapyfrogs.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
> For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
>
>
--
Thanks,
Hamed Mohammed,
Email: mohdhamedmscse@gmail.com.
Re: Memory Leak issue -- FOP
Posted by Craig Ringer <cr...@postnewspapers.com.au>.
On 9/09/2010 1:55 AM, Hamed Mohammed wrote:
> Hi,
>
> This the expection I get when I try to convert XSL:FO to PDF
>
>
> .
Did you try to attach a screenshot or something?
Put the exception text in a text file attachment (to protect it from
email client word wrapping) or paste it here:
http://pastebin.com/
... and include the link it give you in your email.
--
Craig Ringer
Tech-related writing at http://soapyfrogs.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Re: Memory Leak issue -- FOP
Posted by Hamed Mohammed <mo...@gmail.com>.
Hi,
This the expection I get when I try to convert XSL:FO to PDF
.
The code for creation of PDF used :
*public* *void* generatePDF(String foXml)
*throws* XMLProcessingException, SAXException
{
FOUserAgent userAgent = fopFactory.newFOUserAgent();
*try
*
{
Fop processor =
fopFactory.newFop(MimeConstants.*MIME_PDF*, userAgent,
pdfStorage.getOutputHolder());
TransformerFactory factory = TransformerFactory.*newInstance*();
Transformer transformer = factory.newTransformer(); // identity transformer
//Source src = new StreamSource(new StringReader(foXml));
Result res = *new* SAXResult(processor.getDefaultHandler());
SAXSource source = *new* SAXSource(*new* InputSource(*new*StringReader(foXml)));
source.setXMLReader(XMLReaderFactory.*createXMLReader*());
transformer.transform(source, res);
}catch....
.........
...............
I have used eclipse memory analyzer tool to analyze the heap dumps created
as a result of the execption above.
On Wed, Sep 1, 2010 at 12:53 AM, Craig Ringer
<cr...@postnewspapers.com.au>wrote:
> On 30/08/10 22:29, Hamed Mohammed wrote:
> > Hi,
> >
> > This is how I am using FOP to process XSL.FO <http://xsl.fo/> <
> http://XSL.FO <http://xsl.fo/>> into PDF. I
> > agree that the issue is to do with too much of memory consumption. Is
> > there a better way to process this large FO file into PDF using FOP?
>
> How much memory is too much? Have you used the JVM's tools to find out
> *where* the memory is going?
>
> If you don't know Java's profiling tools well, you might want to try the
> rather user-friendly memory profiler in NetBeans. NetBeans opens Maven
> projects directly, and can be set up for a generic Ant project quite
> quickly, so if you're not already using it there's no reason not to just
> for this.
>
> > String xml = builder.getReport(); /// gets the XSL:FO file data approx
> 50mb
>
> If you're concerned about memory consumption, writing your XSL-FO to a
> tempfile rather than an in-memory string and passing that to FOP's SAX
> engine would be a big help. Or, for that matter, writing directly to the
> SAX handler's InputStream, progressively generating the document as it's
> required so it never has to exist entirely in memory or on disk.
>
> > PdfStorage pdfHolder = pdfStorageFactory.createStorage(reportRequest,
>
> What's PdfStorage? It doesn't seem to be in fop's packages...
>
> Any chance you can post a self-contained test program that demonstrates
> the problem you describe, along with sample input for it?
>
> --
> Craig Ringer
>
> Tech-related writing: http://soapyfrogs.blogspot.com/
>
--
Thanks,
Hamed Mohammed,
Email: mohdhamedmscse@gmail.com.
Re: Memory Leak issue -- FOP
Posted by Craig Ringer <cr...@postnewspapers.com.au>.
On 30/08/10 22:29, Hamed Mohammed wrote:
> Hi,
>
> This is how I am using FOP to process XSL.FO <http://XSL.FO> into PDF. I
> agree that the issue is to do with too much of memory consumption. Is
> there a better way to process this large FO file into PDF using FOP?
How much memory is too much? Have you used the JVM's tools to find out
*where* the memory is going?
If you don't know Java's profiling tools well, you might want to try the
rather user-friendly memory profiler in NetBeans. NetBeans opens Maven
projects directly, and can be set up for a generic Ant project quite
quickly, so if you're not already using it there's no reason not to just
for this.
> String xml = builder.getReport(); /// gets the XSL:FO file data approx 50mb
If you're concerned about memory consumption, writing your XSL-FO to a
tempfile rather than an in-memory string and passing that to FOP's SAX
engine would be a big help. Or, for that matter, writing directly to the
SAX handler's InputStream, progressively generating the document as it's
required so it never has to exist entirely in memory or on disk.
> PdfStorage pdfHolder = pdfStorageFactory.createStorage(reportRequest,
What's PdfStorage? It doesn't seem to be in fop's packages...
Any chance you can post a self-contained test program that demonstrates
the problem you describe, along with sample input for it?
--
Craig Ringer
Tech-related writing: http://soapyfrogs.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Re: Memory Leak issue -- FOP
Posted by Hamed Mohammed <mo...@gmail.com>.
Hi,
This is how I am using FOP to process XSL.FO into PDF. I agree that
the issue is to do with too much of memory consumption. Is there a better
way to process this large FO file into PDF using FOP?
String xml = builder.getReport(); /// gets the XSL:FO file data approx 50mb
PdfStorage pdfHolder = pdfStorageFactory.createStorage(reportRequest,
contentType);
pdfMaker.generatePDF(pdfHolder, xml); // Instance of PdfCreator injected
through Spring.
On Mon, Aug 30, 2010 at 1:07 AM, Dominik Stadler <
dominik.stadler@dynatrace.com> wrote:
>
> Hi,
>
> Craig is right, it's it's not a "memory leak", but rather memory kept
> active too long, i.e. after processing is done. I saw the same thing when I
> integrated FOP into an application over here.
>
> It is basically caused by the SAX implementation that you use as it caches
> the SAX-Handler, where the FOP-SAX-Handler keeps references to a lot of
> internal stuff. In my case it did this in a ThreadLocal, so with different
> threads using FOP I ended up having this memory kept in every thread, which
> accumulated to quite an amount of memory depending on the usage pattern.
>
> I tried various different ways to solve this, when using SAX/XML provided
> as part of Sun JDK 1.6 the following was removing this memory bloat:
>
> Result result = new SAXResult(fop.getDefaultHandler());
> SAXSource source = new SAXSource(inputSource);
> // set our own XMLReader to prevent Sun XML from caching
> stuff in ThreadLocals which keeps FOP in memory
> // we want to free up any FOP resources when not reporting
> and for this ThreadLocals is very bad, especially
> // as we are called in different threads via REST and
> Scheduling thread pools!
> source.setXMLReader(XMLReaderFactory.createXMLReader());
>
> transformer.transform(source, result);
>
> if (logger.isLoggable(Level.FINE) && fop.getResults() !=
> null) {
> logger.info("Processing resulted in " +
> fop.getResults().getPageCount() + " pages."); //$NON-NLS-1$ //$NON-NLS-2$
> }
> }
>
> Maybe a similar thing solves this for you as well.
>
> Dominik.
>
>
> -----Original Message-----
> From: Craig Ringer [mailto:craig@postnewspapers.com.au]
> Sent: Donnerstag, 26. August 2010 02:35
> To: fop-users@xmlgraphics.apache.org
> Cc: Hamed Mohammed
> Subject: Re: Memory Leak issue -- FOP
>
> On 26/08/2010 3:39 AM, Hamed Mohammed wrote:
> > Hello Users:
> > Any idea on how to solve the FOP memory leak issue. Below snap shots are
> > taken from Eclipse Memory Analyzer of a IBM portable heap dump.
>
> What makes you think that's a memory leak?
>
> It's a large amount of memory consumption, sure. But is there any
> evidence it's growing over time? Or growing whenever a new document is
> processed, without shinking again when the document is finished?
>
> How do you know it's not your own code keeping references to the fop
> instances around? Can you produce a small, self-contained program that
> demonstrates the leak?
>
> If your real question is "why is FOP using so much memory" ... well, to
> answer that it'd be necessary to know more about your XSL-FO input and
> the associated resources like graphics.
>
> --
> Craig Ringer
>
> Tech-related writing at http://soapyfrogs.blogspot.com/
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
> For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
>
>
>
>
--
Thanks,
Hamed Mohammed,
Email: mohdhamedmscse@gmail.com.
RE: Memory Leak issue -- FOP
Posted by Dominik Stadler <do...@dynatrace.com>.
Hi,
Craig is right, it's it's not a "memory leak", but rather memory kept active too long, i.e. after processing is done. I saw the same thing when I integrated FOP into an application over here.
It is basically caused by the SAX implementation that you use as it caches the SAX-Handler, where the FOP-SAX-Handler keeps references to a lot of internal stuff. In my case it did this in a ThreadLocal, so with different threads using FOP I ended up having this memory kept in every thread, which accumulated to quite an amount of memory depending on the usage pattern.
I tried various different ways to solve this, when using SAX/XML provided as part of Sun JDK 1.6 the following was removing this memory bloat:
Result result = new SAXResult(fop.getDefaultHandler());
SAXSource source = new SAXSource(inputSource);
// set our own XMLReader to prevent Sun XML from caching stuff in ThreadLocals which keeps FOP in memory
// we want to free up any FOP resources when not reporting and for this ThreadLocals is very bad, especially
// as we are called in different threads via REST and Scheduling thread pools!
source.setXMLReader(XMLReaderFactory.createXMLReader());
transformer.transform(source, result);
if (logger.isLoggable(Level.FINE) && fop.getResults() != null) {
logger.info("Processing resulted in " + fop.getResults().getPageCount() + " pages."); //$NON-NLS-1$ //$NON-NLS-2$
}
}
Maybe a similar thing solves this for you as well.
Dominik.
-----Original Message-----
From: Craig Ringer [mailto:craig@postnewspapers.com.au]
Sent: Donnerstag, 26. August 2010 02:35
To: fop-users@xmlgraphics.apache.org
Cc: Hamed Mohammed
Subject: Re: Memory Leak issue -- FOP
On 26/08/2010 3:39 AM, Hamed Mohammed wrote:
> Hello Users:
> Any idea on how to solve the FOP memory leak issue. Below snap shots are
> taken from Eclipse Memory Analyzer of a IBM portable heap dump.
What makes you think that's a memory leak?
It's a large amount of memory consumption, sure. But is there any
evidence it's growing over time? Or growing whenever a new document is
processed, without shinking again when the document is finished?
How do you know it's not your own code keeping references to the fop
instances around? Can you produce a small, self-contained program that
demonstrates the leak?
If your real question is "why is FOP using so much memory" ... well, to
answer that it'd be necessary to know more about your XSL-FO input and
the associated resources like graphics.
--
Craig Ringer
Tech-related writing at http://soapyfrogs.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org
Re: Memory Leak issue -- FOP
Posted by Craig Ringer <cr...@postnewspapers.com.au>.
On 26/08/2010 3:39 AM, Hamed Mohammed wrote:
> Hello Users:
> Any idea on how to solve the FOP memory leak issue. Below snap shots are
> taken from Eclipse Memory Analyzer of a IBM portable heap dump.
What makes you think that's a memory leak?
It's a large amount of memory consumption, sure. But is there any
evidence it's growing over time? Or growing whenever a new document is
processed, without shinking again when the document is finished?
How do you know it's not your own code keeping references to the fop
instances around? Can you produce a small, self-contained program that
demonstrates the leak?
If your real question is "why is FOP using so much memory" ... well, to
answer that it'd be necessary to know more about your XSL-FO input and
the associated resources like graphics.
--
Craig Ringer
Tech-related writing at http://soapyfrogs.blogspot.com/
---------------------------------------------------------------------
To unsubscribe, e-mail: fop-users-unsubscribe@xmlgraphics.apache.org
For additional commands, e-mail: fop-users-help@xmlgraphics.apache.org