You are viewing a plain text version of this content. The canonical link for it is here.
Posted to fop-dev@xmlgraphics.apache.org by John Austin <jw...@sympatico.ca> on 2003/11/21 02:32:01 UTC
Memory measurement -- importance of Driver.reset()
After reading the Sept 2003 thread about Memory Performance, leaks (and
how wonderful ADA is), I modified my test program that generates
3 PDF files. The program now sleeps 30 seconds, calls Driver.reset(),
nulls the reference and sleeps again. In JMP this plots a square wave
between that you can read on the attached graphs.
It is clear that there is a fair bit of memory freed by Driver.reset().
After thinking it over, I modified the same test to skip reset() and
just null the reference and issue System.gc().
This should be the same as letting it go out of scope (which happens
afterwards but this way I get the square wave on the graph).
Guess what ?
Attachment 2: footprint2.png has about 1Mb more heap in use!
And this is a very short test file with just one member name & address.
The test prints a letter, envelope and a renewal form for a non-profit
Gardening group.
The difference ... no call to Driver.reset() !!!
Why ? Does this suggest that there are finalizers (destructors) that are
not being called ? References set to null inside reset() should all
be unreachable when the reference to Driver goes out of scope.
This might explain problems people are reporting when
generating multiple PDF files using FOP. Especially if their
programs don't lose references to instances of Driver.
Personally, I suspect there are a lot of logical memory leaks
inside FOP. A reset() at the end of using a Driver instance is a
catch-all way of releasing all of the logically leaked memory
allocated from inside Driver() (and therefore inside FOP).
This approach is of little help to the developer who builds an
application that dies of memory exhaustion in production. We will have
to fix the logical leaks inside FOP to improve the user experience.
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset()
Posted by "J.Pietschmann" <j3...@yahoo.de>.
John Austin wrote:
> Why bother implementing reset() ?
Reusing the driver keeps the element mappings and perhaps some
other settings which are expensive to build (building element
settings involves reading service files from the jar and populating
some *big* has tables).
J.Pietschmann
RE: Memory measurement -- importance of Driver.reset()
Posted by Victor Mote <vi...@outfitr.com>.
John Austin wrote:
> I still object to the extreme high water marks in FOP. The saw-tooth
> pattern is described in the JMP documentation as characteristic of
> the creation of too many objects. This is probably fundamental to
> XSL-FO but the high-water mark of memory use would be lower if objects
> were collected sooner. This would also speed up collection as there
> would be fewer objects to inspect.
Actually, there are some places where I think we can significantly reduce
the number of objects *created*, specifically: 1) properties, and 2) fonts.
Most properties are, I think, easily represented by primitive data types,
and I hope that we will eventually use them, where possible, instead of the
current scheme. With fonts, I think we are currently creating an object for
each FOText object (I may be wrong about this -- I haven't looked in a
while). The plan is that existing objects will be searched & reused where
possible, which should score a large decrease.
I guess I don't know what you mean by "I still object ...". None of the FOP
developers is happy with performance right now. It is just that we are busy
on other problems ATM.
Victor Mote
Re: Memory measurement -- importance of Driver.reset()
Posted by John Austin <jw...@sympatico.ca>.
On Fri, 2003-11-21 at 15:50, J.Pietschmann wrote:
> John Austin wrote:
> > It is clear that there is a fair bit of memory freed by Driver.reset().
> >
> > After thinking it over, I modified the same test to skip reset() and
> > just null the reference and issue System.gc().
> >
> > This should be the same as letting it go out of scope (which happens
> > afterwards but this way I get the square wave on the graph).
> >
> > Attachment 2: footprint2.png has about 1Mb more heap in use!
>
> Shrug. The Driver.reset() is
> _source = null;
> _stream = null;
> _reader = null;
> _treeBuilder.reset();
> and the tree builder's reset in turn is
> currentFObj = null;
> rootFObj = null;
> streamRenderer = null;
> this.errorCount = 0;
Why bother implementing reset() ?
> There are no static variables explicitely freed (there are not much
> static variables in FOP in general). I don't see any difference calling
> reset() can make compared to simply nulling the Driver reference.
>
> > Why ? Does this suggest that there are finalizers (destructors) that are
> > not being called ? References set to null inside reset() should all
> > be unreachable when the reference to Driver goes out of scope.
>
> You realize that gc() doesn't *force* a GC?
> The most reliable way to measure allocated heap space I know off is
> allocating a large byte[][] array, then allocate 1k byte[] or so
> until you run out of memory.
Yes. But the observations were convincing. It is obvious that much
garbage was collected as expected. It was possible that the garbage
collector did not collect ALL of the garbage at that point but I
didn't understand how that could happen. I thought that the GC
would examine every object to determine it's status. This seemed to
imply that you MUST collect all garbage every time.
I overlooked the possibility of multiple storage pools.
> > This might explain problems people are reporting when
> > generating multiple PDF files using FOP. Especially if their
> > programs don't lose references to instances of Driver.
> Well, if they hang on to the Driver object, they are doomed.
>
> > Personally, I suspect there are a lot of logical memory leaks
> > inside FOP.
> What's a logical memory leak?
The only kind you can get in Java. Referenced memory that is never
used again.
In the case of FOP, this includes objects which have been laid out
in the PDF never to be used again. FOP memory use increases >= O(n)
where n is the number of pages being created. This is easy to show.
Massive amounts of memory are released when Driver is reset() or
unreferenced.
I still object to the extreme high water marks in FOP. The saw-tooth
pattern is described in the JMP documentation as characteristic of
the creation of too many objects. This is probably fundamental to
XSL-FO but the high-water mark of memory use would be lower if objects
were collected sooner. This would also speed up collection as there
would be fewer objects to inspect.
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset()
Posted by "J.Pietschmann" <j3...@yahoo.de>.
John Austin wrote:
> It is clear that there is a fair bit of memory freed by Driver.reset().
>
> After thinking it over, I modified the same test to skip reset() and
> just null the reference and issue System.gc().
>
> This should be the same as letting it go out of scope (which happens
> afterwards but this way I get the square wave on the graph).
>
> Attachment 2: footprint2.png has about 1Mb more heap in use!
Shrug. The Driver.reset() is
_source = null;
_stream = null;
_reader = null;
_treeBuilder.reset();
and the tree builder's reset in turn is
currentFObj = null;
rootFObj = null;
streamRenderer = null;
this.errorCount = 0;
There are no static variables explicitely freed (there are not much
static variables in FOP in general). I don't see any difference calling
reset() can make compared to simply nulling the Driver reference.
> Why ? Does this suggest that there are finalizers (destructors) that are
> not being called ? References set to null inside reset() should all
> be unreachable when the reference to Driver goes out of scope.
You realize that gc() doesn't *force* a GC?
The most reliable way to measure allocated heap space I know off is
allocating a large byte[][] array, then allocate 1k byte[] or so
until you run out of memory.
> This might explain problems people are reporting when
> generating multiple PDF files using FOP. Especially if their
> programs don't lose references to instances of Driver.
Well, if they hang on to the Driver object, they are doomed.
> Personally, I suspect there are a lot of logical memory leaks
> inside FOP.
What's a logical memory leak?
J.Pietschmann
Re: Memory measurement -- importance of Driver.reset()
Posted by Ben Galbraith <ju...@galbraiths.org>.
John Austin wrote:
> After reading the Sept 2003 thread about Memory Performance, leaks (and
> how wonderful ADA is),
As one of the perpetrators of that thread, I need to mention that I
never finished telling the story. The last analysis I ran seemed to
indicate that it was my image loading code, not FOP at all, that was the
source of the leak in my case.
In my app, I load CMYK images, convert them to RGB, write them to the
file system, and then run FOP, all within a loop that runs 1-50 times.
When I analyzed the heap, arrays underneath images where the heap hogs,
but instead them being referenced by FOP, as I supposed, they were
referenced by the Advanced Image I/O API.
So -- for my part I suspect my allegations of FOP memory leakage were
quite false. As soon as I get some time, I'll post a follow-up thread
to fop-user with the details. I need to run a few more tests before I'm
sure (and I should probably use a profiler instead of analyzing heap
dump files ;-).
Ben
Re: Memory measurement -- importance of Driver.reset() in Cocoon -
NOT!
Posted by John Austin <jw...@sympatico.ca>.
On Mon, 2003-11-24 at 16:23, Glen Mazza wrote:
> --- John Austin <jw...@sympatico.ca> wrote:
> >
> > My own feeling is that FOP
> > will remain
> > problematic for large documents and this will be
> > especially so in
> > server environments such as Cocoon.
> >
>
> We hear you, and we do emphasize performance as one of
> our main goals [1]. As always, you may wish to add a
> Wiki page on things we're not doing that we should be
> doing in order to speed things up. Bugzilla may also
> be a good place. Others can comment on them that way,
> and we can refine what needs to be done.
>
> For me though, depending on the problem, there can be
> a "grokking delay"--I tend not to act until I fully
> understand the issues involved. Also, I am currently
> tied up with layout issues which preclude me from
> getting too much into properties code at this time.
>
> But many of the other committers/contributors have
> more thought-out opinions on how properties should be
> handled and should be able to contribute more quickly
> to your ideas.
Thanks. Sometimes Ijump on things (like a cop on a donut).
Part of the ENTJ personality type (most engineers are ENTP
- Perceiving vs Judgemental).
I want my focus to be on the PropertyList issue, as I feel
there are significant benefits there. I have a few miles to
go to understand the issues in PropertyList and
PropertyListBuilder.
I intentionally have not brought up my new observation:
<obs>
When I run the same test twice with a Driver.reset(), GC and
a wait in between, heap usage the first time is TWICE
the usage of the second run. Time taken is the same.
Just somethng to wonder about, in case we stumble on a
reason.
JMP graphic available on request.
I have learned quite a bit about running FOP (and Java) for
larger files. A year ago, I had terrible trouble running FOP
in Cocoon becasue I had only 256Mb. The same task is a breeze
in the current release.
1) Large (equal) values of -Xms and -Xmx can reduce GC and
speed execution of a task. There is documented support
for this on the Sun site.
2) Use of the Server HotSpot VM makes a bigger difference
than point (1). I have submitted a fix for 'cocoon.sh' to
select this JVM. I am sure this was an oversight at Cocoon.
3) FOP and related libraries compiled with Sun's SDK can all
be run with the current (1.4.1) IBM SDK. The GC performance
of this IBM SDK is too ugly for words. More to thnk about.
4) You can observe Garbage Collection events with the Java option:
-verbose:gc and this works similarly on both IBM and Sun run-times.
Trace syntax is quite different but you can see times and sizes.
</obs>
Given that I may have p*ss*d off some Germans (my Neandertal*
comment ;-), I shall refrain from stating that there may be
something rotten in Denmark - that might annoy Shakespeare-averse
Danes ...
* I lived on Ramstein Air Force Base for 4 years.
Many Neandertals in my high-school at Kaiserslautern!
> Glen
>
> [1]
> http://marc.theaimsgroup.com/?l=fop-dev&m=106735192618324&w=2
>
>
> __________________________________
> Do you Yahoo!?
> Free Pop-Up Blocker - Get it now
> http://companion.yahoo.com/
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset() in Cocoon - NOT!
Posted by Glen Mazza <gr...@yahoo.com>.
--- John Austin <jw...@sympatico.ca> wrote:
>
> My own feeling is that FOP
> will remain
> problematic for large documents and this will be
> especially so in
> server environments such as Cocoon.
>
We hear you, and we do emphasize performance as one of
our main goals [1]. As always, you may wish to add a
Wiki page on things we're not doing that we should be
doing in order to speed things up. Bugzilla may also
be a good place. Others can comment on them that way,
and we can refine what needs to be done.
For me though, depending on the problem, there can be
a "grokking delay"--I tend not to act until I fully
understand the issues involved. Also, I am currently
tied up with layout issues which preclude me from
getting too much into properties code at this time.
But many of the other committers/contributors have
more thought-out opinions on how properties should be
handled and should be able to contribute more quickly
to your ideas.
Glen
[1]
http://marc.theaimsgroup.com/?l=fop-dev&m=106735192618324&w=2
__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/
Re: Memory measurement -- importance of Driver.reset() in Cocoon -
NOT!
Posted by John Austin <jw...@sympatico.ca>.
I took a further look at some of the JVM options and ran a few
more tests. I found a couple of things that may be useful.
1) I have to use -Xmx400m to produce the 2000 page PDF in a
large PDF test, 'time' reported: real ~5m user 2m57.880 ...
2) When I added -Xms400m to the same work 'time' reported: real
4m35.968s user 2m35.190s ...
This is drop in user time of 22 seconds, about 12 percent
The Cocoon Performance Tips page says we should not use -Xms.
One problem with statements of performance is they don't always
discuss the rationale. My own feeling is that FOP will remain
problematic for large documents and this will be especially so in
server environments such as Cocoon.
3) When I add the JVM 1.4 option: '-server', executiuon gets faster.
Time reports: real 2m52.100s user 2m1.520s as improvement of
about 20 percent.
Note that 'cocoon.sh' in the top-level directory for cocoon-2.1.3
supplies the defaults: JAVA_OPTIONS=-Xms32m -Xmx512m but does not
specify '-server'.
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset() in Cocoon -
NOT!
Posted by John Austin <jw...@sympatico.ca>.
On Fri, 2003-11-21 at 00:43, John Austin wrote:
> I mean, I wonder what Cocoon does ?
In FOPSerializer:
/**
* Recycle serializer by removing references
*/
public void recycle() {
super.recycle();
this.driver = null;
this.renderer = null;
}
Apparently, we don't need no stinkin reset();
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset()
Posted by John Austin <jw...@sympatico.ca>.
On Fri, 2003-11-21 at 00:02, Glen Mazza wrote:
> Please do not cross-post to both lists--keep
> development-related issues on fop-dev.
Sorry. Not something I do a lot.
It was posted to fop-user to be available in the
archives for anyone searching on memory useage.
Users will want to make sure they use Driver.reset().
Hmm. I wonder what would Jimmy Buffet do ?
I mean, I wonder what Cocoon does ?
--
John Austin <jw...@sympatico.ca>
Re: Memory measurement -- importance of Driver.reset()
Posted by Glen Mazza <gr...@yahoo.com>.
Please do not cross-post to both lists--keep
development-related issues on fop-dev.
Thanks,
Glen
--- John Austin <jw...@sympatico.ca> wrote:
> After reading the Sept 2003 thread about Memory
__________________________________
Do you Yahoo!?
Free Pop-Up Blocker - Get it now
http://companion.yahoo.com/