Large documents

["Bondi: Runar Oli Bjarnason" <ro...@bi.bondi.is>]

Is is possible to change the garbage collection strategy used by FOP? I
keep getting OutOfMemoryErrors when processing large .fo files (14 MB or
so).

Re: Large documents

[Arved Sandstrom <Ar...@chebucto.ns.ca>]

At 03:55 PM 9/21/00 +0000, Bondi: Runar Oli Bjarnason wrote:
>
>Is is possible to change the garbage collection strategy used by FOP? I
>keep getting OutOfMemoryErrors when processing large .fo files (14 MB or
>so).

FOP is not really optimized in any way - yet. This is mostly deliberate: we 
have a whack of stuff yet to implement and it's not the right time to do 
optimizations. That being said, various spots have been pointed out where we 
can assist GC and we do want to get that happening.

.fo files that start at 14 MB are going to result in a humongous DOM tree 
internally, I would have to think. I wouldn't expect you to have much joy 
with _any_ DOM-based processor, so the preferable strategy (processor 
optimizations or not) is to help out some by looking at the source and 
seeing if parts of it are not independent, and processing the FO piecewise. 
I don't think this will change for a long time.

An anecdote from way back: I was processing oceanographic data sets for a 
researcher as a contract programmer. The available UNIX and VAX machines 
were overloaded and not overly gifted with resources, so I ended up doing 
the primary processing on the latest and greatest home computer. I recall 
carefully segmenting binary data, putting in error checks for off-by-one 
mistakes because I was doing convolutions across segment boundaries, etc 
etc. Typical size of a full data set: ~500 K. Moral of the story: how soon 
we forget. :-) Or in other words, just because we have 512 MB RAM and 25 GB 
of HD doesn't mean that we should squander resources. And I mean this as an 
observation on everything I see happening with Java processing of XML, and 
_not_ just the size of your .fo file.

Arved Sandstrom

Senior Developer
e-plicity.com (www.e-plicity.com)
Halifax, Nova Scotia
"B2B Wireless in Canada's Ocean Playground"