You are viewing a plain text version of this content. The canonical link for it is here.
Posted to j-users@xalan.apache.org by Tom Fennelly <to...@gmail.com> on 2008/04/23 16:11:27 UTC
Processing a huge message?
Hi there.
I have a 2Gb message (file based) and was wondering could I transform it
using Xalan. I tried (supplying as a StreamSource) and it bombed out
(OOM errors). I actually tried then with smaller files and went down to
100M and it's still bombing out. Am I doing something wrong?
Regards,
Tom.
Re: Processing a huge message?
Posted by ke...@us.ibm.com.
I've been told the XSLT processor in the Datapower "network appliance" box
has some XSLT streaming capability. I don't know what cases that can or
can't handle, but if this is a serious business need you may want to
contact them and ask.
(Claimer: Datapower is now part of IBM.)
______________________________________
"... Three things see no end: A loop with exit code done wrong,
A semaphore untested, And the change that comes along. ..."
-- "Threes" Rev 1.1 - Duane Elms / Leslie Fish (
http://www.ovff.org/pegasus/songs/threes-rev-11.html)
Re: Processing a huge message?
Posted by Tom Fennelly <to...@gmail.com>.
Ah OK... thanks for the reply and sorry for being slow in replying :-)
I was curious because I was always under the impression that Stream/SAX
based XSLT could (in theory) handle messages of any size (huge
messages). I'd never thought about the full random access aspect.
Anyway, I tried it with Xalan, Saxon and the default in JDK 1.5 and non
of them managed the 100 Mb file (out-of-the-box -- I didn't want to go
playing with VM memory setting). I thought I might be doing something
wrong.
So my interest in this stems from a Codehaus project that I am the lead
on. The main component there is a tool called Smooks
(http://milyn.codehaus.org/Smooks), which allows you to filter a range
of different message input formats (XML, EDI, Java etc) and target
"Visitor" logic at message the fragments to produce a range of different
message output formats (XML, EDI, Java etc) . Among other things, it
has support for fragment based templating using XSLT and a few other
templating solutions (FreeMarker etc). So, it actually does something
similar to your suggested "subtree through Xalan" approach wrt XSL (but
only via DOM - not SAX). Using the SAX filter, we've also managed to do
certain types of processing on huge messages (GBs) i.e. complex
splitting, routing, persistence.
Thanks again for your reply :-)
Regards,
Tom.
keshlam@us.ibm.com wrote:
> In general, XSLT -- because it has full random access to the source
> document -- needs acces to the whole input at once, which means it has
> trouble with huge documents. There have been attempts at "streaming"
> processors, but they generally handle only limited subsets of the
> language, and Xalan doesn't have much (if any) capability in that area.
>
> For more detailed background, check the archives of this mailing list;
> search for discussion of "streaming" and "pruning".
>
> Depending on the exact characteristics of your problem, the simplest
> solution may be to rewrite it as a custom SAX application, possibly
> passing selected subtrees through Xalan. Or you might want to check
> whether your needs fit into the range where one of the existing
> limited-streaming processors (such as the one in the Datapower product)
> could handle it.
>
> ______________________________________
> "... Three things see no end: A loop with exit code done wrong,
> A semaphore untested, And the change that comes along. ..."
> -- "Threes" Rev 1.1 - Duane Elms / Leslie Fish (
> http://www.ovff.org/pegasus/songs/threes-rev-11.html)
>
Re: Processing a huge message?
Posted by ke...@us.ibm.com.
In general, XSLT -- because it has full random access to the source
document -- needs acces to the whole input at once, which means it has
trouble with huge documents. There have been attempts at "streaming"
processors, but they generally handle only limited subsets of the
language, and Xalan doesn't have much (if any) capability in that area.
For more detailed background, check the archives of this mailing list;
search for discussion of "streaming" and "pruning".
Depending on the exact characteristics of your problem, the simplest
solution may be to rewrite it as a custom SAX application, possibly
passing selected subtrees through Xalan. Or you might want to check
whether your needs fit into the range where one of the existing
limited-streaming processors (such as the one in the Datapower product)
could handle it.
______________________________________
"... Three things see no end: A loop with exit code done wrong,
A semaphore untested, And the change that comes along. ..."
-- "Threes" Rev 1.1 - Duane Elms / Leslie Fish (
http://www.ovff.org/pegasus/songs/threes-rev-11.html)
Re: Processing a huge message?
Posted by Tom Fennelly <to...@gmail.com>.
Saxon-SA... isn't that the commercial version??
Thank you Michael.
Michael Ludwig wrote:
> Tom Fennelly schrieb:
>> Hi there.
>>
>> I have a 2Gb message (file based) and was wondering could I transform
>> it using Xalan. I tried (supplying as a StreamSource) and it bombed
>> out (OOM errors). I actually tried then with smaller files and went
>> down to 100M and it's still bombing out. Am I doing something wrong?
>
> 100 MB is still huge. For such huge files, I'd consider using the SAX
> API, or Saxon-SA, which has a streaming mode.
>
> http://www.saxonica.com/documentation/sourcedocs/serial.html
>
> I've seen Xalan-C to be relatively economical with memory.
>
> If you want to use Xalan-J, you must give the JVM enough memory in order
> to avoid java.lang.OutOfMemory. Can't say how much you'll need.
>
> java -Xmx1024MB
>
> Michael Ludwig
>
>
Re: Processing a huge message?
Posted by Michael Ludwig <ml...@as-guides.com>.
Tom Fennelly schrieb:
> Hi there.
>
> I have a 2Gb message (file based) and was wondering could I transform
> it using Xalan. I tried (supplying as a StreamSource) and it bombed
> out (OOM errors). I actually tried then with smaller files and went
> down to 100M and it's still bombing out. Am I doing something wrong?
100 MB is still huge. For such huge files, I'd consider using the SAX
API, or Saxon-SA, which has a streaming mode.
http://www.saxonica.com/documentation/sourcedocs/serial.html
I've seen Xalan-C to be relatively economical with memory.
If you want to use Xalan-J, you must give the JVM enough memory in order
to avoid java.lang.OutOfMemory. Can't say how much you'll need.
java -Xmx1024MB
Michael Ludwig
Re: Processing a huge message?
Posted by Tatu Saloranta <co...@yahoo.com>.
--- Tom Fennelly <to...@gmail.com> wrote:
> Hi there.
>
> I have a 2Gb message (file based) and was wondering
> could I transform it
> using Xalan. I tried (supplying as a StreamSource)
> and it bombed out
> (OOM errors). I actually tried then with smaller
> files and went down to
> 100M and it's still bombing out. Am I doing
> something wrong?
The only realistic way with xsl would be processing
such data subtree by subtree. With 100 meg docs, you'd
need 400-500 megs just to hold DOM tree (or such) in
memory (typical memory usage for DOM is 3x - 5x size
of input doc; part is from utf8 -> Java's 2-byte char
expansion, rest object/structural overhead).
Aside from possibly using SAX, StAX may be bit simpler
to use for explicitly written conversion code.
There is also a streaming alternative to xsl called
stx, which might work for you if you prefer
declarative style.
Hope this helps,
-+ Tatu +-
____________________________________________________________________________________
Be a better friend, newshound, and
know-it-all with Yahoo! Mobile. Try it now. http://mobile.yahoo.com/;_ylt=Ahu06i62sR8HDtDypao8Wcj9tAcJ