You are viewing a plain text version of this content. The canonical link for it is here.
Posted to j-dev@xerces.apache.org by Srikanth Kurkal <sk...@uucom.com> on 2000/11/30 00:16:55 UTC
Parser runs out of memory parsing large xml files...
Hi all,
I am using Xerces 1.2.0 SAX parser and have a simple application that just
takes as input an xml file and parses, validates and prints out the
contents. It works fine with small xml files but when I try large files (>
50 MB), after a while, the application dies with a
java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
memory. Is this a known bug? Is anyone aware of this bug? Is this fixed
in a later version of xerces?
Srikanth.
--
outbounder...Messages that Move...
Srikanth Kurkal
Outbounder Inc.,
1916 Isaac Newton Square West,
Reston, VA 20190.
Email: skurkal@outbounderinc.com
Phone: (703)464-3231 (W)
Fax: (703)464-3235 (Fax)
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
I am not sure what exactly you mean by reusing the parser over and over
again. But I don't think I am doing that. My test application uses the
parser to parse this large file. And it writes out as it parses it. Sounds
simple. Just like SAX2Writer.java does.
I am curious to try Xerces 2.
Srikanth.
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Wednesday, December 06, 2000 3:04 PM
To: xerces-j-dev@xml.apache.org
Cc: Srikanth Kurkal
Subject: Re: Parser runs out of memory parsing large xml files...
That is really interesting, Are you reusing the parser to parse the big
file over and over again? if in that case the resident memory of JVM keeps
growing and never dropping, then Xerces-J probably have a slight memory
leak. Xerces-J 1 is prone to have memory leak than other parsers with much
simpler designs, since lots of design decisions are based on the
consideration of performance, the parser itself maintains a StringPool and
takes up more memory management tasks away from the JVM. It would be
interested if you can do the same test on Xerces 2 alpha release which will
probably out today or tomorrow, Xerces 2 carries a much simpler design.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: "Eric Ye" <er...@locus.apache.org>
Cc: <xe...@xml.apache.org>
Sent: Wednesday, December 06, 2000 8:11 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> I downloaded the latest xerces code in the morning yesterday. And I also
> tried jdk1.3 on solaris. I get a lot further. Actually, six times as
far.
> But fail after that with a very distinct OutOfMemoryError.
>
> When I do a top on solaris as the application is running, I notice that
the
> resident memory keeps increasing and does not decrease. This does not
> happen with the same application using the AElfred parser. It does not
> increase beyond 14M. Thus, I am able to parse much larger xml
files(atleast
> 10 times larger than 50M) with the AElfred parser. I would've thought
that
> the xerces parser should do the same.
>
>
> Srikanth.
>
>
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Tuesday, December 05, 2000 1:56 PM
> To: Srikanth Kurkal
> Cc: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Simple, go to http://xml.apache.org, click on repositories or get
involved.
>
> BTW, the fix I posted yesterday only fix the problem that the other guy
> reported, not yours, cause his file cause a memory leak in the parser, but
> yours didn't. I haven't gotten time to try your file on the a Solaris box,
> have you tried jdk1.3 on Solaris?
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> Sent: Tuesday, December 05, 2000 7:21 AM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Eric,
> >
> > I am not as familiar with the xerces parser code as I would like to be.
> > Besides, I am not too familiar with the bug fix procedure on xerces-dev.
> > Can you please tell me where exactly to apply the fix, or how to get the
> > parser where the bug is fixed.
> >
> > Thank you for all your help.
> >
> > Thanks,
> > Srikanth.
> >
> >
> > -----Original Message-----
> > From: Eric Ye [mailto:ericye@locus.apache.org]
> > Sent: Monday, December 04, 2000 9:54 PM
> > To: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> >
> >
> > ----- Original Message -----
> > From: "Andy Clark" <an...@apache.org>
> > To: <xe...@xml.apache.org>
> > Sent: Monday, December 04, 2000 6:28 PM
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > > Eric Ye wrote:
> > > > So I fixed this problem by changing to use the addSymbol(String).
> > >
> > > This is a fine solution as long as symbols can never be released
> > > from the StringPool. We should double-check that. Because if
> > > StringPool#orphanString(int) allows symbols to be released, then
> > > the solution breaks down as soon as someone down the pipeline
> > > releases the attribute value. (Assuming that I'm following the
> > > problem and resolution correctly.)
> > >
> >
> >
> > StringPool#orphanString(int) does allow symbols to be released.
> >
> > However, the symbol added actually holds the enumerated type, e.g.
> > (true|false) of an attribute decl, not the actual value of an attribute
> in
> > a instance document, It'd be very unlikely to be orphaned, almost as
> > impossible as releasing the symbol "IDREF". But as you said, there is
no
> > guarantee that symbol will not be released, so I also did a full search
in
> > the source tree, no such release was found.
> >
> >
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
That is really interesting, Are you reusing the parser to parse the big
file over and over again? if in that case the resident memory of JVM keeps
growing and never dropping, then Xerces-J probably have a slight memory
leak. Xerces-J 1 is prone to have memory leak than other parsers with much
simpler designs, since lots of design decisions are based on the
consideration of performance, the parser itself maintains a StringPool and
takes up more memory management tasks away from the JVM. It would be
interested if you can do the same test on Xerces 2 alpha release which will
probably out today or tomorrow, Xerces 2 carries a much simpler design.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: "Eric Ye" <er...@locus.apache.org>
Cc: <xe...@xml.apache.org>
Sent: Wednesday, December 06, 2000 8:11 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> I downloaded the latest xerces code in the morning yesterday. And I also
> tried jdk1.3 on solaris. I get a lot further. Actually, six times as
far.
> But fail after that with a very distinct OutOfMemoryError.
>
> When I do a top on solaris as the application is running, I notice that
the
> resident memory keeps increasing and does not decrease. This does not
> happen with the same application using the AElfred parser. It does not
> increase beyond 14M. Thus, I am able to parse much larger xml
files(atleast
> 10 times larger than 50M) with the AElfred parser. I would've thought
that
> the xerces parser should do the same.
>
>
> Srikanth.
>
>
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Tuesday, December 05, 2000 1:56 PM
> To: Srikanth Kurkal
> Cc: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Simple, go to http://xml.apache.org, click on repositories or get
involved.
>
> BTW, the fix I posted yesterday only fix the problem that the other guy
> reported, not yours, cause his file cause a memory leak in the parser, but
> yours didn't. I haven't gotten time to try your file on the a Solaris box,
> have you tried jdk1.3 on Solaris?
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> Sent: Tuesday, December 05, 2000 7:21 AM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Eric,
> >
> > I am not as familiar with the xerces parser code as I would like to be.
> > Besides, I am not too familiar with the bug fix procedure on xerces-dev.
> > Can you please tell me where exactly to apply the fix, or how to get the
> > parser where the bug is fixed.
> >
> > Thank you for all your help.
> >
> > Thanks,
> > Srikanth.
> >
> >
> > -----Original Message-----
> > From: Eric Ye [mailto:ericye@locus.apache.org]
> > Sent: Monday, December 04, 2000 9:54 PM
> > To: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> >
> >
> > ----- Original Message -----
> > From: "Andy Clark" <an...@apache.org>
> > To: <xe...@xml.apache.org>
> > Sent: Monday, December 04, 2000 6:28 PM
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > > Eric Ye wrote:
> > > > So I fixed this problem by changing to use the addSymbol(String).
> > >
> > > This is a fine solution as long as symbols can never be released
> > > from the StringPool. We should double-check that. Because if
> > > StringPool#orphanString(int) allows symbols to be released, then
> > > the solution breaks down as soon as someone down the pipeline
> > > releases the attribute value. (Assuming that I'm following the
> > > problem and resolution correctly.)
> > >
> >
> >
> > StringPool#orphanString(int) does allow symbols to be released.
> >
> > However, the symbol added actually holds the enumerated type, e.g.
> > (true|false) of an attribute decl, not the actual value of an attribute
> in
> > a instance document, It'd be very unlikely to be orphaned, almost as
> > impossible as releasing the symbol "IDREF". But as you said, there is
no
> > guarantee that symbol will not be released, so I also did a full search
in
> > the source tree, no such release was found.
> >
> >
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Eric,
I downloaded the latest xerces code in the morning yesterday. And I also
tried jdk1.3 on solaris. I get a lot further. Actually, six times as far.
But fail after that with a very distinct OutOfMemoryError.
When I do a top on solaris as the application is running, I notice that the
resident memory keeps increasing and does not decrease. This does not
happen with the same application using the AElfred parser. It does not
increase beyond 14M. Thus, I am able to parse much larger xml files(atleast
10 times larger than 50M) with the AElfred parser. I would've thought that
the xerces parser should do the same.
Srikanth.
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Tuesday, December 05, 2000 1:56 PM
To: Srikanth Kurkal
Cc: xerces-j-dev@xml.apache.org
Subject: Re: Parser runs out of memory parsing large xml files...
Simple, go to http://xml.apache.org, click on repositories or get involved.
BTW, the fix I posted yesterday only fix the problem that the other guy
reported, not yours, cause his file cause a memory leak in the parser, but
yours didn't. I haven't gotten time to try your file on the a Solaris box,
have you tried jdk1.3 on Solaris?
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
Sent: Tuesday, December 05, 2000 7:21 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> I am not as familiar with the xerces parser code as I would like to be.
> Besides, I am not too familiar with the bug fix procedure on xerces-dev.
> Can you please tell me where exactly to apply the fix, or how to get the
> parser where the bug is fixed.
>
> Thank you for all your help.
>
> Thanks,
> Srikanth.
>
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Monday, December 04, 2000 9:54 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
>
>
> ----- Original Message -----
> From: "Andy Clark" <an...@apache.org>
> To: <xe...@xml.apache.org>
> Sent: Monday, December 04, 2000 6:28 PM
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> > Eric Ye wrote:
> > > So I fixed this problem by changing to use the addSymbol(String).
> >
> > This is a fine solution as long as symbols can never be released
> > from the StringPool. We should double-check that. Because if
> > StringPool#orphanString(int) allows symbols to be released, then
> > the solution breaks down as soon as someone down the pipeline
> > releases the attribute value. (Assuming that I'm following the
> > problem and resolution correctly.)
> >
>
>
> StringPool#orphanString(int) does allow symbols to be released.
>
> However, the symbol added actually holds the enumerated type, e.g.
> (true|false) of an attribute decl, not the actual value of an attribute
in
> a instance document, It'd be very unlikely to be orphaned, almost as
> impossible as releasing the symbol "IDREF". But as you said, there is no
> guarantee that symbol will not be released, so I also did a full search in
> the source tree, no such release was found.
>
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
Simple, go to http://xml.apache.org, click on repositories or get involved.
BTW, the fix I posted yesterday only fix the problem that the other guy
reported, not yours, cause his file cause a memory leak in the parser, but
yours didn't. I haven't gotten time to try your file on the a Solaris box,
have you tried jdk1.3 on Solaris?
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
Sent: Tuesday, December 05, 2000 7:21 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> I am not as familiar with the xerces parser code as I would like to be.
> Besides, I am not too familiar with the bug fix procedure on xerces-dev.
> Can you please tell me where exactly to apply the fix, or how to get the
> parser where the bug is fixed.
>
> Thank you for all your help.
>
> Thanks,
> Srikanth.
>
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Monday, December 04, 2000 9:54 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
>
>
> ----- Original Message -----
> From: "Andy Clark" <an...@apache.org>
> To: <xe...@xml.apache.org>
> Sent: Monday, December 04, 2000 6:28 PM
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> > Eric Ye wrote:
> > > So I fixed this problem by changing to use the addSymbol(String).
> >
> > This is a fine solution as long as symbols can never be released
> > from the StringPool. We should double-check that. Because if
> > StringPool#orphanString(int) allows symbols to be released, then
> > the solution breaks down as soon as someone down the pipeline
> > releases the attribute value. (Assuming that I'm following the
> > problem and resolution correctly.)
> >
>
>
> StringPool#orphanString(int) does allow symbols to be released.
>
> However, the symbol added actually holds the enumerated type, e.g.
> (true|false) of an attribute decl, not the actual value of an attribute
in
> a instance document, It'd be very unlikely to be orphaned, almost as
> impossible as releasing the symbol "IDREF". But as you said, there is no
> guarantee that symbol will not be released, so I also did a full search in
> the source tree, no such release was found.
>
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Eric,
I am not as familiar with the xerces parser code as I would like to be.
Besides, I am not too familiar with the bug fix procedure on xerces-dev.
Can you please tell me where exactly to apply the fix, or how to get the
parser where the bug is fixed.
Thank you for all your help.
Thanks,
Srikanth.
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Monday, December 04, 2000 9:54 PM
To: xerces-j-dev@xml.apache.org
Subject: Re: Parser runs out of memory parsing large xml files...
----- Original Message -----
From: "Andy Clark" <an...@apache.org>
To: <xe...@xml.apache.org>
Sent: Monday, December 04, 2000 6:28 PM
Subject: Re: Parser runs out of memory parsing large xml files...
> Eric Ye wrote:
> > So I fixed this problem by changing to use the addSymbol(String).
>
> This is a fine solution as long as symbols can never be released
> from the StringPool. We should double-check that. Because if
> StringPool#orphanString(int) allows symbols to be released, then
> the solution breaks down as soon as someone down the pipeline
> releases the attribute value. (Assuming that I'm following the
> problem and resolution correctly.)
>
StringPool#orphanString(int) does allow symbols to be released.
However, the symbol added actually holds the enumerated type, e.g.
(true|false) of an attribute decl, not the actual value of an attribute in
a instance document, It'd be very unlikely to be orphaned, almost as
impossible as releasing the symbol "IDREF". But as you said, there is no
guarantee that symbol will not be released, so I also did a full search in
the source tree, no such release was found.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
----- Original Message -----
From: "Andy Clark" <an...@apache.org>
To: <xe...@xml.apache.org>
Sent: Monday, December 04, 2000 6:28 PM
Subject: Re: Parser runs out of memory parsing large xml files...
> Eric Ye wrote:
> > So I fixed this problem by changing to use the addSymbol(String).
>
> This is a fine solution as long as symbols can never be released
> from the StringPool. We should double-check that. Because if
> StringPool#orphanString(int) allows symbols to be released, then
> the solution breaks down as soon as someone down the pipeline
> releases the attribute value. (Assuming that I'm following the
> problem and resolution correctly.)
>
StringPool#orphanString(int) does allow symbols to be released.
However, the symbol added actually holds the enumerated type, e.g.
(true|false) of an attribute decl, not the actual value of an attribute in
a instance document, It'd be very unlikely to be orphaned, almost as
impossible as releasing the symbol "IDREF". But as you said, there is no
guarantee that symbol will not be released, so I also did a full search in
the source tree, no such release was found.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Andy Clark <an...@apache.org>.
Eric Ye wrote:
> So I fixed this problem by changing to use the addSymbol(String).
This is a fine solution as long as symbols can never be released
from the StringPool. We should double-check that. Because if
StringPool#orphanString(int) allows symbols to be released, then
the solution breaks down as soon as someone down the pipeline
releases the attribute value. (Assuming that I'm following the
problem and resolution correctly.)
We don't have to worry about this problem at all in Xerces2
because we don't assign unique IDs to strings/symbols -- we just
pass the java.lang.String object along. The uniqueness of the
reference will ensure that it's a symbol.
--
Andy Clark * IBM, TRL - Japan * andyc@apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
I can reproduce the syptom, but I don't think you got the right analysis
though, the ChunkByteArray will not allocated memory of the size of the file
or even half of that, that is not the algorithm it deployed, and a UCSReader
would not be used for a UTF-8 encoded file, UTF8Reader will be used.
I also find where the problem is orignated, it is due to an
fString.addString(String) method was used in place of
fString.addSymbol(String) when converting a internal attribute enumerated
value representation to a String. I guess whoever wrote this method must be
considering that addString() is faster, but the price is that a lot more
objects would be created when a elements with a default attribute of
enumerated type recurrs for 500,000 times, as in the case of the
hugepersonal.xml, string "true false" was created for at least 500,000 times
(or even double that ), since there is attribute decl like this:
<!ATTLIST person contr (true|false) 'false'>
So I fixed this problem by changing to use the addSymbol(String).
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Tosiki IGA" <to...@nifty.ne.jp>
To: <xe...@xml.apache.org>
Sent: Monday, December 04, 2000 2:21 PM
Subject: Re: Parser runs out of memory parsing large xml files...
> I also found the bug 'runs out of memory' by parsing huge xml file
> on Xerces-J 1.2.1.
>
> [Environment]
> Xerces-J 1.2.1
> Windows2000 Professional
> SUN JDK 1.3.0
>
> It simply appear by using huge 'personal.xml', sample data of Xerces-J.
>
> And I attached source code to make huge 'personal.xml' which
> 500000 times data of original 'personal.xml'.
> It create 111MB size of 'personal.xml'.
> Parsing this huge 'personal.xml' can simply appear this probrem.
>
>
> This probrem is by Xerces-J SAX2 (and SAX) implementation uses
> org.apache.xerces.readers.UCSReader
>
> This UCSReader uses
> org.apache.xerces.utils.ChunkyByteArray.
>
> ChunkyByteArray causes memory allocation up to size of XML file on
> processing XML file.
> (I found another case memory allocate up to half of XML file size)
>
> Tosiki IGA
>
----------------------------------------------------------------------------
----
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Tosiki IGA <to...@nifty.ne.jp>.
I also found the bug 'runs out of memory' by parsing huge xml file
on Xerces-J 1.2.1.
[Environment]
Xerces-J 1.2.1
Windows2000 Professional
SUN JDK 1.3.0
It simply appear by using huge 'personal.xml', sample data of Xerces-J.
And I attached source code to make huge 'personal.xml' which
500000 times data of original 'personal.xml'.
It create 111MB size of 'personal.xml'.
Parsing this huge 'personal.xml' can simply appear this probrem.
This probrem is by Xerces-J SAX2 (and SAX) implementation uses
org.apache.xerces.readers.UCSReader
This UCSReader uses
org.apache.xerces.utils.ChunkyByteArray.
ChunkyByteArray causes memory allocation up to size of XML file on
processing XML file.
(I found another case memory allocate up to half of XML file size)
Tosiki IGA
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
I am running it on NT and jdk1.3.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: "Eric Ye" <er...@locus.apache.org>; <xe...@xml.apache.org>
Sent: Monday, December 04, 2000 10:49 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> I am not sure what machine you ran it on. I am working on a solaris box,
> although I don't know what difference that would make. I tried the
> SAX2Writer and the SAX2Count, and I ran it on a large xml file (110M),
they
> both ran out of memory. This is using the default heap size. I did this
> because you'd mentioned in your previous email that you were able to run
> with 2M heap.
>
> But all of this runs on an NT box without a hitch.
>
> Oh, I used the jdk1.2.2.
>
>
> Thanks,
> Srikanth.
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Friday, December 01, 2000 8:01 PM
> To: xerces-j-dev@xml.apache.org; Srikanth Kurkal
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Which JDK version are you working with? I tried jdk1.3 and jdk1.2.2
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Eric Ye" <er...@locus.apache.org>
> To: "Srikanth Kurkal" <sk...@uucom.com>; <xe...@xml.apache.org>
> Sent: Friday, December 01, 2000 4:07 PM
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> > I just created the 50Mbytes xml file with exactly the same content as
the
> > one you sent me, and I ran it through the sample program, it works fine
> with
> > even 2Mbytes max heapsize for JVM, here is the output:
> >
> > java -ms1M -mx2M sax.SAXCount testbig.xml // this is the
50Mbytes
> > xmlfile
> > testbig.xml : 16765ms (800004 elements, 400000 attributes, 0 spaces,
> > 26655600 chars)
> >
> > You can try to run the sample that comes with the parser instead you own
> > application, there might be something in your application that holds the
> > reference to everything that was passed in from SAX call backs.
> >
> >
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> > ----- Original Message -----
> > From: "Srikanth Kurkal" <sk...@uucom.com>
> > To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> > Sent: Friday, December 01, 2000 1:32 PM
> > Subject: RE: Parser runs out of memory parsing large xml files...
> >
> >
> > >
> > > Eric,
> > >
> > > Actually, no. The elements are the same. They just repeat many times
> > with
> > > different data.
> > >
> > > Anyway, here is the chunk that is repeated over 100000 times.
> Otherwise,
> > > the xml file is innocuous.
> > >
> > > Srikanth.
> > >
> > >
> > >
> > > --- BEGIN XML section ------
> > >
> > > <recipient-section>
> > > <add-recipients>
> > > <num-recipients>0100000</num-recipients>
> > > <recipients>
> > > <recipient>
> > >
<recipient-identifier>GRP100SK-5-0</recipient-identifier>
> > > <details>
> > > <first-name>First0</first-name>
> > > <last-name>Last0</last-name>
> > > <address-info>
> > > <email-source>UUCom, Inc.</email-source>
> > > <address da-type="EMAIL" da-opt="HTML"
da-pref="1"
> > > da-specific-address="RFC-822">email0@sink.efoton.net</address>
> > > </address-info>
> > > </details>
> > > </recipient>
> > >
> > > <recipient>
> > >
<recipient-identifier>GRP100SK-5-1</recipient-identifier>
> > > <details>
> > > <first-name>First1</first-name>
> > > <last-name>Last1</last-name>
> > > <address-info>
> > > <email-source>UUCom, Inc.</email-source>
> > > <address da-type="EMAIL" da-opt="HTML"
da-pref="1"
> > > da-specific-address="RFC-822">email1@sink.efoton.net</address>
> > > </address-info>
> > > </details>
> > > </recipient>
> > > ... and so forth 100 thousand recipients...
> > >
> > > </add-recipients>
> > > <recipient-section>
> > >
> > > -- END XML section -----
> > >
> > >
> > >
> > >
> > > -----Original Message-----
> > > From: Eric Ye [mailto:ericye@locus.apache.org]
> > > Sent: Friday, December 01, 2000 1:25 PM
> > > To: Srikanth Kurkal
> > > Cc: xerces-j-dev@xml.apache.org
> > > Subject: Re: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > I assume these elements are all different then the size of symbol
table
> > or
> > > StringPool) will just keep growing.
> > > if you have a mini version of the file, send it to me, I am more than
> > happy
> > > to take a look.
> > > _____
> > >
> > >
> > > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> > >
> > > ----- Original Message -----
> > > From: "Srikanth Kurkal" <sk...@uucom.com>
> > > To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> > > Sent: Friday, December 01, 2000 10:01 AM
> > > Subject: RE: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > >
> > > >
> > > > Yes. The xml file has a whole bunch of elements. I ran the
SAXCount
> > > > example program that came with the parser on Windows NT on this
large
> > xml
> > > > file. I think it has more than 80K elements. The text data is
> actually
> > > > very small. Maybe only a few bytes. Of course, not all elements
have
> > > data.
> > > > Some elements nest other elements and so forth.
> > > >
> > > > Srikanth.
> > > >
> > > > -----Original Message-----
> > > > From: Eric Ye [mailto:ericye@locus.apache.org]
> > > > Sent: Thursday, November 30, 2000 7:32 PM
> > > > To: xerces-j-dev@xml.apache.org
> > > > Subject: Re: Parser runs out of memory parsing large xml files...
> > > >
> > > >
> > > > What is your huge XML file looks files? large chunk of text content
> or
> > > lots
> > > > of elements and attributes?
> > > > _____
> > > >
> > > >
> > > > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> > > >
> > > > ----- Original Message -----
> > > > From: "Srikanth Kurkal" <sk...@uucom.com>
> > > > To: <xe...@xml.apache.org>
> > > > Sent: Thursday, November 30, 2000 11:20 AM
> > > > Subject: RE: Parser runs out of memory parsing large xml files...
> > > >
> > > >
> > > > >
> > > > > Thanks Luke for your response. I had actually tried you
suggestion
> of
> > > > using
> > > > > the -Xmx option. I bumped it up to 64MB. I get a little further,
> but
> > > > then
> > > > > it dies whenever it hits the max. I have no control over how big
> the
> > > > input
> > > > > xml files are. Therefore, I have no way of estimating how much I
> > should
> > > > set
> > > > > the max heap size to.
> > > > >
> > > > > It seems to me like the parser is not releasing the memory that it
> is
> > > > using.
> > > > > I would have thought that DOM would be more memory intensive and
not
> > > SAX.
> > > > > But somehow, I am seeing this behavior using the SAX2 API.
> > > > >
> > > > > Srikanth.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Luke Blanshard [mailto:luke@quiq.com]
> > > > > Sent: Wednesday, November 29, 2000 7:04 PM
> > > > > To: xerces-j-dev@xml.apache.org
> > > > > Subject: Re: Parser runs out of memory parsing large xml files...
> > > > >
> > > > >
> > > > > Srikanth Kurkal wrote:
> > > > >
> > > > > > I am using Xerces 1.2.0 SAX parser and have a simple application
> > that
> > > > just
> > > > > > takes as input an xml file and parses, validates and prints out
> the
> > > > > > contents. It works fine with small xml files but when I try
large
> > > files
> > > > > (>
> > > > > > 50 MB), after a while, the application dies with a
> > > > > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7
with
> > 2G
> > > of
> > > > > > memory. Is this a known bug? Is anyone aware of this bug? Is
> this
> > > > fixed
> > > > > > in a later version of xerces?
> > > > >
> > > > > How much memory you have on your box is irrelevant. What matters
is
> > how
> > > > > much
> > > > > memory you have set as your JVM's max. I believe the default is
> > either
> > > 16
> > > > > or
> > > > > 32 megs. Try bumping it up with -mx.
> > > > >
> > > > > Of course, this begs the question: why would a SAX application
grow
> so
> > > > > large?
> > > > > I haven't ever mixed SAX with validation, but I'll bet that this
> > > > combination
> > > > > grows with the file for some reason. It does sound like a bug,
> > > actually.
> > > > >
> > > > > Luke
> > > > >
> > > > >
> > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > > >
> > > > >
> > > > >
> > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > > >
> > > > >
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Eric,
I am not sure what machine you ran it on. I am working on a solaris box,
although I don't know what difference that would make. I tried the
SAX2Writer and the SAX2Count, and I ran it on a large xml file (110M), they
both ran out of memory. This is using the default heap size. I did this
because you'd mentioned in your previous email that you were able to run
with 2M heap.
But all of this runs on an NT box without a hitch.
Oh, I used the jdk1.2.2.
Thanks,
Srikanth.
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Friday, December 01, 2000 8:01 PM
To: xerces-j-dev@xml.apache.org; Srikanth Kurkal
Subject: Re: Parser runs out of memory parsing large xml files...
Which JDK version are you working with? I tried jdk1.3 and jdk1.2.2
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Eric Ye" <er...@locus.apache.org>
To: "Srikanth Kurkal" <sk...@uucom.com>; <xe...@xml.apache.org>
Sent: Friday, December 01, 2000 4:07 PM
Subject: Re: Parser runs out of memory parsing large xml files...
> I just created the 50Mbytes xml file with exactly the same content as the
> one you sent me, and I ran it through the sample program, it works fine
with
> even 2Mbytes max heapsize for JVM, here is the output:
>
> java -ms1M -mx2M sax.SAXCount testbig.xml // this is the 50Mbytes
> xmlfile
> testbig.xml : 16765ms (800004 elements, 400000 attributes, 0 spaces,
> 26655600 chars)
>
> You can try to run the sample that comes with the parser instead you own
> application, there might be something in your application that holds the
> reference to everything that was passed in from SAX call backs.
>
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> Sent: Friday, December 01, 2000 1:32 PM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Eric,
> >
> > Actually, no. The elements are the same. They just repeat many times
> with
> > different data.
> >
> > Anyway, here is the chunk that is repeated over 100000 times.
Otherwise,
> > the xml file is innocuous.
> >
> > Srikanth.
> >
> >
> >
> > --- BEGIN XML section ------
> >
> > <recipient-section>
> > <add-recipients>
> > <num-recipients>0100000</num-recipients>
> > <recipients>
> > <recipient>
> > <recipient-identifier>GRP100SK-5-0</recipient-identifier>
> > <details>
> > <first-name>First0</first-name>
> > <last-name>Last0</last-name>
> > <address-info>
> > <email-source>UUCom, Inc.</email-source>
> > <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> > da-specific-address="RFC-822">email0@sink.efoton.net</address>
> > </address-info>
> > </details>
> > </recipient>
> >
> > <recipient>
> > <recipient-identifier>GRP100SK-5-1</recipient-identifier>
> > <details>
> > <first-name>First1</first-name>
> > <last-name>Last1</last-name>
> > <address-info>
> > <email-source>UUCom, Inc.</email-source>
> > <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> > da-specific-address="RFC-822">email1@sink.efoton.net</address>
> > </address-info>
> > </details>
> > </recipient>
> > ... and so forth 100 thousand recipients...
> >
> > </add-recipients>
> > <recipient-section>
> >
> > -- END XML section -----
> >
> >
> >
> >
> > -----Original Message-----
> > From: Eric Ye [mailto:ericye@locus.apache.org]
> > Sent: Friday, December 01, 2000 1:25 PM
> > To: Srikanth Kurkal
> > Cc: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > I assume these elements are all different then the size of symbol table
> or
> > StringPool) will just keep growing.
> > if you have a mini version of the file, send it to me, I am more than
> happy
> > to take a look.
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> > ----- Original Message -----
> > From: "Srikanth Kurkal" <sk...@uucom.com>
> > To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> > Sent: Friday, December 01, 2000 10:01 AM
> > Subject: RE: Parser runs out of memory parsing large xml files...
> >
> >
> > >
> > >
> > > Yes. The xml file has a whole bunch of elements. I ran the SAXCount
> > > example program that came with the parser on Windows NT on this large
> xml
> > > file. I think it has more than 80K elements. The text data is
actually
> > > very small. Maybe only a few bytes. Of course, not all elements have
> > data.
> > > Some elements nest other elements and so forth.
> > >
> > > Srikanth.
> > >
> > > -----Original Message-----
> > > From: Eric Ye [mailto:ericye@locus.apache.org]
> > > Sent: Thursday, November 30, 2000 7:32 PM
> > > To: xerces-j-dev@xml.apache.org
> > > Subject: Re: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > What is your huge XML file looks files? large chunk of text content
or
> > lots
> > > of elements and attributes?
> > > _____
> > >
> > >
> > > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> > >
> > > ----- Original Message -----
> > > From: "Srikanth Kurkal" <sk...@uucom.com>
> > > To: <xe...@xml.apache.org>
> > > Sent: Thursday, November 30, 2000 11:20 AM
> > > Subject: RE: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > >
> > > > Thanks Luke for your response. I had actually tried you suggestion
of
> > > using
> > > > the -Xmx option. I bumped it up to 64MB. I get a little further,
but
> > > then
> > > > it dies whenever it hits the max. I have no control over how big
the
> > > input
> > > > xml files are. Therefore, I have no way of estimating how much I
> should
> > > set
> > > > the max heap size to.
> > > >
> > > > It seems to me like the parser is not releasing the memory that it
is
> > > using.
> > > > I would have thought that DOM would be more memory intensive and not
> > SAX.
> > > > But somehow, I am seeing this behavior using the SAX2 API.
> > > >
> > > > Srikanth.
> > > >
> > > > -----Original Message-----
> > > > From: Luke Blanshard [mailto:luke@quiq.com]
> > > > Sent: Wednesday, November 29, 2000 7:04 PM
> > > > To: xerces-j-dev@xml.apache.org
> > > > Subject: Re: Parser runs out of memory parsing large xml files...
> > > >
> > > >
> > > > Srikanth Kurkal wrote:
> > > >
> > > > > I am using Xerces 1.2.0 SAX parser and have a simple application
> that
> > > just
> > > > > takes as input an xml file and parses, validates and prints out
the
> > > > > contents. It works fine with small xml files but when I try large
> > files
> > > > (>
> > > > > 50 MB), after a while, the application dies with a
> > > > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with
> 2G
> > of
> > > > > memory. Is this a known bug? Is anyone aware of this bug? Is
this
> > > fixed
> > > > > in a later version of xerces?
> > > >
> > > > How much memory you have on your box is irrelevant. What matters is
> how
> > > > much
> > > > memory you have set as your JVM's max. I believe the default is
> either
> > 16
> > > > or
> > > > 32 megs. Try bumping it up with -mx.
> > > >
> > > > Of course, this begs the question: why would a SAX application grow
so
> > > > large?
> > > > I haven't ever mixed SAX with validation, but I'll bet that this
> > > combination
> > > > grows with the file for some reason. It does sound like a bug,
> > actually.
> > > >
> > > > Luke
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
Which JDK version are you working with? I tried jdk1.3 and jdk1.2.2
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Eric Ye" <er...@locus.apache.org>
To: "Srikanth Kurkal" <sk...@uucom.com>; <xe...@xml.apache.org>
Sent: Friday, December 01, 2000 4:07 PM
Subject: Re: Parser runs out of memory parsing large xml files...
> I just created the 50Mbytes xml file with exactly the same content as the
> one you sent me, and I ran it through the sample program, it works fine
with
> even 2Mbytes max heapsize for JVM, here is the output:
>
> java -ms1M -mx2M sax.SAXCount testbig.xml // this is the 50Mbytes
> xmlfile
> testbig.xml : 16765ms (800004 elements, 400000 attributes, 0 spaces,
> 26655600 chars)
>
> You can try to run the sample that comes with the parser instead you own
> application, there might be something in your application that holds the
> reference to everything that was passed in from SAX call backs.
>
>
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> Sent: Friday, December 01, 2000 1:32 PM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Eric,
> >
> > Actually, no. The elements are the same. They just repeat many times
> with
> > different data.
> >
> > Anyway, here is the chunk that is repeated over 100000 times.
Otherwise,
> > the xml file is innocuous.
> >
> > Srikanth.
> >
> >
> >
> > --- BEGIN XML section ------
> >
> > <recipient-section>
> > <add-recipients>
> > <num-recipients>0100000</num-recipients>
> > <recipients>
> > <recipient>
> > <recipient-identifier>GRP100SK-5-0</recipient-identifier>
> > <details>
> > <first-name>First0</first-name>
> > <last-name>Last0</last-name>
> > <address-info>
> > <email-source>UUCom, Inc.</email-source>
> > <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> > da-specific-address="RFC-822">email0@sink.efoton.net</address>
> > </address-info>
> > </details>
> > </recipient>
> >
> > <recipient>
> > <recipient-identifier>GRP100SK-5-1</recipient-identifier>
> > <details>
> > <first-name>First1</first-name>
> > <last-name>Last1</last-name>
> > <address-info>
> > <email-source>UUCom, Inc.</email-source>
> > <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> > da-specific-address="RFC-822">email1@sink.efoton.net</address>
> > </address-info>
> > </details>
> > </recipient>
> > ... and so forth 100 thousand recipients...
> >
> > </add-recipients>
> > <recipient-section>
> >
> > -- END XML section -----
> >
> >
> >
> >
> > -----Original Message-----
> > From: Eric Ye [mailto:ericye@locus.apache.org]
> > Sent: Friday, December 01, 2000 1:25 PM
> > To: Srikanth Kurkal
> > Cc: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > I assume these elements are all different then the size of symbol table
> or
> > StringPool) will just keep growing.
> > if you have a mini version of the file, send it to me, I am more than
> happy
> > to take a look.
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> > ----- Original Message -----
> > From: "Srikanth Kurkal" <sk...@uucom.com>
> > To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> > Sent: Friday, December 01, 2000 10:01 AM
> > Subject: RE: Parser runs out of memory parsing large xml files...
> >
> >
> > >
> > >
> > > Yes. The xml file has a whole bunch of elements. I ran the SAXCount
> > > example program that came with the parser on Windows NT on this large
> xml
> > > file. I think it has more than 80K elements. The text data is
actually
> > > very small. Maybe only a few bytes. Of course, not all elements have
> > data.
> > > Some elements nest other elements and so forth.
> > >
> > > Srikanth.
> > >
> > > -----Original Message-----
> > > From: Eric Ye [mailto:ericye@locus.apache.org]
> > > Sent: Thursday, November 30, 2000 7:32 PM
> > > To: xerces-j-dev@xml.apache.org
> > > Subject: Re: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > What is your huge XML file looks files? large chunk of text content
or
> > lots
> > > of elements and attributes?
> > > _____
> > >
> > >
> > > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> > >
> > > ----- Original Message -----
> > > From: "Srikanth Kurkal" <sk...@uucom.com>
> > > To: <xe...@xml.apache.org>
> > > Sent: Thursday, November 30, 2000 11:20 AM
> > > Subject: RE: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > >
> > > > Thanks Luke for your response. I had actually tried you suggestion
of
> > > using
> > > > the -Xmx option. I bumped it up to 64MB. I get a little further,
but
> > > then
> > > > it dies whenever it hits the max. I have no control over how big
the
> > > input
> > > > xml files are. Therefore, I have no way of estimating how much I
> should
> > > set
> > > > the max heap size to.
> > > >
> > > > It seems to me like the parser is not releasing the memory that it
is
> > > using.
> > > > I would have thought that DOM would be more memory intensive and not
> > SAX.
> > > > But somehow, I am seeing this behavior using the SAX2 API.
> > > >
> > > > Srikanth.
> > > >
> > > > -----Original Message-----
> > > > From: Luke Blanshard [mailto:luke@quiq.com]
> > > > Sent: Wednesday, November 29, 2000 7:04 PM
> > > > To: xerces-j-dev@xml.apache.org
> > > > Subject: Re: Parser runs out of memory parsing large xml files...
> > > >
> > > >
> > > > Srikanth Kurkal wrote:
> > > >
> > > > > I am using Xerces 1.2.0 SAX parser and have a simple application
> that
> > > just
> > > > > takes as input an xml file and parses, validates and prints out
the
> > > > > contents. It works fine with small xml files but when I try large
> > files
> > > > (>
> > > > > 50 MB), after a while, the application dies with a
> > > > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with
> 2G
> > of
> > > > > memory. Is this a known bug? Is anyone aware of this bug? Is
this
> > > fixed
> > > > > in a later version of xerces?
> > > >
> > > > How much memory you have on your box is irrelevant. What matters is
> how
> > > > much
> > > > memory you have set as your JVM's max. I believe the default is
> either
> > 16
> > > > or
> > > > 32 megs. Try bumping it up with -mx.
> > > >
> > > > Of course, this begs the question: why would a SAX application grow
so
> > > > large?
> > > > I haven't ever mixed SAX with validation, but I'll bet that this
> > > combination
> > > > grows with the file for some reason. It does sound like a bug,
> > actually.
> > > >
> > > > Luke
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > >
> > > >
> > >
> ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
I just created the 50Mbytes xml file with exactly the same content as the
one you sent me, and I ran it through the sample program, it works fine with
even 2Mbytes max heapsize for JVM, here is the output:
java -ms1M -mx2M sax.SAXCount testbig.xml // this is the 50Mbytes
xmlfile
testbig.xml : 16765ms (800004 elements, 400000 attributes, 0 spaces,
26655600 chars)
You can try to run the sample that comes with the parser instead you own
application, there might be something in your application that holds the
reference to everything that was passed in from SAX call backs.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
Sent: Friday, December 01, 2000 1:32 PM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Eric,
>
> Actually, no. The elements are the same. They just repeat many times
with
> different data.
>
> Anyway, here is the chunk that is repeated over 100000 times. Otherwise,
> the xml file is innocuous.
>
> Srikanth.
>
>
>
> --- BEGIN XML section ------
>
> <recipient-section>
> <add-recipients>
> <num-recipients>0100000</num-recipients>
> <recipients>
> <recipient>
> <recipient-identifier>GRP100SK-5-0</recipient-identifier>
> <details>
> <first-name>First0</first-name>
> <last-name>Last0</last-name>
> <address-info>
> <email-source>UUCom, Inc.</email-source>
> <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> da-specific-address="RFC-822">email0@sink.efoton.net</address>
> </address-info>
> </details>
> </recipient>
>
> <recipient>
> <recipient-identifier>GRP100SK-5-1</recipient-identifier>
> <details>
> <first-name>First1</first-name>
> <last-name>Last1</last-name>
> <address-info>
> <email-source>UUCom, Inc.</email-source>
> <address da-type="EMAIL" da-opt="HTML" da-pref="1"
> da-specific-address="RFC-822">email1@sink.efoton.net</address>
> </address-info>
> </details>
> </recipient>
> ... and so forth 100 thousand recipients...
>
> </add-recipients>
> <recipient-section>
>
> -- END XML section -----
>
>
>
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Friday, December 01, 2000 1:25 PM
> To: Srikanth Kurkal
> Cc: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> I assume these elements are all different then the size of symbol table
or
> StringPool) will just keep growing.
> if you have a mini version of the file, send it to me, I am more than
happy
> to take a look.
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
> Sent: Friday, December 01, 2000 10:01 AM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> >
> > Yes. The xml file has a whole bunch of elements. I ran the SAXCount
> > example program that came with the parser on Windows NT on this large
xml
> > file. I think it has more than 80K elements. The text data is actually
> > very small. Maybe only a few bytes. Of course, not all elements have
> data.
> > Some elements nest other elements and so forth.
> >
> > Srikanth.
> >
> > -----Original Message-----
> > From: Eric Ye [mailto:ericye@locus.apache.org]
> > Sent: Thursday, November 30, 2000 7:32 PM
> > To: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > What is your huge XML file looks files? large chunk of text content or
> lots
> > of elements and attributes?
> > _____
> >
> >
> > Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
> >
> > ----- Original Message -----
> > From: "Srikanth Kurkal" <sk...@uucom.com>
> > To: <xe...@xml.apache.org>
> > Sent: Thursday, November 30, 2000 11:20 AM
> > Subject: RE: Parser runs out of memory parsing large xml files...
> >
> >
> > >
> > > Thanks Luke for your response. I had actually tried you suggestion of
> > using
> > > the -Xmx option. I bumped it up to 64MB. I get a little further, but
> > then
> > > it dies whenever it hits the max. I have no control over how big the
> > input
> > > xml files are. Therefore, I have no way of estimating how much I
should
> > set
> > > the max heap size to.
> > >
> > > It seems to me like the parser is not releasing the memory that it is
> > using.
> > > I would have thought that DOM would be more memory intensive and not
> SAX.
> > > But somehow, I am seeing this behavior using the SAX2 API.
> > >
> > > Srikanth.
> > >
> > > -----Original Message-----
> > > From: Luke Blanshard [mailto:luke@quiq.com]
> > > Sent: Wednesday, November 29, 2000 7:04 PM
> > > To: xerces-j-dev@xml.apache.org
> > > Subject: Re: Parser runs out of memory parsing large xml files...
> > >
> > >
> > > Srikanth Kurkal wrote:
> > >
> > > > I am using Xerces 1.2.0 SAX parser and have a simple application
that
> > just
> > > > takes as input an xml file and parses, validates and prints out the
> > > > contents. It works fine with small xml files but when I try large
> files
> > > (>
> > > > 50 MB), after a while, the application dies with a
> > > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with
2G
> of
> > > > memory. Is this a known bug? Is anyone aware of this bug? Is this
> > fixed
> > > > in a later version of xerces?
> > >
> > > How much memory you have on your box is irrelevant. What matters is
how
> > > much
> > > memory you have set as your JVM's max. I believe the default is
either
> 16
> > > or
> > > 32 megs. Try bumping it up with -mx.
> > >
> > > Of course, this begs the question: why would a SAX application grow so
> > > large?
> > > I haven't ever mixed SAX with validation, but I'll bet that this
> > combination
> > > grows with the file for some reason. It does sound like a bug,
> actually.
> > >
> > > Luke
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Eric,
Actually, no. The elements are the same. They just repeat many times with
different data.
Anyway, here is the chunk that is repeated over 100000 times. Otherwise,
the xml file is innocuous.
Srikanth.
--- BEGIN XML section ------
<recipient-section>
<add-recipients>
<num-recipients>0100000</num-recipients>
<recipients>
<recipient>
<recipient-identifier>GRP100SK-5-0</recipient-identifier>
<details>
<first-name>First0</first-name>
<last-name>Last0</last-name>
<address-info>
<email-source>UUCom, Inc.</email-source>
<address da-type="EMAIL" da-opt="HTML" da-pref="1"
da-specific-address="RFC-822">email0@sink.efoton.net</address>
</address-info>
</details>
</recipient>
<recipient>
<recipient-identifier>GRP100SK-5-1</recipient-identifier>
<details>
<first-name>First1</first-name>
<last-name>Last1</last-name>
<address-info>
<email-source>UUCom, Inc.</email-source>
<address da-type="EMAIL" da-opt="HTML" da-pref="1"
da-specific-address="RFC-822">email1@sink.efoton.net</address>
</address-info>
</details>
</recipient>
... and so forth 100 thousand recipients...
</add-recipients>
<recipient-section>
-- END XML section -----
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Friday, December 01, 2000 1:25 PM
To: Srikanth Kurkal
Cc: xerces-j-dev@xml.apache.org
Subject: Re: Parser runs out of memory parsing large xml files...
I assume these elements are all different then the size of symbol table ( or
StringPool) will just keep growing.
if you have a mini version of the file, send it to me, I am more than happy
to take a look.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
Sent: Friday, December 01, 2000 10:01 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> Yes. The xml file has a whole bunch of elements. I ran the SAXCount
> example program that came with the parser on Windows NT on this large xml
> file. I think it has more than 80K elements. The text data is actually
> very small. Maybe only a few bytes. Of course, not all elements have
data.
> Some elements nest other elements and so forth.
>
> Srikanth.
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Thursday, November 30, 2000 7:32 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> What is your huge XML file looks files? large chunk of text content or
lots
> of elements and attributes?
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>
> Sent: Thursday, November 30, 2000 11:20 AM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Thanks Luke for your response. I had actually tried you suggestion of
> using
> > the -Xmx option. I bumped it up to 64MB. I get a little further, but
> then
> > it dies whenever it hits the max. I have no control over how big the
> input
> > xml files are. Therefore, I have no way of estimating how much I should
> set
> > the max heap size to.
> >
> > It seems to me like the parser is not releasing the memory that it is
> using.
> > I would have thought that DOM would be more memory intensive and not
SAX.
> > But somehow, I am seeing this behavior using the SAX2 API.
> >
> > Srikanth.
> >
> > -----Original Message-----
> > From: Luke Blanshard [mailto:luke@quiq.com]
> > Sent: Wednesday, November 29, 2000 7:04 PM
> > To: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > Srikanth Kurkal wrote:
> >
> > > I am using Xerces 1.2.0 SAX parser and have a simple application that
> just
> > > takes as input an xml file and parses, validates and prints out the
> > > contents. It works fine with small xml files but when I try large
files
> > (>
> > > 50 MB), after a while, the application dies with a
> > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G
of
> > > memory. Is this a known bug? Is anyone aware of this bug? Is this
> fixed
> > > in a later version of xerces?
> >
> > How much memory you have on your box is irrelevant. What matters is how
> > much
> > memory you have set as your JVM's max. I believe the default is either
16
> > or
> > 32 megs. Try bumping it up with -mx.
> >
> > Of course, this begs the question: why would a SAX application grow so
> > large?
> > I haven't ever mixed SAX with validation, but I'll bet that this
> combination
> > grows with the file for some reason. It does sound like a bug,
actually.
> >
> > Luke
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
I assume these elements are all different then the size of symbol table ( or
StringPool) will just keep growing.
if you have a mini version of the file, send it to me, I am more than happy
to take a look.
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>; "Eric Ye" <er...@locus.apache.org>
Sent: Friday, December 01, 2000 10:01 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> Yes. The xml file has a whole bunch of elements. I ran the SAXCount
> example program that came with the parser on Windows NT on this large xml
> file. I think it has more than 80K elements. The text data is actually
> very small. Maybe only a few bytes. Of course, not all elements have
data.
> Some elements nest other elements and so forth.
>
> Srikanth.
>
> -----Original Message-----
> From: Eric Ye [mailto:ericye@locus.apache.org]
> Sent: Thursday, November 30, 2000 7:32 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> What is your huge XML file looks files? large chunk of text content or
lots
> of elements and attributes?
> _____
>
>
> Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
>
> ----- Original Message -----
> From: "Srikanth Kurkal" <sk...@uucom.com>
> To: <xe...@xml.apache.org>
> Sent: Thursday, November 30, 2000 11:20 AM
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
> >
> > Thanks Luke for your response. I had actually tried you suggestion of
> using
> > the -Xmx option. I bumped it up to 64MB. I get a little further, but
> then
> > it dies whenever it hits the max. I have no control over how big the
> input
> > xml files are. Therefore, I have no way of estimating how much I should
> set
> > the max heap size to.
> >
> > It seems to me like the parser is not releasing the memory that it is
> using.
> > I would have thought that DOM would be more memory intensive and not
SAX.
> > But somehow, I am seeing this behavior using the SAX2 API.
> >
> > Srikanth.
> >
> > -----Original Message-----
> > From: Luke Blanshard [mailto:luke@quiq.com]
> > Sent: Wednesday, November 29, 2000 7:04 PM
> > To: xerces-j-dev@xml.apache.org
> > Subject: Re: Parser runs out of memory parsing large xml files...
> >
> >
> > Srikanth Kurkal wrote:
> >
> > > I am using Xerces 1.2.0 SAX parser and have a simple application that
> just
> > > takes as input an xml file and parses, validates and prints out the
> > > contents. It works fine with small xml files but when I try large
files
> > (>
> > > 50 MB), after a while, the application dies with a
> > > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G
of
> > > memory. Is this a known bug? Is anyone aware of this bug? Is this
> fixed
> > > in a later version of xerces?
> >
> > How much memory you have on your box is irrelevant. What matters is how
> > much
> > memory you have set as your JVM's max. I believe the default is either
16
> > or
> > 32 megs. Try bumping it up with -mx.
> >
> > Of course, this begs the question: why would a SAX application grow so
> > large?
> > I haven't ever mixed SAX with validation, but I'll bet that this
> combination
> > grows with the file for some reason. It does sound like a bug,
actually.
> >
> > Luke
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Yes. The xml file has a whole bunch of elements. I ran the SAXCount
example program that came with the parser on Windows NT on this large xml
file. I think it has more than 80K elements. The text data is actually
very small. Maybe only a few bytes. Of course, not all elements have data.
Some elements nest other elements and so forth.
Srikanth.
-----Original Message-----
From: Eric Ye [mailto:ericye@locus.apache.org]
Sent: Thursday, November 30, 2000 7:32 PM
To: xerces-j-dev@xml.apache.org
Subject: Re: Parser runs out of memory parsing large xml files...
What is your huge XML file looks files? large chunk of text content or lots
of elements and attributes?
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>
Sent: Thursday, November 30, 2000 11:20 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Thanks Luke for your response. I had actually tried you suggestion of
using
> the -Xmx option. I bumped it up to 64MB. I get a little further, but
then
> it dies whenever it hits the max. I have no control over how big the
input
> xml files are. Therefore, I have no way of estimating how much I should
set
> the max heap size to.
>
> It seems to me like the parser is not releasing the memory that it is
using.
> I would have thought that DOM would be more memory intensive and not SAX.
> But somehow, I am seeing this behavior using the SAX2 API.
>
> Srikanth.
>
> -----Original Message-----
> From: Luke Blanshard [mailto:luke@quiq.com]
> Sent: Wednesday, November 29, 2000 7:04 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Srikanth Kurkal wrote:
>
> > I am using Xerces 1.2.0 SAX parser and have a simple application that
just
> > takes as input an xml file and parses, validates and prints out the
> > contents. It works fine with small xml files but when I try large files
> (>
> > 50 MB), after a while, the application dies with a
> > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> > memory. Is this a known bug? Is anyone aware of this bug? Is this
fixed
> > in a later version of xerces?
>
> How much memory you have on your box is irrelevant. What matters is how
> much
> memory you have set as your JVM's max. I believe the default is either 16
> or
> 32 megs. Try bumping it up with -mx.
>
> Of course, this begs the question: why would a SAX application grow so
> large?
> I haven't ever mixed SAX with validation, but I'll bet that this
combination
> grows with the file for some reason. It does sound like a bug, actually.
>
> Luke
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Eric Ye <er...@locus.apache.org>.
What is your huge XML file looks files? large chunk of text content or lots
of elements and attributes?
_____
Eric Ye * IBM, JTC - Silicon Valley * ericye@locus.apache.org
----- Original Message -----
From: "Srikanth Kurkal" <sk...@uucom.com>
To: <xe...@xml.apache.org>
Sent: Thursday, November 30, 2000 11:20 AM
Subject: RE: Parser runs out of memory parsing large xml files...
>
> Thanks Luke for your response. I had actually tried you suggestion of
using
> the -Xmx option. I bumped it up to 64MB. I get a little further, but
then
> it dies whenever it hits the max. I have no control over how big the
input
> xml files are. Therefore, I have no way of estimating how much I should
set
> the max heap size to.
>
> It seems to me like the parser is not releasing the memory that it is
using.
> I would have thought that DOM would be more memory intensive and not SAX.
> But somehow, I am seeing this behavior using the SAX2 API.
>
> Srikanth.
>
> -----Original Message-----
> From: Luke Blanshard [mailto:luke@quiq.com]
> Sent: Wednesday, November 29, 2000 7:04 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Srikanth Kurkal wrote:
>
> > I am using Xerces 1.2.0 SAX parser and have a simple application that
just
> > takes as input an xml file and parses, validates and prints out the
> > contents. It works fine with small xml files but when I try large files
> (>
> > 50 MB), after a while, the application dies with a
> > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> > memory. Is this a known bug? Is anyone aware of this bug? Is this
fixed
> > in a later version of xerces?
>
> How much memory you have on your box is irrelevant. What matters is how
> much
> memory you have set as your JVM's max. I believe the default is either 16
> or
> 32 megs. Try bumping it up with -mx.
>
> Of course, this begs the question: why would a SAX application grow so
> large?
> I haven't ever mixed SAX with validation, but I'll bet that this
combination
> grows with the file for some reason. It does sound like a bug, actually.
>
> Luke
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
This test application is actually trying to isolate the problem. The
original application is actually running as a session bean in WLS. The test
application does not have any database interaction.
Another note of interest. When I ran the application, and it ran for a few
minutes, I ran the top command to check the memory situation. What I
noticed was when I ran the test application with 64M of maximum heap size,
the resident size of the application started at 26M and kept increasing to
78M just before it died with a java.lang.OutOfMemoryError. I made sure
that there are no memory leaks in the application itself. It seems to me
like something in the parser is not releasing memory. I am only guessing at
this point. I am trying to get some profiling tool to actually corroborate
my hunch.
Thanks for your response,
Srikanth.
-----Original Message-----
From: Dennis Thrysoe [mailto:dt@netnord.dk]
Sent: Thursday, November 30, 2000 3:20 PM
To: xerces-j-dev@xml.apache.org
Subject: RE: Parser runs out of memory parsing large xml files...
Just thought I'd mention that we have had a similar problem.
It turned out to be the Oracle JDBC driver though. Just in case.
-dennis
> -----Original Message-----
> From: Srikanth Kurkal [mailto:skurkal@uucom.com]
> Sent: 30. november 2000 20:20
> To: xerces-j-dev@xml.apache.org
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
>
> Thanks Luke for your response. I had actually tried you
> suggestion of using
> the -Xmx option. I bumped it up to 64MB. I get a little
> further, but then
> it dies whenever it hits the max. I have no control over how big
> the input
> xml files are. Therefore, I have no way of estimating how much I
> should set
> the max heap size to.
>
> It seems to me like the parser is not releasing the memory that
> it is using.
> I would have thought that DOM would be more memory intensive and not SAX.
> But somehow, I am seeing this behavior using the SAX2 API.
>
> Srikanth.
>
> -----Original Message-----
> From: Luke Blanshard [mailto:luke@quiq.com]
> Sent: Wednesday, November 29, 2000 7:04 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Srikanth Kurkal wrote:
>
> > I am using Xerces 1.2.0 SAX parser and have a simple
> application that just
> > takes as input an xml file and parses, validates and prints out the
> > contents. It works fine with small xml files but when I try large files
> (>
> > 50 MB), after a while, the application dies with a
> > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> > memory. Is this a known bug? Is anyone aware of this bug? Is
> this fixed
> > in a later version of xerces?
>
> How much memory you have on your box is irrelevant. What matters is how
> much
> memory you have set as your JVM's max. I believe the default is either 16
> or
> 32 megs. Try bumping it up with -mx.
>
> Of course, this begs the question: why would a SAX application grow so
> large?
> I haven't ever mixed SAX with validation, but I'll bet that this
> combination
> grows with the file for some reason. It does sound like a bug, actually.
>
> Luke
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
RE: Parser runs out of memory parsing large xml files...
Posted by Dennis Thrysoe <dt...@netnord.dk>.
Just thought I'd mention that we have had a similar problem.
It turned out to be the Oracle JDBC driver though. Just in case.
-dennis
> -----Original Message-----
> From: Srikanth Kurkal [mailto:skurkal@uucom.com]
> Sent: 30. november 2000 20:20
> To: xerces-j-dev@xml.apache.org
> Subject: RE: Parser runs out of memory parsing large xml files...
>
>
>
> Thanks Luke for your response. I had actually tried you
> suggestion of using
> the -Xmx option. I bumped it up to 64MB. I get a little
> further, but then
> it dies whenever it hits the max. I have no control over how big
> the input
> xml files are. Therefore, I have no way of estimating how much I
> should set
> the max heap size to.
>
> It seems to me like the parser is not releasing the memory that
> it is using.
> I would have thought that DOM would be more memory intensive and not SAX.
> But somehow, I am seeing this behavior using the SAX2 API.
>
> Srikanth.
>
> -----Original Message-----
> From: Luke Blanshard [mailto:luke@quiq.com]
> Sent: Wednesday, November 29, 2000 7:04 PM
> To: xerces-j-dev@xml.apache.org
> Subject: Re: Parser runs out of memory parsing large xml files...
>
>
> Srikanth Kurkal wrote:
>
> > I am using Xerces 1.2.0 SAX parser and have a simple
> application that just
> > takes as input an xml file and parses, validates and prints out the
> > contents. It works fine with small xml files but when I try large files
> (>
> > 50 MB), after a while, the application dies with a
> > java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> > memory. Is this a known bug? Is anyone aware of this bug? Is
> this fixed
> > in a later version of xerces?
>
> How much memory you have on your box is irrelevant. What matters is how
> much
> memory you have set as your JVM's max. I believe the default is either 16
> or
> 32 megs. Try bumping it up with -mx.
>
> Of course, this begs the question: why would a SAX application grow so
> large?
> I haven't ever mixed SAX with validation, but I'll bet that this
> combination
> grows with the file for some reason. It does sound like a bug, actually.
>
> Luke
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
>
>
RE: Parser runs out of memory parsing large xml files...
Posted by Srikanth Kurkal <sk...@uucom.com>.
Thanks Luke for your response. I had actually tried you suggestion of using
the -Xmx option. I bumped it up to 64MB. I get a little further, but then
it dies whenever it hits the max. I have no control over how big the input
xml files are. Therefore, I have no way of estimating how much I should set
the max heap size to.
It seems to me like the parser is not releasing the memory that it is using.
I would have thought that DOM would be more memory intensive and not SAX.
But somehow, I am seeing this behavior using the SAX2 API.
Srikanth.
-----Original Message-----
From: Luke Blanshard [mailto:luke@quiq.com]
Sent: Wednesday, November 29, 2000 7:04 PM
To: xerces-j-dev@xml.apache.org
Subject: Re: Parser runs out of memory parsing large xml files...
Srikanth Kurkal wrote:
> I am using Xerces 1.2.0 SAX parser and have a simple application that just
> takes as input an xml file and parses, validates and prints out the
> contents. It works fine with small xml files but when I try large files
(>
> 50 MB), after a while, the application dies with a
> java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> memory. Is this a known bug? Is anyone aware of this bug? Is this fixed
> in a later version of xerces?
How much memory you have on your box is irrelevant. What matters is how
much
memory you have set as your JVM's max. I believe the default is either 16
or
32 megs. Try bumping it up with -mx.
Of course, this begs the question: why would a SAX application grow so
large?
I haven't ever mixed SAX with validation, but I'll bet that this combination
grows with the file for some reason. It does sound like a bug, actually.
Luke
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
Re: Parser runs out of memory parsing large xml files...
Posted by Luke Blanshard <lu...@quiq.com>.
Srikanth Kurkal wrote:
> I am using Xerces 1.2.0 SAX parser and have a simple application that just
> takes as input an xml file and parses, validates and prints out the
> contents. It works fine with small xml files but when I try large files (>
> 50 MB), after a while, the application dies with a
> java.lang.OutOfMemoryError. I am running this on Solaris 2.7 with 2G of
> memory. Is this a known bug? Is anyone aware of this bug? Is this fixed
> in a later version of xerces?
How much memory you have on your box is irrelevant. What matters is how much
memory you have set as your JVM's max. I believe the default is either 16 or
32 megs. Try bumping it up with -mx.
Of course, this begs the question: why would a SAX application grow so large?
I haven't ever mixed SAX with validation, but I'll bet that this combination
grows with the file for some reason. It does sound like a bug, actually.
Luke