You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-dev@xerces.apache.org by Jeremy Ford <ca...@hotmail.com> on 2001/07/18 16:58:27 UTC
memory usage/performance for xerces-c 1.5.0
I'm involved with a project that is trying to parse xml files of size 20
Megs or greater. We are using xerces-c 1.5.0 xml parser on Windows 2000.
Validation is turned on. We are consistently running out of memory (Windows
2000 reports that we are running out of virtual memory and then programs
promptly start to crash as they can't allocate more) around the 3rd or 4th
file. As a test, I decided to run the parser repeatedly with one 3kb file
instead. At first, the parser went very quickly, parsing the same file
approximately 1000 times in 1 minute. After letting it run for several
hours (10+), it now takes the parser over 3 minutes to parse the file.
Also, using the task manager, I noticed that memory usage peaks at 190,000 K
while parsing the file over the 3 minute period.
My assumptions:
When parsing over a 3k file, memory should increase to some small finite
level and then go back down when parsing is finished (but not to previous
level due to overhead). This should happen repeatedly. I didn't expect to
see 190,000K and the slow down.
XML Description:
Very simplistic. Root node contains multiple nodes that are PCData. It
also contains a few nodes that have children. The children are PCData as
well.
This leads me to the following questions.
1) How large of a file can xerces handle?
2) Is this the kind of performance I should expect after continuous
parsing?
3) Is there anything that I can do differently? Is there anything that I
should be calling that I'm not.
Here is the code that I'm using.
void main(int argc, char* argv[])
{
XMLPlatformUtils::Initialize();
for(;;)
{
DOMParser parser;
parser.setDoValidation(true);
parser.setIncludeIgnorableWhitespace(false);
parser.setErrorHandler(new XmlErrorReporter());
parser.parse("test.xml");
DOM_Document doc = parser.getDocument();
DOM_Element root = doc.getDocumentElement();
}
XMLPlatformUtils::Terminate();
return 0;
}
_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
Re: memory usage/performance for xerces-c 1.5.0
Posted by Tinny Ng <tn...@ca.ibm.com>.
There is a known memory leak in Xerces 1.5. A bug-fix release Xerces 1.5.1 is
going to be released soon (within these two days). You can try our latest
nightly build http://xml.apache.org/dist/xerces-c/nightly/2001-07-17/ which is
the candidate for Xerces 1.5.1 and see if this solves your problem.
Tinny
Jeremy Ford wrote:
> I'm involved with a project that is trying to parse xml files of size 20
> Megs or greater. We are using xerces-c 1.5.0 xml parser on Windows 2000.
> Validation is turned on. We are consistently running out of memory (Windows
> 2000 reports that we are running out of virtual memory and then programs
> promptly start to crash as they can't allocate more) around the 3rd or 4th
> file. As a test, I decided to run the parser repeatedly with one 3kb file
> instead. At first, the parser went very quickly, parsing the same file
> approximately 1000 times in 1 minute. After letting it run for several
> hours (10+), it now takes the parser over 3 minutes to parse the file.
> Also, using the task manager, I noticed that memory usage peaks at 190,000 K
> while parsing the file over the 3 minute period.
>
> My assumptions:
>
> When parsing over a 3k file, memory should increase to some small finite
> level and then go back down when parsing is finished (but not to previous
> level due to overhead). This should happen repeatedly. I didn't expect to
> see 190,000K and the slow down.
>
> XML Description:
>
> Very simplistic. Root node contains multiple nodes that are PCData. It
> also contains a few nodes that have children. The children are PCData as
> well.
>
> This leads me to the following questions.
>
> 1) How large of a file can xerces handle?
> 2) Is this the kind of performance I should expect after continuous
> parsing?
> 3) Is there anything that I can do differently? Is there anything that I
> should be calling that I'm not.
>
> Here is the code that I'm using.
>
> void main(int argc, char* argv[])
> {
> XMLPlatformUtils::Initialize();
>
> for(;;)
> {
> DOMParser parser;
> parser.setDoValidation(true);
> parser.setIncludeIgnorableWhitespace(false);
> parser.setErrorHandler(new XmlErrorReporter());
> parser.parse("test.xml");
>
> DOM_Document doc = parser.getDocument();
> DOM_Element root = doc.getDocumentElement();
> }
>
> XMLPlatformUtils::Terminate();
> return 0;
> }
>
> _________________________________________________________________
> Get your FREE download of MSN Explorer at http://explorer.msn.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org
RE: memory usage/performance for xerces-c 1.5.0
Posted by Evert Haasdijk <ev...@zukkespijkers.nl>.
Ehmm, Jeremy, you're creating a new XmlErrorReporter() in every iteration;
maybe you should try deleting that?
> -----Original Message-----
> From: Jeremy Ford [mailto:caius1440@hotmail.com]
> Sent: Wednesday, July 18, 2001 16:58 PM
> To: xerces-c-dev@xml.apache.org
> Subject: memory usage/performance for xerces-c 1.5.0
>
SINPPETY SNIP
>
> void main(int argc, char* argv[])
> {
> XMLPlatformUtils::Initialize();
>
> for(;;)
> {
> DOMParser parser;
> parser.setDoValidation(true);
> parser.setIncludeIgnorableWhitespace(false);
> parser.setErrorHandler(new XmlErrorReporter());
^^^^^^^^^^^^^^^^^^^^^^^ Aha!
> parser.parse("test.xml");
>
> DOM_Document doc = parser.getDocument();
> DOM_Element root = doc.getDocumentElement();
> }
>
> XMLPlatformUtils::Terminate();
> return 0;
> }
>
Ta, Evert
---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-c-dev-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-c-dev-help@xml.apache.org