Processing speed slow down for large files

["Conley, Daryl" <Da...@ccra-adrc.gc.ca>]

Hello, 

   We have an application that processes from very small to very large XML
files.  Up to this point we have not had two many issues with the SAX parser
in Xerces. Unfortunately we are going to need to process very large files,
so I have been running some tests to see how the performance is.  I have
found that I can parse a file up to about 1.15 Gig with a quarter of the
records with invalid data (to simulate the bad data we will be getting) and
it processes in about 1.5 hours, a file with the same error rate but 1.2 Gig
in size takes 27.5 hours.  These are running under WebLogic on a Sun dual
processor server.  I have tested the same file but our process did not
output anything because I was testing to see if the IO was slowing the
process but it still took 27.5 hours.   What could be the cause of such an
abrupt change in processing time??  Is there any tools that I could use to
see what is going on under the hood?  By the way a clean file would take
about an hour to process.

Any help would be appreciated.

Thanks

Daryl

---------------------------------------------------------------------
To unsubscribe, e-mail: xerces-j-user-unsubscribe@xml.apache.org
For additional commands, e-mail: xerces-j-user-help@xml.apache.org