You are viewing a plain text version of this content. The canonical link for it is here.
Posted to c-users@xerces.apache.org by Je...@t-systems.com on 2010/05/27 11:56:12 UTC
Performance Issues between 2.8 and 3.1.0/3.1.1
Hi Xerces-C List,
I did port a program from 2.8 to 3.1/1.
Its a large program, and I spare you the details, there is a Database, and large XML Files get importet through Xerces.
Where large means 50 > 500 mb Files. My Testfile is about 150 mb.
The program runs under Solaris8 and XP, I refer mostly to Solaris, as I could test there more easily, but I think that this is not an Solaris only issue.
We hoped to improve the performance of the importprocess, that would be a reason to switch to 3.x series.
For Windows I have only found, that now the process uses about 50 - 100 mb less memory, I haven't checked that under solaris yet.
But I have noticed, that at least under Solaris, the 2.8 Version is faster. While 2.8 takes 18-19 Minutes to import all Data, with 3.1 it takes 19 - 21 Minutes. I know that this isn't the right testcase to do performance reviews about both versions, but I have run multiple imports with both versions, and the 2.8 version is taking less time then 3.1.x.
The programs are the same, except the linkage against the xerces versions.
Do people have similar expierence with this?
Are there maybe some options in 3.1.x or in general to speed up parsing of large files?
Regards,
Jens Weller
AW: Performance Issues between 2.8 and 3.1.0/3.1.1
Posted by Je...@t-systems.com.
Hi,
well, as I said, the program is complex, and the 20 Minutes aren't fully spend on the xml file, its more or less a continueing process, reading from the file and writing results to the DB. Also we use the full featureset, Validation, and have our own XML Template format, which knows when to find what. I only wonder, why the version with 3.x is a Minute slower.
Regards,
Jens Weller
-----Ursprüngliche Nachricht-----
Von: Jones, Monte J CTR USAF AFSPC SLG/WMLE [mailto:Monte.Jones.ctr@Peterson.af.mil]
Gesendet: Donnerstag, 27. Mai 2010 15:21
An: c-users@xerces.apache.org
Betreff: RE: Performance Issues between 2.8 and 3.1.0/3.1.1
Jens,
I would recommend checking your system settings and configuration with
your system administrator. I am using Xerces 2.8.x, Solaris 10, and old
Sun 4800 hardware with 900 MHz processors (end of life on it was
6/2005). I ingest 100-150 MB files and process them for the intelligent
information they contain in approximately 100 seconds. We do have plans
to migrate to 3.1.x later this summer, after we get upgraded hardware.
Monte J. Jones, Contractor
Colorado Professional Resources
SMC/SLG/WMLE
1050 E Stewart Ave
Colorado Springs, CO 80914
719-556-9903
This electronic message contains information from Colorado Professional
Resources, LLC and is confidential or privileged. The information is
intended solely for the use of the individual(s) or entity(ies) named
above. If you are not the intended recipient, be aware that any
disclosure, copying, distribution, or use of the contents of this
message is prohibited. If you have received this email in error, please
notify us immediately by telephone at (719) 599-1353, or by email and
then immediately delete this message.
-----Original Message-----
From: Jens.Weller@t-systems.com [mailto:Jens.Weller@t-systems.com]
Sent: Thursday, May 27, 2010 3:56 AM
To: c-users@xerces.apache.org
Subject: Performance Issues between 2.8 and 3.1.0/3.1.1
Hi Xerces-C List,
I did port a program from 2.8 to 3.1/1.
Its a large program, and I spare you the details, there is a Database,
and large XML Files get importet through Xerces.
Where large means 50 > 500 mb Files. My Testfile is about 150 mb.
The program runs under Solaris8 and XP, I refer mostly to Solaris, as I
could test there more easily, but I think that this is not an Solaris
only issue.
We hoped to improve the performance of the importprocess, that would be
a reason to switch to 3.x series.
For Windows I have only found, that now the process uses about 50 - 100
mb less memory, I haven't checked that under solaris yet.
But I have noticed, that at least under Solaris, the 2.8 Version is
faster. While 2.8 takes 18-19 Minutes to import all Data, with 3.1 it
takes 19 - 21 Minutes. I know that this isn't the right testcase to do
performance reviews about both versions, but I have run multiple imports
with both versions, and the 2.8 version is taking less time then 3.1.x.
The programs are the same, except the linkage against the xerces
versions.
Do people have similar expierence with this?
Are there maybe some options in 3.1.x or in general to speed up parsing
of large files?
Regards,
Jens Weller
RE: Performance Issues between 2.8 and 3.1.0/3.1.1
Posted by "Jones, Monte J CTR USAF AFSPC SLG/WMLE" <Mo...@Peterson.af.mil>.
Jens,
I would recommend checking your system settings and configuration with
your system administrator. I am using Xerces 2.8.x, Solaris 10, and old
Sun 4800 hardware with 900 MHz processors (end of life on it was
6/2005). I ingest 100-150 MB files and process them for the intelligent
information they contain in approximately 100 seconds. We do have plans
to migrate to 3.1.x later this summer, after we get upgraded hardware.
Monte J. Jones, Contractor
Colorado Professional Resources
SMC/SLG/WMLE
1050 E Stewart Ave
Colorado Springs, CO 80914
719-556-9903
This electronic message contains information from Colorado Professional
Resources, LLC and is confidential or privileged. The information is
intended solely for the use of the individual(s) or entity(ies) named
above. If you are not the intended recipient, be aware that any
disclosure, copying, distribution, or use of the contents of this
message is prohibited. If you have received this email in error, please
notify us immediately by telephone at (719) 599-1353, or by email and
then immediately delete this message.
-----Original Message-----
From: Jens.Weller@t-systems.com [mailto:Jens.Weller@t-systems.com]
Sent: Thursday, May 27, 2010 3:56 AM
To: c-users@xerces.apache.org
Subject: Performance Issues between 2.8 and 3.1.0/3.1.1
Hi Xerces-C List,
I did port a program from 2.8 to 3.1/1.
Its a large program, and I spare you the details, there is a Database,
and large XML Files get importet through Xerces.
Where large means 50 > 500 mb Files. My Testfile is about 150 mb.
The program runs under Solaris8 and XP, I refer mostly to Solaris, as I
could test there more easily, but I think that this is not an Solaris
only issue.
We hoped to improve the performance of the importprocess, that would be
a reason to switch to 3.x series.
For Windows I have only found, that now the process uses about 50 - 100
mb less memory, I haven't checked that under solaris yet.
But I have noticed, that at least under Solaris, the 2.8 Version is
faster. While 2.8 takes 18-19 Minutes to import all Data, with 3.1 it
takes 19 - 21 Minutes. I know that this isn't the right testcase to do
performance reviews about both versions, but I have run multiple imports
with both versions, and the 2.8 version is taking less time then 3.1.x.
The programs are the same, except the linkage against the xerces
versions.
Do people have similar expierence with this?
Are there maybe some options in 3.1.x or in general to speed up parsing
of large files?
Regards,
Jens Weller
AW: Performance Issues between 2.8 and 3.1.0/3.1.1
Posted by Je...@t-systems.com.
Hi Boris,
> > Do people have similar expierence with this?
>
> A number of optimizations have been done for Xerces-C++ 3-series so,
> at least on our benchmarks (parsing without validation, parsing with
> XML Schema validation, and serialization), it is at least 20-30% faster
> than 2-series.
Good to know :)
> > Are there maybe some options in 3.1.x or in general to speed up parsing
> > of large files?
> In Xerces-C++ 3-series the build system has changed to autoconf/automake.
> As a result, you may be using different optimization options when building
> the two versions. For example, by default with the GCC compiler Xerces-C++
> 3-series is built with -O2 -g while 2-series with -O3, if I remember
> correctly. So I suggest that you explicitly pass optimization flags to
> the 3-series build during configure:
>
>./configure CFLAGS=-O3 CXXFLAGS=-O3 ...
Ah, thats an interesting point. But I do build with sunworkshoppro under solaris, so I have to see, which optimization features I do have there.
> Let us know if you see any difference with this.
I have compiled now the library with -xO3, that makes the .so file a lot smaller (about 15%).
I have to do a lot of testing now, to see what the impact is on my program.
The first two testruns were below 20 Minutes, which is a good sign sofar.
regards,
Jens Weller
--
Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog
Open-source XML data binding for C++ http://codesynthesis.com/products/xsd
XML data binding for embedded systems http://codesynthesis.com/products/xsde
Command line interface to C++ compiler http://codesynthesis.com/projects/cli
Re: Performance Issues between 2.8 and 3.1.0/3.1.1
Posted by Boris Kolpackov <bo...@codesynthesis.com>.
Hi Jens,
Jens.Weller@t-systems.com <Je...@t-systems.com> writes:
> Do people have similar expierence with this?
A number of optimizations have been done for Xerces-C++ 3-series so,
at least on our benchmarks (parsing without validation, parsing with
XML Schema validation, and serialization), it is at least 20-30% faster
than 2-series.
> Are there maybe some options in 3.1.x or in general to speed up parsing
> of large files?
In Xerces-C++ 3-series the build system has changed to autoconf/automake.
As a result, you may be using different optimization options when building
the two versions. For example, by default with the GCC compiler Xerces-C++
3-series is built with -O2 -g while 2-series with -O3, if I remember
correctly. So I suggest that you explicitly pass optimization flags to
the 3-series build during configure:
./configure CFLAGS=-O3 CXXFLAGS=-O3 ...
Let us know if you see any difference with this.
Boris
--
Boris Kolpackov, Code Synthesis http://codesynthesis.com/~boris/blog
Open-source XML data binding for C++ http://codesynthesis.com/products/xsd
XML data binding for embedded systems http://codesynthesis.com/products/xsde
Command line interface to C++ compiler http://codesynthesis.com/projects/cli