Re: Benchmarks (XalanC + XalanJ)

[Holger Flörke <fl...@doctronic.de>]

Hi David,

david_n_bertoni@us.ibm.com schrieb:
>    1. Xalan-C might not do well on some of the tests performed.
The point is, Xalan-C 1.3 is slower than saxon in 14 of 15 (nearly all) tests.

>    2. Xalan-C has not seen much work lately, and there are lots of
>    optimizations in Xalan-J which could be applied which would help improve
>    performance.
In the study Xalan-J 2.3.1 is slower than saxon in all the tests and slower than 
Xalan-C 1.3 in 13 of 15 tests.

>    3. They may have chosen a bad platform for Xalan-C.  Xalan-C is
>    sensitive to the quality of code generated by the compiler for a
>    particular platform, and some platforms don't do well.  For example,GCC
>    on Linux does not always do a good job optimizing.
>    4. Older versions of Xalan-C may not produce the same results that newer
>    versions do.  If an older version is used for the benchmark, the results
>    may not be as good as they would be if a newer version were used.
They have tested Xalan-C 1.3, the benchmark is available (http://www.sarvega.com). I 
run the saxon/xalanc tests on my Win32 (W2KSP3) machine (PIII 800MHz) with Xalan-C 
1.5. I found Xalan-C 1.5 is *faster than saxon* in 10 of 15 tests. This may be the 
result of a) the win32 platform, b) the newer version of Xalan-C or c) the slower 
processor. Maybe I run some other tests to answer this question.

> It's very curious, because there are such conflicting reports.  Dmitre
> Novatchev stated in was unfortunate that he couldn't get timing information
> out of Xalan-C, because he felt it was one of the fastest processors.
> Others have also stated they've switched from Xalan-J to Xalan-C for speed
> reasons.
Hey, don't question yourself or Xalan-C. I like Xalan-C. I had written an article for 
a german XML magazin (Holger Floerke, "XML-Verarbeitung in C++", Der Entwickler (XML 
Magazin), 1/2002, Software&Support Verlag - no English version available). The 
conclusion is: If you have to transform *large* documents (espacially for converting 
publikation data), you *have to* use Xalan-C because of the small memory usage. 
Xalan-C is good, small, and fast.

Maybe such benchmarks can help us to concentrate on real bottlenecks. In this case 
there is one test, where Xalan-C achieves only 17% of the performance of saxon.

> P.S. Is there an online version of this article, perhaps translated into
> English?
There is only one picture of the article (an errata) online:

http://www.heise.de/ix/artikel/2003/07/104/xslt-proz.bench-korr.tiff
On the left side you see the different processors (without Xalan-C). The x-axis is 
performance relative to saxon/linux.

HolgeR
-- 
holger floerke                      d  o  c  t  r  o  n  i  c
email floerke@doctronic.de          information publishing + retrieval
phone +49 2222 9292 90              http://www.doctronic.de

Re: Benchmarks (XalanC + XalanJ)

[da...@us.ibm.com]

Hi HolgeR,

> Hi David,
>
> david_n_bertoni@us.ibm.com schrieb:
> >    1. Xalan-C might not do well on some of the tests performed.
> The point is, Xalan-C 1.3 is slower than saxon in 14 of 15 (nearly all)
tests.

Hmmm, I guess I need to take a look at this.

> >    2. Xalan-C has not seen much work lately, and there are lots of
> >    optimizations in Xalan-J which could be applied which would help
improve
> >    performance.
> In the study Xalan-J 2.3.1 is slower than saxon in all the tests and
slower than
> Xalan-C 1.3 in 13 of 15 tests.

Those are both old versions of Xalan, so that might explain it.  There are
lots of new optimizations that have been introduced into Xalan-J since that
version.

> >    3. They may have chosen a bad platform for Xalan-C.  Xalan-C is
> >    sensitive to the quality of code generated by the compiler for a
> >    particular platform, and some platforms don't do well.  For
example,GCC
> >    on Linux does not always do a good job optimizing.
> >    4. Older versions of Xalan-C may not produce the same results that
newer
> >    versions do.  If an older version is used for the benchmark, the
results
> >    may not be as good as they would be if a newer version were used.
> They have tested Xalan-C 1.3, the benchmark is available (
http://www.sarvega.com). I
> run the saxon/xalanc tests on my Win32 (W2KSP3) machine (PIII 800MHz)
with Xalan-C
> 1.5. I found Xalan-C 1.5 is *faster than saxon* in 10 of 15 tests. This
may be the
> result of a) the win32 platform, b) the newer version of Xalan-C or c)
the slower
> processor. Maybe I run some other tests to answer this question.

Probably a combination of a and b.

> > It's very curious, because there are such conflicting reports.  Dmitre
> > Novatchev stated in was unfortunate that he couldn't get timing
information
> > out of Xalan-C, because he felt it was one of the fastest processors.
> > Others have also stated they've switched from Xalan-J to Xalan-C for
speed
> > reasons.
> Hey, don't question yourself or Xalan-C. I like Xalan-C. I had written an
article for
> a german XML magazin (Holger Floerke, "XML-Verarbeitung in C++", Der
Entwickler (XML
> Magazin), 1/2002, Software&Support Verlag - no English version
available). The
> conclusion is: If you have to transform *large* documents (espacially for
converting
> publikation data), you *have to* use Xalan-C because of the small memory
usage.
> Xalan-C is good, small, and fast.

Don't worry -- I'm not questioning myself or whether Xalan-C has any value,
it's simply that the Xalan-C team has not had the resources in the past to
do many things I would like to do. There is also value in being a
cross-platform C++ processor, which MSXSL will never be.  It's good to know
you confirm what I've seen -- that Xalan-C's memory footprint is much
smaller than the Java processors.

Is there, by any chance, an on-line version of your article?  I would love
to read it, even in German.

> Maybe such benchmarks can help us to concentrate on real bottlenecks. In
this case
> there is one test, where Xalan-C achieves only 17% of the performance of
saxon.

Absolutely.  I will look at this as soon as I can.

Dave