Re: Performance of XSLT engines - remember the saying about statistics...

[Lo...@lotus.com]

Be sure to evaluate carefully *how* each benchmark gets it's statistics,
and *what* they're processing, and ensure they're doing transformations the
same way you will need to in your application.  The devil is often in the
details, especially when you want to get real results out of someone's
performance numbers.  Caveat emptor.

I've seen benchmarks that simply call each processor on the command line -
probably not a very common practice in production environments, where you
simply create a processor (or a 'transformer') in memory and hold it around
for multiple processes or stylesheets.  Some benchmarks use very simple
sets of data files and stylesheets, and there are big performance
differences (and not always the same differences in various processors)
with even subtly different kinds of stylesheets.  And one benchmark
compared the time for one processor using a previously-prepared stylesheet
with a processor that had to parse and prepare the stylesheet within the
timing loop - clearly an unfair comparison (which may have been due to
unfamiliarity about the processor's API's in that case).

Getting good performance measurements is hard.  It may be especially hard
to get fair benchmarks between processors that have somewhat different
usage models that may not appear to matchup - especially for developers who
aren't familiar with the source code.

Also, it's hard to get a set of truly representative stylesheets to run
through the processors - we're still looking for suggestions here!

- Shane