Re: Preliminary DOM parser perf numbers and proposal

[Mike Pogue <mp...@apache.org>]

No, we have to do it on the same exact systems, or the results can't be compared.

Trust me on this one.  Which CPU you have (e.g. how much L2 cache you have, etc.)
often makes a big difference in performance of Java programs!  So, you can't
run Hotspot on a PII, and the Symantec JIT on a different PII, and expect to
be able to compare the results....

Anybody running perf tests needs to run both JVM's on the same exact machine
configuration.  Same memory, same disk, same CPU, everything.

Sorry, I know this is a pain.  But if we don't do it, the results are literally
meaningless for making performance decisions (I know this from hard lessons 
learned in this area...).

Mike

Rajiv Mordani wrote:
> 
> Ok why don't we do something of the sort - you run all the tests on win32
> and linux platforms using the IBM JVM and we will run it using the Sun VM
> on win32 and solaris on 1.1.8 and 1.2 . Post all the results that you get
> along with the platform / processor and VM info on some site and then take
> it from there.. Does that sound like a plan?? Tell me what you think about
> it...
> 
> - Rajiv
> 
> --
> :wq
> 
> On Tue, 30 May 2000, Mike Pogue wrote:
> 
> > Yep...what I'm more concerned about is that we're getting completely
> > different results when we change from the Sun/Hotspot JVM to the IBM/Tokyo JIT
> > or the Sun/Symantec JIT JVM.  (This is on the same tests that Edwin is running,
> > I think!)
> >
> > If these results can be duplicated by Edwin, that means that we can't trust
> > any performance tuning, because it will help one JVM, but hurt a different one!
> > In other words, we end up tuning for quirks of the JIT, rather than tuning
> > the code overall!
> >
> > ERGH!
> >
> > Mike
> >
> > Rajiv Mordani wrote:
> > >
> > > The tests that are being run are in the metrics directory.. The whole
> > > purpose of creating the metrics workspace was to place all the performance
> > > tests in that so anyone can take it and run them. If Arnaud / anyone else
> > > has any tests that show performance nos they should put it in the metrics
> > > directory and if they want to take a look at the tests that are developed
> > > by us, it is already in there. If anyone is running tests (maybe the ones
> > > in metrics / any other) then maybe posting a similar kind of result like
> > > Ed's would help understand the differences (whether it is different VM /
> > > platform / intial conditions / geometric mean etc.) and then we can take
> > > it from there. Mike since you have already been part of the Java
> > > performance group any help from you / anyone in your group would be highly
> > > appreciated.
> > >
> > > - Rajiv
> > >
> > > --
> > > :wq
> > >
> > > On Fri, 26 May 2000, Mike Pogue wrote:
> > >
> > > > Can I make a suggestion?
> > > >
> > > > Could you guys get together, and swap your exact tests and build info?
> > > > It still appears to me that we're comparing apples and oranges here.
> > > >
> > > > Mike
> > > >
> > > > Arnaud Le Hors wrote:
> > > > >
> > > > > Edwin Goei wrote:
> > > > > >
> > > > > > In this case since we are running a single benchmark, I think an arithmetic
> > > > > > mean is appropriate.  Programs such as x11perf use arithmetic means in a
> > > > > > similar way.  Geometric means are used for collections of programs that
> > > > > > comprise a single benchmark number such as for Specmarks or to compute
> > > > > > relative performance ratios.  In any case, I don't think it probably makes
> > > > > > much difference in this case which is used since the values are so close.
> > > > >
> > > > > As I understand the advantage of the geometric mean is that it would
> > > > > basically discard results that are completely off.
> > > > >
> > > > > > > These tests had no
> > > > > > > warmup phase, I think they should, the numbers for the first trial
> > > > > > > always are completely off. Finally, while you're setting the maximum
> > > > > > > heap size you're not setting the startup size.
> > > > > >
> > > > > > It may not have been clear, but in the "Explaination" section, it says that
> > > > > > I run 100 iterations and then look at the output until the total memory size
> > > > > > stabilizes.  On Sun's JDK1.2.2 at least, the gc increases the total heap
> > > > > > size and then stops once it is large enough.  I then throw out the initial
> > > > > > iterations and average the rest.
> > > > >
> > > > > Where is this being done?? I don't see anything that fancy in your
> > > > > program DOMCountPerf. Your average time takes into account every trial
> > > > > after the warmup.
> > > > >
> > > > > > So this is equivalent to setting a minimum
> > > > > > startup heap size and also having a warmup phase.  See
> > > > > > http://xml.apache.org/~edwingo/.
> > > > >
> > > > > I don't understand how this can be.
> > > > >
> > > > > > I have in the past looked at some xerces code, but I am not as familiar with
> > > > > > it because of time constraints.  I think that merging two equivalent atomic
> > > > > > components that do the same thing isn't realistic, while composing separable
> > > > > > atomic components to create a larger component is possible.  If replacing
> > > > > > the current non-deferred DOM is impractical, then what about adding crimson
> > > > > > as an alternate first class implementation?
> > > > >
> > > > > One thing I'd also like to understand is what is it that you're calling
> > > > > crimson (parser+DOM or just DOM?), and what is the crimson we're testing
> > > > > here. My understanding was that whiteboard/crimson is a modified version
> > > > > of crimson DOM changed to work with xerces parser. But I don't think
> > > > > this is what's being used in this test, since you're using
> > > > > xml-contrib/crimson which I think includes both a parser and a DOM. I'm
> > > > > confused.
> > > > >
> > > > > > One thing I have noticed is
> > > > > > that the crimson DOM currently uses less memory.  Maybe some users would be
> > > > > > interested in that.
> > > > >
> > > > > Again, based on my analysis of the code of the two DOM implementations,
> > > > > apart from the array vs linked list, I don't see anything in crimson DOM
> > > > > that could lead to an improvement in xerces. In many respects xerces DOM
> > > > > is far more optimized than crimson DOM. I believe the difference in size
> > > > > merely comes from the difference in features being supported. This said,
> > > > > I completely agree that there is probably a need for a DOM
> > > > > implementation that's not fully featured. I'd be happy to discuss what
> > > > > we can do here. I've already given it quite some thought.
> > > > >
> > > > > > > Also, at this point, I'm wondering if this comparison is relevant in any
> > > > > > > way. We're comparing apples and oranges here. Xerces DOM is DOM Level 2
> > > > > > > compliant with support for mutation events, and traversal. Each of these
> > > > > > > adds to the size used and to the time taken. Although I'm not so much
> > > > > > > familiar with the paser, I think the same is true since xerces parser
> > > > > > > now supports XML Schemas.
> > > > > >
> > > > > > Actually, the current version of crimson I tested supports DOM Level2 core
> > > > > > (or most of it).  I was under the impression that the other Level 2
> > > > > > extensions such as Traversal were built on top of core so we could use the
> > > > > > current implementations of those.
> > > > >
> > > > > No, supporting mutation events for instance cost both in memory and in
> > > > > speed throughout the DOM.
> > > > >
> > > > > >  As for XML Schemas, does this affect the
> > > > > > DOMParser code?  I would expect only the lower levels such as SAXParser
> > > > > > would be affected.  The last time I looked at xerces code was before the
> > > > > > latest XML schema code was checked in, though.
> > > > > >
> > > > > > > But, I encourage you to try any merge/reuse you'd like on a branch. Once
> > > > > > > we've got two implementations with the same set of features we'll really
> > > > > > > be able to compare.
> > > > > >
> > > > > > Right now, I working on making crimson more compliant with DOM2 core.
> > > > >
> > > > > We need to get a plan on how to progress on this. I don't think there is
> > > > > much value in developing a second project in parallel.
> > > > > --
> > > > > Arnaud  Le Hors - IBM Cupertino, XML Technology Group
> > > > >
> > > > > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> > For additional commands, e-mail: xerces-j-dev-help@xml.apache.org
> >
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: xerces-j-dev-unsubscribe@xml.apache.org
> For additional commands, e-mail: xerces-j-dev-help@xml.apache.org

Re: Preliminary DOM parser perf numbers and proposal

[co...@costin.dnt.ro]

Hi,

What about some micro-performance tests - if we want to figure why the
performance is so different on different VMs. It seems the StringPool and
SymbolCache ( and few other ) classes are very used - and it's one point
where the JIT does make a big difference. 

Some JITs are very sensitive to try { } catch(), and all string and object
manipulation uses  int[] or char[] - and it seems ArrayOutOfBounds is used
to re-allocate. I noticed that some classes have a switch to allow 
"pos<array.length" style.

I'll try something after J1, I'm very interested in finding out how
useful/harmful is this optimization.

BTW, the int[][] and the chunks are great !

Costin


On Wed, 31 May 2000, Mike Pogue wrote:

> No, we have to do it on the same exact systems, or the results can't be compared.
> 
> Trust me on this one.  Which CPU you have (e.g. how much L2 cache you have, etc.)
> often makes a big difference in performance of Java programs!  So, you can't
> run Hotspot on a PII, and the Symantec JIT on a different PII, and expect to
> be able to compare the results....
> 
> Anybody running perf tests needs to run both JVM's on the same exact machine
> configuration.  Same memory, same disk, same CPU, everything.