You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Stefano Mazzocchi <st...@apache.org> on 2006/11/17 06:26:17 UTC
[performance] a few early benchmarks
There are lies, damn lies and benchmarks.... which don't really tell you
if an implementation of a program is *faster* but at least it tells you
where you're at.
So, as Geir managed to get the DSO linking problem go away in DRLVM, I
was able to start running some benchmarks.
The machine is the following:
Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
01:50:50 UTC 2006 x86_64 GNU/Linux
dual Intel(R) Pentium(R) D CPU 3.20GHz
bogomips 6410.31 (per CPU)
There is nothing else running on the machine (load is 0.04 at the time
of testing).
The various virtual machines tested are:
harmony
-------
Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
Foundation or its licensors, as applicable.
java version "1.5.0"
pre-alpha : not complete or compatible
svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
sun5
---
java version "1.5.0_09"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
sun6
----
java version "1.6.0-rc"
Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
ibm
---
java version "1.5.0"
Java(TM) 2 Runtime Environment, Standard Edition (build
pxa64dev-20061002a (SR3) )
IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
j9vmxa6423-20061001 (JIT enabled)
J9VM - 20060915_08260_LHdSMr
JIT - 20060908_1811_r8
GC - 20060906_AA)
JCL - 20061002
bea
---
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
BEA JRockit(R) (build
R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
--------------------------------------------------------------------------
Test #1: java scimark2 (http://math.nist.gov/scimark2/)
command: java jnt.scimark2.commandline
NOTE: bigger number is better
Sun6
Composite Score: 364.5832265230057
FFT (1024): 220.8458713892794
SOR (100x100): 696.1542342357722
Monte Carlo : 149.37978088875656
Sparse matmult (N=1000, nz=5000): 326.37451873283845
LU (100x100): 430.1617273683819
BEA
Composite Score: 359.13480378697835
FFT (1024): 303.8746880751562
SOR (100x100): 454.25628897202307
Monte Carlo : 93.23913192138497
Sparse matmult (N=1000, nz=5000): 530.44112637391
LU (100x100): 413.8627835924175
Sun5
Composite Score: 332.84987587548574
FFT (1024): 216.5144595799027
SOR (100x100): 689.429322146947
Monte Carlo : 25.791262124978065
Sparse matmult (N=1000, nz=5000): 317.5193965699373
LU (100x100): 414.99493895566377
IBM
Composite Score: 259.8249218693683
FFT (1024): 296.8415012789055
SOR (100x100): 428.974881649179
Monte Carlo : 89.15159857584082
Sparse matmult (N=1000, nz=5000): 144.3524241203982
LU (100x100): 339.8042037225181
Harmony
Composite Score: 113.65082278962575
FFT (1024): 203.76641991778123
SOR (100x100): 224.37761309236748
Monte Carlo : 9.063866256533116
Sparse matmult (N=1000, nz=5000): 65.4051866327227
LU (100x100): 65.6410280487242
In this test harmony is clearly lagging behind... at about 30%
performance of the best JVM, it's a little crappy. Please note how FFT's
performance is not so bad awhile monte carlo is pretty bad compared to
BEA or IBM.
Overall, it seems like there is some serious work to do here to catch up.
--------------------------------------------------------------------------
Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
command: java dhry 100000000
NOTE: bigger is better
NB: I modified the code to accept the count at input from the command line!
sun6: 8552856 dhrystones/sec
sun5: 6605892
bea: 5678914
harmony: 669734
ibm: 501562
The performance here is horrific but what's surprising is that J9 is
even worse. No idea what's going on but it seems like something is not
working as it should (in both harmony and J9)
--------------------------------------------------------------------------
Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
command: java Sieve 30
NB: I modified the test to run for a configurable amount of seconds.
sun6 8545 sieves/sec
sun5 8364
bea 6174
harmony 1836
ibm 225
IBM J9 clearly has something wrong on x86_64 but harmony is clearly
lagging behind.
Stay tuned for more tests.
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Pavel Ozhdikhin <pa...@gmail.com>.
+1
I'm even impressed how good is Harmony performance shown in debug mode! :)
Also, by default DRLVM is tuned for client workloads - fast startup and
reasonable performance.
If you run "heavy" benchmarks it worth to use "-Xem:server" or
"-Xem:server_static" mode to determine the potential of DRLVM.
thanks,
Pavel
On 11/17/06, Alexey Varlamov <al...@gmail.com> wrote:
>
> Stefano,
>
> It is a bit unfair to compare *debug* build of Harmony with other
> release versions :)
> I suppose all VMs where run in default mode (i.e. no special cmd-line
> switches)?
>
> 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> > There are lies, damn lies and benchmarks.... which don't really tell you
> > if an implementation of a program is *faster* but at least it tells you
> > where you're at.
> >
> > So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> > was able to start running some benchmarks.
> >
> > The machine is the following:
> >
> > Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> > 01:50:50 UTC 2006 x86_64 GNU/Linux
> >
> > dual Intel(R) Pentium(R) D CPU 3.20GHz
> > bogomips 6410.31 (per CPU)
> >
> > There is nothing else running on the machine (load is 0.04 at the time
> > of testing).
> >
> > The various virtual machines tested are:
> >
> > harmony
> > -------
> > Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> > Foundation or its licensors, as applicable.
> > java version "1.5.0"
> > pre-alpha : not complete or compatible
> > svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >
> > sun5
> > ---
> > java version "1.5.0_09"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >
> > sun6
> > ----
> > java version "1.6.0-rc"
> > Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >
> > ibm
> > ---
> > java version "1.5.0"
> > Java(TM) 2 Runtime Environment, Standard Edition (build
> > pxa64dev-20061002a (SR3) )
> > IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > j9vmxa6423-20061001 (JIT enabled)
> > J9VM - 20060915_08260_LHdSMr
> > JIT - 20060908_1811_r8
> > GC - 20060906_AA)
> > JCL - 20061002
> >
> > bea
> > ---
> > java version "1.5.0_06"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > BEA JRockit(R) (build
> > R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >
> >
> >
> --------------------------------------------------------------------------
> >
> > Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >
> > command: java jnt.scimark2.commandline
> >
> > NOTE: bigger number is better
> >
> > Sun6
> > Composite Score: 364.5832265230057
> > FFT (1024): 220.8458713892794
> > SOR (100x100): 696.1542342357722
> > Monte Carlo : 149.37978088875656
> > Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > LU (100x100): 430.1617273683819
> >
> > BEA
> > Composite Score: 359.13480378697835
> > FFT (1024): 303.8746880751562
> > SOR (100x100): 454.25628897202307
> > Monte Carlo : 93.23913192138497
> > Sparse matmult (N=1000, nz=5000): 530.44112637391
> > LU (100x100): 413.8627835924175
> >
> > Sun5
> > Composite Score: 332.84987587548574
> > FFT (1024): 216.5144595799027
> > SOR (100x100): 689.429322146947
> > Monte Carlo : 25.791262124978065
> > Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > LU (100x100): 414.99493895566377
> >
> > IBM
> > Composite Score: 259.8249218693683
> > FFT (1024): 296.8415012789055
> > SOR (100x100): 428.974881649179
> > Monte Carlo : 89.15159857584082
> > Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > LU (100x100): 339.8042037225181
> >
> > Harmony
> > Composite Score: 113.65082278962575
> > FFT (1024): 203.76641991778123
> > SOR (100x100): 224.37761309236748
> > Monte Carlo : 9.063866256533116
> > Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > LU (100x100): 65.6410280487242
> >
> > In this test harmony is clearly lagging behind... at about 30%
> > performance of the best JVM, it's a little crappy. Please note how FFT's
> > performance is not so bad awhile monte carlo is pretty bad compared to
> > BEA or IBM.
> >
> > Overall, it seems like there is some serious work to do here to catch
> up.
> >
> >
> --------------------------------------------------------------------------
> >
> > Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
> >
> > command: java dhry 100000000
> >
> > NOTE: bigger is better
> >
> > NB: I modified the code to accept the count at input from the command
> line!
> >
> > sun6: 8552856 dhrystones/sec
> > sun5: 6605892
> > bea: 5678914
> > harmony: 669734
> > ibm: 501562
> >
> > The performance here is horrific but what's surprising is that J9 is
> > even worse. No idea what's going on but it seems like something is not
> > working as it should (in both harmony and J9)
> >
> >
> --------------------------------------------------------------------------
> >
> > Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >
> > command: java Sieve 30
> >
> > NB: I modified the test to run for a configurable amount of seconds.
> >
> > sun6 8545 sieves/sec
> > sun5 8364
> > bea 6174
> > harmony 1836
> > ibm 225
> >
> > IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> > lagging behind.
> >
> > Stay tuned for more tests.
> >
> > --
> > Stefano.
> >
> >
>
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
> can we run harmony headless yet?)
no :(
2006/11/22, Stefano Mazzocchi <st...@apache.org>:
> Sergey Kuksenko wrote:
>
> > Lets do the simplest thing fist. :)
> > We can do it. We only need to specify a set of workloads.
>
> I've tried running dacapo with 10 warming stages and we are constantly
> around 25% speed against the leading JVM (which is always sun5 or sun6),
> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> results shortly.
>
> Robin, any chance you guys can add 'skymark2' to the dacapo benchmarks?
> I think that would be helpful.
>
> Also, a great thing would be for dacapo to output an XML serialization
> of the benchmark results.
>
> It also appears that we can't run some of the dacapo tests, most notably
> antlr (jvm fails with no error and just terminates right during the
> first warmup) and chart (because it requires a DISPLAY to be
> available...
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Robin Garner wrote:
>> Sergey Kuksenko wrote:
>>
>>> Lets do the simplest thing fist. :)
>>> We can do it. We only need to specify a set of workloads.
>> I've tried running dacapo with 10 warming stages and we are constantly
>> around 25% speed against the leading JVM (which is always sun5 or sun6),
>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>> results shortly.
>
> In my graphs, I find that for drlvm iterations 2 and 3 have very similar
> performance - my assumption would be that by the third iteration, most of
> the benefit of recompilation has gone.
>
>> Robin, any chance you guys can add 'skymark2' to the dacapo benchmarks?
>> I think that would be helpful.
>
> Do you have a link for it ? I've googled without success. We are looking
> for benchmarks for the next major release, so I'd like to see if it meets
> our criteria.
sorry, that would be "scimark" :-) my bad.
>> Also, a great thing would be for dacapo to output an XML serialization
>> of the benchmark results.
>
> The MMTk measurement harness does that, and it sounds like a good idea.
> Any thoughts on what the output would look like ? It's easy to write a
> new callback that will do this.
I would do something minimal yet complete like:
<results version="" date="" os="" arch="" jvm="">
<test name="antlr">
<iteration type="warmup" time="3243"/>
<iteration type="warmup" time="2433"/>
<iteration type="warmup" time="2322"/>
<iteration type="measured" time="2322"/>
</test>
<test name="...">
...
</test>
...
</results>
This would make it a lot easier to automate some sort of graphical results.
>> It also appears that we can't run some of the dacapo tests, most notably
>> antlr (jvm fails with no error and just terminates right during the
>> first warmup) and chart (because it requires a DISPLAY to be
>> available... can we run harmony headless yet?)
>
> I've created a JIRA about the antlr issue. I manage to run it in the
> regression tests by pulling out the DaCapo antlr classes and pre-pending
> them to the boot classpath.
hmmm, I wonder if this is due to version conflicts betwen various
versions of antlr because harmony does ship with it in the bootclasspath.
Which makes me wonder: have you guys thought about running dacapo in a
'reversed classloader' such as the one used by tomcat?
A 'reversed classloader' is one that loads classes first in its own
classpath and then delegates back to the system classloader (the JLS
suggests to do otherwise to avoid security problems, but if you know
what code you're running so this is not a big deal).
Here is such a thing that I've written a while ago for cocoon and kept
using ever since:
http://simile.mit.edu/repository/tools/loader/trunk/src/Loader.java
> Chart is easy to run in batch if you create a virtual framebuffer with
> Xvfb, but I've created a JIRA about this issue too. Gnu Classpath has the
> same issue.
I see. Modern JVMs have the ability to run 'headless', are we planning
of having harmony do the same? it would *seriously* suck if one had to
have X11 installed on their servers just to generate static jpeg/png out
of charts (things that java publishing frameworks do all the time).
> From memory, installing xvfb requires the debian/ubuntu packages xvfb,
> x11-base and xfonts-base.
ok, I'll do that for now.
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Robin Garner <ro...@anu.edu.au>.
> Sergey Kuksenko wrote:
>
>> Lets do the simplest thing fist. :)
>> We can do it. We only need to specify a set of workloads.
>
> I've tried running dacapo with 10 warming stages and we are constantly
> around 25% speed against the leading JVM (which is always sun5 or sun6),
> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> results shortly.
In my graphs, I find that for drlvm iterations 2 and 3 have very similar
performance - my assumption would be that by the third iteration, most of
the benefit of recompilation has gone.
> Robin, any chance you guys can add 'skymark2' to the dacapo benchmarks?
> I think that would be helpful.
Do you have a link for it ? I've googled without success. We are looking
for benchmarks for the next major release, so I'd like to see if it meets
our criteria.
> Also, a great thing would be for dacapo to output an XML serialization
> of the benchmark results.
The MMTk measurement harness does that, and it sounds like a good idea.
Any thoughts on what the output would look like ? It's easy to write a
new callback that will do this.
> It also appears that we can't run some of the dacapo tests, most notably
> antlr (jvm fails with no error and just terminates right during the
> first warmup) and chart (because it requires a DISPLAY to be
> available... can we run harmony headless yet?)
I've created a JIRA about the antlr issue. I manage to run it in the
regression tests by pulling out the DaCapo antlr classes and pre-pending
them to the boot classpath.
Chart is easy to run in batch if you create a virtual framebuffer with
Xvfb, but I've created a JIRA about this issue too. Gnu Classpath has the
same issue.
>From memory, installing xvfb requires the debian/ubuntu packages xvfb,
x11-base and xfonts-base.
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Stefano Mazzocchi wrote:
> Egor Pasko wrote:
> > On the 0x22A day of Apache Harmony Robin Garner wrote:
> >> I'm normalising to the best commercial JVM performance (across Sun
> >> 1.5, JRockit and IBM J9) on the same box. It's a Pentium D 2.8GHz, in
> >> 32-bit mode.
> >
> > Penium D is Prescott, it is reasonable to try on Conroe, Jitrino is
> > likely to behave better on it. And more, Harmony is not enough tuned
> > for performance on Linux. Those are probably the two major reasons to
> > get such different numbers.
>
> the goal is not to understand what is the best platform so that we can
> show off our results on a brochure, the goal is to kick ass everywhere :-)
We need to track this "everywhere" then, but not a single
platform. Everywhere is an interesting goal, BTW :)
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Egor Pasko wrote:
> On the 0x22A day of Apache Harmony Robin Garner wrote:
>> I'm normalising to the best commercial JVM performance (across Sun
>> 1.5, JRockit and IBM J9) on the same box. It's a Pentium D 2.8GHz, in
>> 32-bit mode.
>
> Penium D is Prescott, it is reasonable to try on Conroe, Jitrino is
> likely to behave better on it. And more, Harmony is not enough tuned
> for performance on Linux. Those are probably the two major reasons to
> get such different numbers.
the goal is not to understand what is the best platform so that we can
show off our results on a brochure, the goal is to kick ass everywhere :-)
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Robin Garner wrote:
> I'm normalising to the best commercial JVM performance (across Sun
> 1.5, JRockit and IBM J9) on the same box. It's a Pentium D 2.8GHz, in
> 32-bit mode.
Penium D is Prescott, it is reasonable to try on Conroe, Jitrino is
likely to behave better on it. And more, Harmony is not enough tuned
for performance on Linux. Those are probably the two major reasons to
get such different numbers.
> Also this is OOTB, in a modest sized heap.
>
> cheers
>
> Geir Magnusson Jr. wrote:
> > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > performance of about 35% of whatever he's measuring against.
> > geir
> > Geir Magnusson Jr. wrote:
> >>
> >>
> >> Stefano Mazzocchi wrote:
> >>> Geir Magnusson Jr. wrote:
> >>>>
> >>>> Stefano Mazzocchi wrote:
> >>>>> Sergey Kuksenko wrote:
> >>>>>
> >>>>>> Lets do the simplest thing fist. :)
> >>>>>> We can do it. We only need to specify a set of workloads.
> >>>>> I've tried running dacapo with 10 warming stages and we are constantly
> >>>>> around 25% speed against the leading JVM (which is always sun5
> >>>>> or sun6),
> >>>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> >>>>> results shortly.
> >>>> I don't understand this at all. It wasn't but a few weeks ago when
> >>>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> >>>> 110% of Sun5.
> >>>
> >>> on x86_64?
> >>
> >> That's true. It was x86.
> >>
> >> But the numbers that Robin is reporting aren't great either, are they?
> >>
> >> geir
> >>
> >>>
>
>
> --
> Robin Garner
> Dept. of Computer Science
> Australian National University
> http://cs.anu.edu.au/people/Robin.Garner/
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Robin Garner <ro...@anu.edu.au>.
I'm normalising to the best commercial JVM performance (across Sun 1.5,
JRockit and IBM J9) on the same box. It's a Pentium D 2.8GHz, in 32-bit
mode.
Also this is OOTB, in a modest sized heap.
cheers
Geir Magnusson Jr. wrote:
> I reviewed - looks like Robin is seeing DRLVM get an aggregate
> performance of about 35% of whatever he's measuring against.
>
> geir
>
>
> Geir Magnusson Jr. wrote:
>>
>>
>> Stefano Mazzocchi wrote:
>>> Geir Magnusson Jr. wrote:
>>>>
>>>> Stefano Mazzocchi wrote:
>>>>> Sergey Kuksenko wrote:
>>>>>
>>>>>> Lets do the simplest thing fist. :)
>>>>>> We can do it. We only need to specify a set of workloads.
>>>>> I've tried running dacapo with 10 warming stages and we are constantly
>>>>> around 25% speed against the leading JVM (which is always sun5 or
>>>>> sun6),
>>>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>>>>> results shortly.
>>>> I don't understand this at all. It wasn't but a few weeks ago when
>>>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
>>>> 110% of Sun5.
>>>
>>> on x86_64?
>>
>> That's true. It was x86.
>>
>> But the numbers that Robin is reporting aren't great either, are they?
>>
>> geir
>>
>>>
--
Robin Garner
Dept. of Computer Science
Australian National University
http://cs.anu.edu.au/people/Robin.Garner/
Re: [performance] a few early benchmarks
Posted by Pavel Ozhdikhin <pa...@gmail.com>.
On 11/25/06, Stefano Mazzocchi <st...@apache.org> wrote:
> Pavel Ozhdikhin wrote:
>
> > In server_static mode optimizations are based solely on heuristics while
> in
> > server mode the dynamic optimization framework is employed.
> > In server_static mode methods are compiled only once by the optimizing
> > Jitrino.OPT compiler.
> > In server mode there is recompilation scheme: first time a method
> compiled
> > using fast optimizations and instrumenting the code to gather edge
> profile
> > information at runtime. When the profile is ready the method is
> > recompiled with agressive optimizations using the edge profile
> information.
>
> BTW, is it possible to change this behavior from the command line or I
> have to recompile the VM?
Yep, it's possible to set the execution mode from the command line with the
'-Xem' switch.
> in any case, are there instructions on how to trigger different
> execution behavior on the VM some where in our docos/wikis?
There is some info here:
http://wiki.apache.org/harmony/DrlvmCommandLineOptions
And more information about configuring the Execution Manager (em) and
pre-defined configurations:
http://harmony.apache.org/subcomponents/drlvm/emguide.html
Thanks,
Pavel
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Pavel Ozhdikhin wrote:
> In server_static mode optimizations are based solely on heuristics while in
> server mode the dynamic optimization framework is employed.
> In server_static mode methods are compiled only once by the optimizing
> Jitrino.OPT compiler.
> In server mode there is recompilation scheme: first time a method compiled
> using fast optimizations and instrumenting the code to gather edge profile
> information at runtime. When the profile is ready the method is
> recompiled with agressive optimizations using the edge profile information.
BTW, is it possible to change this behavior from the command line or I
have to recompile the VM?
in any case, are there instructions on how to trigger different
execution behavior on the VM some where in our docos/wikis?
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Pavel Ozhdikhin <pa...@gmail.com>.
On 11/24/06, Alexey Varlamov <al...@gmail.com> wrote:
>
> 2006/11/24, Alexey Petrenko <al...@gmail.com>:
> > 2006/11/24, Alexey Varlamov <al...@gmail.com>:
> > > 2006/11/24, Alexey Petrenko <al...@gmail.com>:
> > > > 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> > > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > > > > > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > > > > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <
> egor.pasko@gmail.com> wrote:
> > > > > > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun
> wrote:
> > > > > > > > > > > > The numbers that I published was received on P4
> under Windows +
> > > > > > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't
> run Dacapo under
> > > > > > > > > > > > x86_64, but I hope we could receive almost the same
> range (10-20 %
> > > > > > > > > > > > slower that Sun) with the mentioned configuration.
> > > > > > > > > > >
> > > > > > > > > > > And Sun was running with "-server" too I guess? :)
> > > > > > > > > >
> > > > > > > > > > Yes, of course.
> > > > > > > > > >
> > > > > > > > > > > Maybe, it is time to track performance comparisons of
> *different
> > > > > > > > > > > platforms* in one place? That should help to avoid
> major differences in
> > > > > > > > > > > our visions for harmony performance.
> > > > > > > > > >
> > > > > > > > > > Yes, good idea. Should we define the configuration for
> all VM's as well?
> > > > > > > > > > For instance, for Sun we could use parameters from spec
> site. What do
> > > > > > > > > > you think about it?
> > > > > > > > > +1 for options from spec site.
> > > > > > > >
> > > > > > > > I am in love with it too. We could present results similar
> to how
> > > > > > > > spec.org does. Just a list of runs.
> > > > > > > > For each:
> > > > > > > > * revision number
> > > > > > > > * hardware/os summary (number of cores)
> > > > > > > > * link to full details
> > > > > > > +1
> > > > > > >
> > > > > > > > * more?
> > > > > > >
> > > > > > > score? :)
> > > > >
> > > > > oops :)
> > > > >
> > > > > > Scores are most visible things but we can not easily publish
> them for
> > > > > > all specs...
> > > > > not allowed -> won't publish. We have great free benchmarks
> (thanks to
> > > > > DaCapo guys!), they will give us a good picture.
> > > > But I think that we should keep these (spec) benchmarks in mind and
> > > > optimize Harmony for them.
> > > For the *default* mode?
> > No. For server mode.
> Agree then, of course. I wonder most benchmarks will hardly conflict
> (significantly at least) on JIT optimization paths, so the more of
> them the better... Though GC requirements most probably is different
> story.
> BTW, pardon my silly question, what is the core difference between
> server and server_static configs?
In server_static mode optimizations are based solely on heuristics while in
server mode the dynamic optimization framework is employed.
In server_static mode methods are compiled only once by the optimizing
Jitrino.OPT compiler.
In server mode there is recompilation scheme: first time a method compiled
using fast optimizations and instrumenting the code to gather edge profile
information at runtime. When the profile is ready the method is
recompiled with agressive optimizations using the edge profile information.
Thanks,
Pavel
>
> > SY, Alexey
> >
>
Re: [performance] a few early benchmarks
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/24, Alexey Petrenko <al...@gmail.com>:
> 2006/11/24, Alexey Varlamov <al...@gmail.com>:
> > 2006/11/24, Alexey Petrenko <al...@gmail.com>:
> > > 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > > > > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > > > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > > > > > > slower that Sun) with the mentioned configuration.
> > > > > > > > > >
> > > > > > > > > > And Sun was running with "-server" too I guess? :)
> > > > > > > > >
> > > > > > > > > Yes, of course.
> > > > > > > > >
> > > > > > > > > > Maybe, it is time to track performance comparisons of *different
> > > > > > > > > > platforms* in one place? That should help to avoid major differences in
> > > > > > > > > > our visions for harmony performance.
> > > > > > > > >
> > > > > > > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > > > > > > For instance, for Sun we could use parameters from spec site. What do
> > > > > > > > > you think about it?
> > > > > > > > +1 for options from spec site.
> > > > > > >
> > > > > > > I am in love with it too. We could present results similar to how
> > > > > > > spec.org does. Just a list of runs.
> > > > > > > For each:
> > > > > > > * revision number
> > > > > > > * hardware/os summary (number of cores)
> > > > > > > * link to full details
> > > > > > +1
> > > > > >
> > > > > > > * more?
> > > > > >
> > > > > > score? :)
> > > >
> > > > oops :)
> > > >
> > > > > Scores are most visible things but we can not easily publish them for
> > > > > all specs...
> > > > not allowed -> won't publish. We have great free benchmarks (thanks to
> > > > DaCapo guys!), they will give us a good picture.
> > > But I think that we should keep these (spec) benchmarks in mind and
> > > optimize Harmony for them.
> > For the *default* mode?
> No. For server mode.
Agree then, of course. I wonder most benchmarks will hardly conflict
(significantly at least) on JIT optimization paths, so the more of
them the better... Though GC requirements most probably is different
story.
BTW, pardon my silly question, what is the core difference between
server and server_static configs?
>
> SY, Alexey
>
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
2006/11/24, Alexey Varlamov <al...@gmail.com>:
> 2006/11/24, Alexey Petrenko <al...@gmail.com>:
> > 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > > > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > > > > > slower that Sun) with the mentioned configuration.
> > > > > > > > >
> > > > > > > > > And Sun was running with "-server" too I guess? :)
> > > > > > > >
> > > > > > > > Yes, of course.
> > > > > > > >
> > > > > > > > > Maybe, it is time to track performance comparisons of *different
> > > > > > > > > platforms* in one place? That should help to avoid major differences in
> > > > > > > > > our visions for harmony performance.
> > > > > > > >
> > > > > > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > > > > > For instance, for Sun we could use parameters from spec site. What do
> > > > > > > > you think about it?
> > > > > > > +1 for options from spec site.
> > > > > >
> > > > > > I am in love with it too. We could present results similar to how
> > > > > > spec.org does. Just a list of runs.
> > > > > > For each:
> > > > > > * revision number
> > > > > > * hardware/os summary (number of cores)
> > > > > > * link to full details
> > > > > +1
> > > > >
> > > > > > * more?
> > > > >
> > > > > score? :)
> > >
> > > oops :)
> > >
> > > > Scores are most visible things but we can not easily publish them for
> > > > all specs...
> > > not allowed -> won't publish. We have great free benchmarks (thanks to
> > > DaCapo guys!), they will give us a good picture.
> > But I think that we should keep these (spec) benchmarks in mind and
> > optimize Harmony for them.
> For the *default* mode?
No. For server mode.
SY, Alexey
Re: [performance] a few early benchmarks
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/24, Alexey Petrenko <al...@gmail.com>:
> 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > > > > slower that Sun) with the mentioned configuration.
> > > > > > > >
> > > > > > > > And Sun was running with "-server" too I guess? :)
> > > > > > >
> > > > > > > Yes, of course.
> > > > > > >
> > > > > > > > Maybe, it is time to track performance comparisons of *different
> > > > > > > > platforms* in one place? That should help to avoid major differences in
> > > > > > > > our visions for harmony performance.
> > > > > > >
> > > > > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > > > > For instance, for Sun we could use parameters from spec site. What do
> > > > > > > you think about it?
> > > > > > +1 for options from spec site.
> > > > >
> > > > > I am in love with it too. We could present results similar to how
> > > > > spec.org does. Just a list of runs.
> > > > > For each:
> > > > > * revision number
> > > > > * hardware/os summary (number of cores)
> > > > > * link to full details
> > > > +1
> > > >
> > > > > * more?
> > > >
> > > > score? :)
> >
> > oops :)
> >
> > > Scores are most visible things but we can not easily publish them for
> > > all specs...
> > not allowed -> won't publish. We have great free benchmarks (thanks to
> > DaCapo guys!), they will give us a good picture.
> But I think that we should keep these (spec) benchmarks in mind and
> optimize Harmony for them.
For the *default* mode?
>
> SY, Alexey
>
> > > For example: http://www.spec.org/jbb2005/docs/FAQ.html#Qpublish
> > > == cut ===
> > > Q15
> > > How can I publish SPECjbb2005 results?
> > >
> > > A15
> > > You need to get a SPECjbb2005 license in order to publish results.
> > > For more information about submitting results, please contact SPEC.
> > > == cut ===
> > > Another link on SPECjbb2005:
> > > http://www.spec.org/jbb2005/docs/RunRules.html#_Toc102806116
> > > This question need more investigation.
> > >
> > > Anyway I think that we should define suite list first.
> > >
> > > SY, Alexey
> >
> > [snip]
> >
> > --
> > Egor Pasko
> >
> >
>
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
2006/11/27, Geir Magnusson Jr. <ge...@pobox.com>:
> Mikhail Loenko wrote:
> >
> > Not sure that when we elect an official benchmark we should take
> > into account whether it's open or not. IMHO we should base our
> > decision on how close it to the real life and how many people are
> > interested in it
>
> I agree for the latter - people are (for reasons I never grok)
> interested in SPEC benchmark results, so I think we definitely need to
> keep the eye on the ball for that one.
>
> You will have a lot of trouble convincing me that SPEC benchmarks are
> close to "real life", because if we're seeing SPECjbb results of around
> 100% of Sun SE 5, and then Dacapo showing us 40% of 'best', there seems
> to be a problem.
In fact Dacapo and SPECjbb exercises different parts of API/VM stack.
So the scores are mostly irrelevant to each other. For instance if we
choose some awt benchmark it may show even worse result comparing to
Sun. This does not necessarily mean that this benchmark is better then
Dacapo or SPECjbb. :)
SY, Alexey
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Mikhail Loenko wrote:
>
> Not sure that when we elect an official benchmark we should take
> into account whether it's open or not. IMHO we should base our
> decision on how close it to the real life and how many people are
> interested in it
I agree for the latter - people are (for reasons I never grok)
interested in SPEC benchmark results, so I think we definitely need to
keep the eye on the ball for that one.
You will have a lot of trouble convincing me that SPEC benchmarks are
close to "real life", because if we're seeing SPECjbb results of around
100% of Sun SE 5, and then Dacapo showing us 40% of 'best', there seems
to be a problem.
geir
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
2006/11/27, Stefano Mazzocchi <st...@apache.org>:
> And, finally, I don't care if Spec it's the industry standard:
I believe that good results in SPEC, which is industry standard as you
said, will give us more users from the industry where SPEC is a
standard. And this is good for Harmony: more users, more testers, more
contributors.
And I agree with Mikhail nobody asks EVERY Harmony contributor to
test ALL the fixes on ALL platforms and run EVERY benchmark before
each patch. Other community members will help!
SY, Alexey
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
I actually agree with both of you.
We definitely need to include other benchmarks - especially open ones -
besides the SPEC benchmarks, because at this point, we as a project
don't have the SPEC ones. I'm very +1 on Dacapo, and actually any
others that we find. Dacapo is nice because there are serious pros
actively working on and around it.
On the other hand, it's a sad reality that we can't ignore SPEC either,
because that is how some people measure performance.
geir
Mikhail Loenko wrote:
> no. not at all
>
> 2006/11/27, Stefano Mazzocchi <st...@apache.org>:
>> Mikhail Loenko wrote:
>>
>> > Not sure that when we elect an official benchmark we should take
>> > into account whether it's open or not.
>>
>> You're kidding, right?
>>
>> --
>> Stefano.
>>
>>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Mikhail Loenko wrote:
> 2006/11/27, Stefano Mazzocchi <st...@apache.org>:
>> Mikhail Loenko wrote:
>>
>> > Not sure that when we elect an official benchmark we should take
>> > into account whether it's open or not.
>>
>> You're kidding, right?
>
> no. not at all
Then you have to realize that not everybody has the $$$ or the intention
to spend money to help us optimize Harmony. No, worse: such carelessness
for open participation is poisonous to the creation a more diverse and
distributed development community.
This is not an closed-development project anymore, the rules have
changed: we need to think in terms of lowering participation obstacles,
not in terms of pleasing our upper management or our potential customers.
Our main and only objective is to please our users, and reducing the
barrier for them to become developers as much as we possibly can. And
that sometimes includes making compromises with our own personal (or
job-related) interests.
If you (or anybody else around you) care about those, you are more than
welcome to continue on that path, submit code changes, results and
ideas... but pretending that an entire community bases its performance
results on something potential developers would have to pay to get
access to is equivalent of destroying the ability for
non-corporate-sponsored individuals to help in that space.
And, I'm sorry, but such discrimination won't make me (and a lot of
people like me) happy.
Let me be crystal clear: you are and will continue to be *more than
welcome* to benchmark with anything you (or anybody around you) cares
about and work with those results. But benchmarks have two objectives:
show the current performance status *and* allow potential developers to
test if their changes have made a positive impact before submitting them.
Now that we are an open development project, we must learn,
collectively, to think in terms of 'what can I do to acquire more
development participation', not only in terms of 'what can I do to
acquire more users'.
DaCapo fits both goals very well. Spec fits the first, partially.
And, finally, I don't care if Spec it's the industry standard: if we
were driven by following what everybody else was doing, there would be
no Harmony in the first place.
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Mikhail Loenko <ml...@gmail.com>.
no. not at all
2006/11/27, Stefano Mazzocchi <st...@apache.org>:
> Mikhail Loenko wrote:
>
> > Not sure that when we elect an official benchmark we should take
> > into account whether it's open or not.
>
> You're kidding, right?
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Mikhail Loenko wrote:
> Not sure that when we elect an official benchmark we should take
> into account whether it's open or not.
You're kidding, right?
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Mikhail Loenko <ml...@gmail.com>.
2006/11/25, Stefano Mazzocchi <st...@apache.org>:
> Alexey Petrenko wrote:
> > 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> >> On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> >> > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> >> > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> >> > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> >> > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> >> > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko
> >> <eg...@gmail.com> wrote:
> >> > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> >> > > > > > > > The numbers that I published was received on P4 under
> >> Windows +
> >> > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run
> >> Dacapo under
> >> > > > > > > > x86_64, but I hope we could receive almost the same
> >> range (10-20 %
> >> > > > > > > > slower that Sun) with the mentioned configuration.
> >> > > > > > >
> >> > > > > > > And Sun was running with "-server" too I guess? :)
> >> > > > > >
> >> > > > > > Yes, of course.
> >> > > > > >
> >> > > > > > > Maybe, it is time to track performance comparisons of
> >> *different
> >> > > > > > > platforms* in one place? That should help to avoid major
> >> differences in
> >> > > > > > > our visions for harmony performance.
> >> > > > > >
> >> > > > > > Yes, good idea. Should we define the configuration for all
> >> VM's as well?
> >> > > > > > For instance, for Sun we could use parameters from spec
> >> site. What do
> >> > > > > > you think about it?
> >> > > > > +1 for options from spec site.
> >> > > >
> >> > > > I am in love with it too. We could present results similar to how
> >> > > > spec.org does. Just a list of runs.
> >> > > > For each:
> >> > > > * revision number
> >> > > > * hardware/os summary (number of cores)
> >> > > > * link to full details
> >> > > +1
> >> > >
> >> > > > * more?
> >> > >
> >> > > score? :)
> >>
> >> oops :)
> >>
> >> > Scores are most visible things but we can not easily publish them for
> >> > all specs...
> >> not allowed -> won't publish. We have great free benchmarks (thanks to
> >> DaCapo guys!), they will give us a good picture.
> > But I think that we should keep these (spec) benchmarks in mind and
> > optimize Harmony for them.
>
> Each and everyone of us is free to run harmony against anything they
> want and send info/patches that optimize for that.... but as a community
> we need to focus on what creates the best practical result.
>
> My proposal is to focus on DaCapo which is, more modern, more open, more
> complete, more thought out and more in line with real-life uses of the
> java technologies that apache users care for, today.
>
> And where DaCapo is lacking (for example, client/server heavily
> concurrent serving applications), we should work with them to add new tests.
>
> Spec on the other end is:
>
> 1) almost a decade old!
> 2) closed both sources, binaries and results
> 3) no community to talk to
> 4) no sign of activity
>
> There is the very high chance that the results of Spec will drive us to
> spend energy in fixing things that nobody cares about.
>
> Personally, I don't care about "brochure-like PR figures". Remember, our
> goal is not to sell harmony and convince people to adopt with figures...
> people will try it, they will run tests and if they find it good enough,
> they'll use it, if not they will use something else, or help us fix what
> they care for.
>
> So, if you care for SPEC, sure, bring it on, but it will be treated as
> just another application and the community will decide if those changes
> need to be fixed or not, depending on the 'short blanket' principle of
> parameter tuning.
>
> The official benchmark for Harmony must be something open, anything else
> is just another application.
>
> Because, believe me, it won't matter to anyone out there if Sun has 2x
> performance against spec when harmony sends 2x more servlet requests ;-)
Not sure that when we elect an official benchmark we should take
into account whether it's open or not. IMHO we should base our
decision on how close it to the real life and how many people are
interested in it
Thanks,
Mikhail
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Alexey Petrenko wrote:
> 23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
>> On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
>> > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
>> > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
>> > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
>> > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
>> > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko
>> <eg...@gmail.com> wrote:
>> > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
>> > > > > > > > The numbers that I published was received on P4 under
>> Windows +
>> > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run
>> Dacapo under
>> > > > > > > > x86_64, but I hope we could receive almost the same
>> range (10-20 %
>> > > > > > > > slower that Sun) with the mentioned configuration.
>> > > > > > >
>> > > > > > > And Sun was running with "-server" too I guess? :)
>> > > > > >
>> > > > > > Yes, of course.
>> > > > > >
>> > > > > > > Maybe, it is time to track performance comparisons of
>> *different
>> > > > > > > platforms* in one place? That should help to avoid major
>> differences in
>> > > > > > > our visions for harmony performance.
>> > > > > >
>> > > > > > Yes, good idea. Should we define the configuration for all
>> VM's as well?
>> > > > > > For instance, for Sun we could use parameters from spec
>> site. What do
>> > > > > > you think about it?
>> > > > > +1 for options from spec site.
>> > > >
>> > > > I am in love with it too. We could present results similar to how
>> > > > spec.org does. Just a list of runs.
>> > > > For each:
>> > > > * revision number
>> > > > * hardware/os summary (number of cores)
>> > > > * link to full details
>> > > +1
>> > >
>> > > > * more?
>> > >
>> > > score? :)
>>
>> oops :)
>>
>> > Scores are most visible things but we can not easily publish them for
>> > all specs...
>> not allowed -> won't publish. We have great free benchmarks (thanks to
>> DaCapo guys!), they will give us a good picture.
> But I think that we should keep these (spec) benchmarks in mind and
> optimize Harmony for them.
Each and everyone of us is free to run harmony against anything they
want and send info/patches that optimize for that.... but as a community
we need to focus on what creates the best practical result.
My proposal is to focus on DaCapo which is, more modern, more open, more
complete, more thought out and more in line with real-life uses of the
java technologies that apache users care for, today.
And where DaCapo is lacking (for example, client/server heavily
concurrent serving applications), we should work with them to add new tests.
Spec on the other end is:
1) almost a decade old!
2) closed both sources, binaries and results
3) no community to talk to
4) no sign of activity
There is the very high chance that the results of Spec will drive us to
spend energy in fixing things that nobody cares about.
Personally, I don't care about "brochure-like PR figures". Remember, our
goal is not to sell harmony and convince people to adopt with figures...
people will try it, they will run tests and if they find it good enough,
they'll use it, if not they will use something else, or help us fix what
they care for.
So, if you care for SPEC, sure, bring it on, but it will be treated as
just another application and the community will decide if those changes
need to be fixed or not, depending on the 'short blanket' principle of
parameter tuning.
The official benchmark for Harmony must be something open, anything else
is just another application.
Because, believe me, it won't matter to anyone out there if Sun has 2x
performance against spec when harmony sends 2x more servlet requests ;-)
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
23 Nov 2006 19:44:23 +0600, Egor Pasko <eg...@gmail.com>:
> On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > > > slower that Sun) with the mentioned configuration.
> > > > > > >
> > > > > > > And Sun was running with "-server" too I guess? :)
> > > > > >
> > > > > > Yes, of course.
> > > > > >
> > > > > > > Maybe, it is time to track performance comparisons of *different
> > > > > > > platforms* in one place? That should help to avoid major differences in
> > > > > > > our visions for harmony performance.
> > > > > >
> > > > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > > > For instance, for Sun we could use parameters from spec site. What do
> > > > > > you think about it?
> > > > > +1 for options from spec site.
> > > >
> > > > I am in love with it too. We could present results similar to how
> > > > spec.org does. Just a list of runs.
> > > > For each:
> > > > * revision number
> > > > * hardware/os summary (number of cores)
> > > > * link to full details
> > > +1
> > >
> > > > * more?
> > >
> > > score? :)
>
> oops :)
>
> > Scores are most visible things but we can not easily publish them for
> > all specs...
> not allowed -> won't publish. We have great free benchmarks (thanks to
> DaCapo guys!), they will give us a good picture.
But I think that we should keep these (spec) benchmarks in mind and
optimize Harmony for them.
SY, Alexey
> > For example: http://www.spec.org/jbb2005/docs/FAQ.html#Qpublish
> > == cut ===
> > Q15
> > How can I publish SPECjbb2005 results?
> >
> > A15
> > You need to get a SPECjbb2005 license in order to publish results.
> > For more information about submitting results, please contact SPEC.
> > == cut ===
> > Another link on SPECjbb2005:
> > http://www.spec.org/jbb2005/docs/RunRules.html#_Toc102806116
> > This question need more investigation.
> >
> > Anyway I think that we should define suite list first.
> >
> > SY, Alexey
>
> [snip]
>
> --
> Egor Pasko
>
>
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > The numbers that I published was received on P4 under Windows +
> > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > slower that Sun) with the mentioned configuration.
> > > >
> > > > And Sun was running with "-server" too I guess? :)
> > >
> > > Yes, of course.
> > >
> > > > Maybe, it is time to track performance comparisons of *different
> > > > platforms* in one place? That should help to avoid major differences in
> > > > our visions for harmony performance.
> > >
> > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > For instance, for Sun we could use parameters from spec site. What do
> > > you think about it?
> > +1 for options from spec site.
>
> I am in love with it too. We could present results similar to how
> spec.org does. Just a list of runs.
> For each:
> * revision number
> * hardware/os summary (number of cores)
> * link to full details
> * more?
>
> that would be great if we had volunteers on this, to avoid performance
> confusions :)
Let's define a performance suite list first.
I can be a volunteer for Dacapo.
SY, Alexey
> > > > * Melody
> > > > * marmonytest.org
> > > > * Robin's site
> > > > * wiki (just for the start, maybe)
> > > >
> > > > > Thanks,
> > > > > Vladimir.
> > > > >
> > > > > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > > > > performance of about 35% of whatever he's measuring against.
> > > > > >
> > > > > > geir
> > > > > >
> > > > > >
> > > > > > Geir Magnusson Jr. wrote:
> > > > > > >
> > > > > > >
> > > > > > > Stefano Mazzocchi wrote:
> > > > > > >> Geir Magnusson Jr. wrote:
> > > > > > >>>
> > > > > > >>> Stefano Mazzocchi wrote:
> > > > > > >>>> Sergey Kuksenko wrote:
> > > > > > >>>>
> > > > > > >>>>> Lets do the simplest thing fist. :)
> > > > > > >>>>> We can do it. We only need to specify a set of workloads.
> > > > > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > > > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > > > > >>>> sun6),
> > > > > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > > > > >>>> results shortly.
> > > > > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > > > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > > > > >>> 110% of Sun5.
> > > > > > >>
> > > > > > >> on x86_64?
> > > > > > >
> > > > > > > That's true. It was x86.
> > > > > > >
> > > > > > > But the numbers that Robin is reporting aren't great either, are they?
> > > > > > >
> > > > > > > geir
> > > > > > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > > --
> > > > Egor Pasko
> > > >
> > > >
> > >
> >
>
> --
> Egor Pasko
>
>
Re: [performance] a few early benchmarks
Posted by Robin Garner <ro...@anu.edu.au>.
Egor Pasko wrote:
> On the 0x22B day of Apache Harmony Stefano Mazzocchi wrote:
>> Egor Pasko wrote:
>>
>>>> Scores are most visible things but we can not easily publish them for
>>>> all specs...
>>> not allowed -> won't publish. We have great free benchmarks (thanks to
>>> DaCapo guys!), they will give us a good picture.
>> Agreed, screw SPEC, I vote on focusing on DaCapo which, IMO, provide an
>> even better picture of real life applications and JVM variability.
>
> Agreed.
> Do we have all necessary bugs filed against DaCapo to enable it
> running? I recall 3 blockers related:
> * SHA-1 (could not find in JIRA)
HARMONY-2135
> * X11 (headless DRLVM) issue (could not find in JIRA)
HARMONY-2128
> * jython (HARMONY-2137)
> // add to this list
There's also
HARMONY-2130 DaCapo antlr fails on drlvm
which can be worked around.
>
> traditionally: "Let's fix 'em all!"
>
>> I have also convinced myself that scimark2 is not that great of a
>> benchmark and tuning for it might mislead other things that are way more
>> important to our user base (like running tomcat or eclipse).
>
> Agreed.
>
--
Robin Garner
Dept. of Computer Science
Australian National University
http://cs.anu.edu.au/people/Robin.Garner/
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22B day of Apache Harmony Stefano Mazzocchi wrote:
> Egor Pasko wrote:
>
> >> Scores are most visible things but we can not easily publish them for
> >> all specs...
> >
> > not allowed -> won't publish. We have great free benchmarks (thanks to
> > DaCapo guys!), they will give us a good picture.
>
> Agreed, screw SPEC, I vote on focusing on DaCapo which, IMO, provide an
> even better picture of real life applications and JVM variability.
Agreed.
Do we have all necessary bugs filed against DaCapo to enable it
running? I recall 3 blockers related:
* SHA-1 (could not find in JIRA)
* X11 (headless DRLVM) issue (could not find in JIRA)
* jython (HARMONY-2137)
// add to this list
traditionally: "Let's fix 'em all!"
> I have also convinced myself that scimark2 is not that great of a
> benchmark and tuning for it might mislead other things that are way more
> important to our user base (like running tomcat or eclipse).
Agreed.
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Egor Pasko wrote:
>> Scores are most visible things but we can not easily publish them for
>> all specs...
>
> not allowed -> won't publish. We have great free benchmarks (thanks to
> DaCapo guys!), they will give us a good picture.
Agreed, screw SPEC, I vote on focusing on DaCapo which, IMO, provide an
even better picture of real life applications and JVM variability.
I have also convinced myself that scimark2 is not that great of a
benchmark and tuning for it might mislead other things that are way more
important to our user base (like running tomcat or eclipse).
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> 2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> > 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > > slower that Sun) with the mentioned configuration.
> > > > > >
> > > > > > And Sun was running with "-server" too I guess? :)
> > > > >
> > > > > Yes, of course.
> > > > >
> > > > > > Maybe, it is time to track performance comparisons of *different
> > > > > > platforms* in one place? That should help to avoid major differences in
> > > > > > our visions for harmony performance.
> > > > >
> > > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > > For instance, for Sun we could use parameters from spec site. What do
> > > > > you think about it?
> > > > +1 for options from spec site.
> > >
> > > I am in love with it too. We could present results similar to how
> > > spec.org does. Just a list of runs.
> > > For each:
> > > * revision number
> > > * hardware/os summary (number of cores)
> > > * link to full details
> > +1
> >
> > > * more?
> >
> > score? :)
oops :)
> Scores are most visible things but we can not easily publish them for
> all specs...
not allowed -> won't publish. We have great free benchmarks (thanks to
DaCapo guys!), they will give us a good picture.
> For example: http://www.spec.org/jbb2005/docs/FAQ.html#Qpublish
> == cut ===
> Q15
> How can I publish SPECjbb2005 results?
>
> A15
> You need to get a SPECjbb2005 license in order to publish results.
> For more information about submitting results, please contact SPEC.
> == cut ===
> Another link on SPECjbb2005:
> http://www.spec.org/jbb2005/docs/RunRules.html#_Toc102806116
> This question need more investigation.
>
> Anyway I think that we should define suite list first.
>
> SY, Alexey
[snip]
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
2006/11/23, Mikhail Loenko <ml...@gmail.com>:
> 23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> > On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > > The numbers that I published was received on P4 under Windows +
> > > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > > slower that Sun) with the mentioned configuration.
> > > > >
> > > > > And Sun was running with "-server" too I guess? :)
> > > >
> > > > Yes, of course.
> > > >
> > > > > Maybe, it is time to track performance comparisons of *different
> > > > > platforms* in one place? That should help to avoid major differences in
> > > > > our visions for harmony performance.
> > > >
> > > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > > For instance, for Sun we could use parameters from spec site. What do
> > > > you think about it?
> > > +1 for options from spec site.
> >
> > I am in love with it too. We could present results similar to how
> > spec.org does. Just a list of runs.
> > For each:
> > * revision number
> > * hardware/os summary (number of cores)
> > * link to full details
> +1
>
> > * more?
>
> score? :)
Scores are most visible things but we can not easily publish them for
all specs...
For example: http://www.spec.org/jbb2005/docs/FAQ.html#Qpublish
== cut ===
Q15
How can I publish SPECjbb2005 results?
A15
You need to get a SPECjbb2005 license in order to publish results.
For more information about submitting results, please contact SPEC.
== cut ===
Another link on SPECjbb2005:
http://www.spec.org/jbb2005/docs/RunRules.html#_Toc102806116
This question need more investigation.
Anyway I think that we should define suite list first.
SY, Alexey
> >
> > that would be great if we had volunteers on this, to avoid performance
> > confusions :)
> >
> > > > > * Melody
> > > > > * marmonytest.org
> > > > > * Robin's site
> > > > > * wiki (just for the start, maybe)
> > > > >
> > > > > > Thanks,
> > > > > > Vladimir.
> > > > > >
> > > > > > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > > > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > > > > > performance of about 35% of whatever he's measuring against.
> > > > > > >
> > > > > > > geir
> > > > > > >
> > > > > > >
> > > > > > > Geir Magnusson Jr. wrote:
> > > > > > > >
> > > > > > > >
> > > > > > > > Stefano Mazzocchi wrote:
> > > > > > > >> Geir Magnusson Jr. wrote:
> > > > > > > >>>
> > > > > > > >>> Stefano Mazzocchi wrote:
> > > > > > > >>>> Sergey Kuksenko wrote:
> > > > > > > >>>>
> > > > > > > >>>>> Lets do the simplest thing fist. :)
> > > > > > > >>>>> We can do it. We only need to specify a set of workloads.
> > > > > > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > > > > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > > > > > >>>> sun6),
> > > > > > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > > > > > >>>> results shortly.
> > > > > > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > > > > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > > > > > >>> 110% of Sun5.
> > > > > > > >>
> > > > > > > >> on x86_64?
> > > > > > > >
> > > > > > > > That's true. It was x86.
> > > > > > > >
> > > > > > > > But the numbers that Robin is reporting aren't great either, are they?
> > > > > > > >
> > > > > > > > geir
> > > > > > > >
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Egor Pasko
> > > > >
> > > > >
> > > >
> > >
> >
> > --
> > Egor Pasko
> >
> >
>
Re: [performance] a few early benchmarks
Posted by Mikhail Loenko <ml...@gmail.com>.
23 Nov 2006 16:34:22 +0600, Egor Pasko <eg...@gmail.com>:
> On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> > 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > > The numbers that I published was received on P4 under Windows +
> > > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > > slower that Sun) with the mentioned configuration.
> > > >
> > > > And Sun was running with "-server" too I guess? :)
> > >
> > > Yes, of course.
> > >
> > > > Maybe, it is time to track performance comparisons of *different
> > > > platforms* in one place? That should help to avoid major differences in
> > > > our visions for harmony performance.
> > >
> > > Yes, good idea. Should we define the configuration for all VM's as well?
> > > For instance, for Sun we could use parameters from spec site. What do
> > > you think about it?
> > +1 for options from spec site.
>
> I am in love with it too. We could present results similar to how
> spec.org does. Just a list of runs.
> For each:
> * revision number
> * hardware/os summary (number of cores)
> * link to full details
+1
> * more?
score? :)
Thanks,
Mikhail
>
> that would be great if we had volunteers on this, to avoid performance
> confusions :)
>
> > > > * Melody
> > > > * marmonytest.org
> > > > * Robin's site
> > > > * wiki (just for the start, maybe)
> > > >
> > > > > Thanks,
> > > > > Vladimir.
> > > > >
> > > > > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > > > > performance of about 35% of whatever he's measuring against.
> > > > > >
> > > > > > geir
> > > > > >
> > > > > >
> > > > > > Geir Magnusson Jr. wrote:
> > > > > > >
> > > > > > >
> > > > > > > Stefano Mazzocchi wrote:
> > > > > > >> Geir Magnusson Jr. wrote:
> > > > > > >>>
> > > > > > >>> Stefano Mazzocchi wrote:
> > > > > > >>>> Sergey Kuksenko wrote:
> > > > > > >>>>
> > > > > > >>>>> Lets do the simplest thing fist. :)
> > > > > > >>>>> We can do it. We only need to specify a set of workloads.
> > > > > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > > > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > > > > >>>> sun6),
> > > > > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > > > > >>>> results shortly.
> > > > > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > > > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > > > > >>> 110% of Sun5.
> > > > > > >>
> > > > > > >> on x86_64?
> > > > > > >
> > > > > > > That's true. It was x86.
> > > > > > >
> > > > > > > But the numbers that Robin is reporting aren't great either, are they?
> > > > > > >
> > > > > > > geir
> > > > > > >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > > > --
> > > > Egor Pasko
> > > >
> > > >
> > >
> >
>
> --
> Egor Pasko
>
>
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Alexey Petrenko wrote:
> 2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> > On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > > The numbers that I published was received on P4 under Windows +
> > > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > > slower that Sun) with the mentioned configuration.
> > >
> > > And Sun was running with "-server" too I guess? :)
> >
> > Yes, of course.
> >
> > > Maybe, it is time to track performance comparisons of *different
> > > platforms* in one place? That should help to avoid major differences in
> > > our visions for harmony performance.
> >
> > Yes, good idea. Should we define the configuration for all VM's as well?
> > For instance, for Sun we could use parameters from spec site. What do
> > you think about it?
> +1 for options from spec site.
I am in love with it too. We could present results similar to how
spec.org does. Just a list of runs.
For each:
* revision number
* hardware/os summary (number of cores)
* link to full details
* more?
that would be great if we had volunteers on this, to avoid performance
confusions :)
> > > * Melody
> > > * marmonytest.org
> > > * Robin's site
> > > * wiki (just for the start, maybe)
> > >
> > > > Thanks,
> > > > Vladimir.
> > > >
> > > > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > > > performance of about 35% of whatever he's measuring against.
> > > > >
> > > > > geir
> > > > >
> > > > >
> > > > > Geir Magnusson Jr. wrote:
> > > > > >
> > > > > >
> > > > > > Stefano Mazzocchi wrote:
> > > > > >> Geir Magnusson Jr. wrote:
> > > > > >>>
> > > > > >>> Stefano Mazzocchi wrote:
> > > > > >>>> Sergey Kuksenko wrote:
> > > > > >>>>
> > > > > >>>>> Lets do the simplest thing fist. :)
> > > > > >>>>> We can do it. We only need to specify a set of workloads.
> > > > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > > > >>>> sun6),
> > > > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > > > >>>> results shortly.
> > > > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > > > >>> 110% of Sun5.
> > > > > >>
> > > > > >> on x86_64?
> > > > > >
> > > > > > That's true. It was x86.
> > > > > >
> > > > > > But the numbers that Robin is reporting aren't great either, are they?
> > > > > >
> > > > > > geir
> > > > > >
> > > > > >>
> > > > >
> > > >
> > >
> > > --
> > > Egor Pasko
> > >
> > >
> >
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Alexey Petrenko <al...@gmail.com>.
2006/11/23, Vladimir Strigun <vs...@gmail.com>:
> On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> > On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > > The numbers that I published was received on P4 under Windows +
> > > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > > x86_64, but I hope we could receive almost the same range (10-20 %
> > > slower that Sun) with the mentioned configuration.
> >
> > And Sun was running with "-server" too I guess? :)
>
> Yes, of course.
>
> > Maybe, it is time to track performance comparisons of *different
> > platforms* in one place? That should help to avoid major differences in
> > our visions for harmony performance.
>
> Yes, good idea. Should we define the configuration for all VM's as well?
> For instance, for Sun we could use parameters from spec site. What do
> you think about it?
+1 for options from spec site.
>
> > * Melody
> > * marmonytest.org
> > * Robin's site
> > * wiki (just for the start, maybe)
> >
> > > Thanks,
> > > Vladimir.
> > >
> > > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > > performance of about 35% of whatever he's measuring against.
> > > >
> > > > geir
> > > >
> > > >
> > > > Geir Magnusson Jr. wrote:
> > > > >
> > > > >
> > > > > Stefano Mazzocchi wrote:
> > > > >> Geir Magnusson Jr. wrote:
> > > > >>>
> > > > >>> Stefano Mazzocchi wrote:
> > > > >>>> Sergey Kuksenko wrote:
> > > > >>>>
> > > > >>>>> Lets do the simplest thing fist. :)
> > > > >>>>> We can do it. We only need to specify a set of workloads.
> > > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > > >>>> sun6),
> > > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > > >>>> results shortly.
> > > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > > >>> 110% of Sun5.
> > > > >>
> > > > >> on x86_64?
> > > > >
> > > > > That's true. It was x86.
> > > > >
> > > > > But the numbers that Robin is reporting aren't great either, are they?
> > > > >
> > > > > geir
> > > > >
> > > > >>
> > > >
> > >
> >
> > --
> > Egor Pasko
> >
> >
>
Re: [performance] a few early benchmarks
Posted by Vladimir Strigun <vs...@gmail.com>.
On 23 Nov 2006 14:37:09 +0600, Egor Pasko <eg...@gmail.com> wrote:
> On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> > The numbers that I published was received on P4 under Windows +
> > server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> > x86_64, but I hope we could receive almost the same range (10-20 %
> > slower that Sun) with the mentioned configuration.
>
> And Sun was running with "-server" too I guess? :)
Yes, of course.
> Maybe, it is time to track performance comparisons of *different
> platforms* in one place? That should help to avoid major differences in
> our visions for harmony performance.
Yes, good idea. Should we define the configuration for all VM's as well?
For instance, for Sun we could use parameters from spec site. What do
you think about it?
> * Melody
> * marmonytest.org
> * Robin's site
> * wiki (just for the start, maybe)
>
> > Thanks,
> > Vladimir.
> >
> > On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > > performance of about 35% of whatever he's measuring against.
> > >
> > > geir
> > >
> > >
> > > Geir Magnusson Jr. wrote:
> > > >
> > > >
> > > > Stefano Mazzocchi wrote:
> > > >> Geir Magnusson Jr. wrote:
> > > >>>
> > > >>> Stefano Mazzocchi wrote:
> > > >>>> Sergey Kuksenko wrote:
> > > >>>>
> > > >>>>> Lets do the simplest thing fist. :)
> > > >>>>> We can do it. We only need to specify a set of workloads.
> > > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > > >>>> sun6),
> > > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > > >>>> results shortly.
> > > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > > >>> 110% of Sun5.
> > > >>
> > > >> on x86_64?
> > > >
> > > > That's true. It was x86.
> > > >
> > > > But the numbers that Robin is reporting aren't great either, are they?
> > > >
> > > > geir
> > > >
> > > >>
> > >
> >
>
> --
> Egor Pasko
>
>
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
We also need to track *precisely* what is being tested.
SVN rev #, any patches, configuration (params) and of course,
platform/OS run on.
geir
Egor Pasko wrote:
> On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
>> The numbers that I published was received on P4 under Windows +
>> server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
>> x86_64, but I hope we could receive almost the same range (10-20 %
>> slower that Sun) with the mentioned configuration.
>
> And Sun was running with "-server" too I guess? :)
> Maybe, it is time to track performance comparisons of *different
> platforms* in one place? That should help to avoid major differences in
> our visions for harmony performance.
>
> * Melody
> * marmonytest.org
> * Robin's site
> * wiki (just for the start, maybe)
>
>> Thanks,
>> Vladimir.
>>
>> On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>>> I reviewed - looks like Robin is seeing DRLVM get an aggregate
>>> performance of about 35% of whatever he's measuring against.
>>>
>>> geir
>>>
>>>
>>> Geir Magnusson Jr. wrote:
>>>>
>>>> Stefano Mazzocchi wrote:
>>>>> Geir Magnusson Jr. wrote:
>>>>>> Stefano Mazzocchi wrote:
>>>>>>> Sergey Kuksenko wrote:
>>>>>>>
>>>>>>>> Lets do the simplest thing fist. :)
>>>>>>>> We can do it. We only need to specify a set of workloads.
>>>>>>> I've tried running dacapo with 10 warming stages and we are constantly
>>>>>>> around 25% speed against the leading JVM (which is always sun5 or
>>>>>>> sun6),
>>>>>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>>>>>>> results shortly.
>>>>>> I don't understand this at all. It wasn't but a few weeks ago when
>>>>>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
>>>>>> 110% of Sun5.
>>>>> on x86_64?
>>>> That's true. It was x86.
>>>>
>>>> But the numbers that Robin is reporting aren't great either, are they?
>>>>
>>>> geir
>>>>
>
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Vladimir Strigun wrote:
> The numbers that I published was received on P4 under Windows +
> server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
> x86_64, but I hope we could receive almost the same range (10-20 %
> slower that Sun) with the mentioned configuration.
And Sun was running with "-server" too I guess? :)
Maybe, it is time to track performance comparisons of *different
platforms* in one place? That should help to avoid major differences in
our visions for harmony performance.
* Melody
* marmonytest.org
* Robin's site
* wiki (just for the start, maybe)
> Thanks,
> Vladimir.
>
> On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > I reviewed - looks like Robin is seeing DRLVM get an aggregate
> > performance of about 35% of whatever he's measuring against.
> >
> > geir
> >
> >
> > Geir Magnusson Jr. wrote:
> > >
> > >
> > > Stefano Mazzocchi wrote:
> > >> Geir Magnusson Jr. wrote:
> > >>>
> > >>> Stefano Mazzocchi wrote:
> > >>>> Sergey Kuksenko wrote:
> > >>>>
> > >>>>> Lets do the simplest thing fist. :)
> > >>>>> We can do it. We only need to specify a set of workloads.
> > >>>> I've tried running dacapo with 10 warming stages and we are constantly
> > >>>> around 25% speed against the leading JVM (which is always sun5 or
> > >>>> sun6),
> > >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> > >>>> results shortly.
> > >>> I don't understand this at all. It wasn't but a few weeks ago when
> > >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> > >>> 110% of Sun5.
> > >>
> > >> on x86_64?
> > >
> > > That's true. It was x86.
> > >
> > > But the numbers that Robin is reporting aren't great either, are they?
> > >
> > > geir
> > >
> > >>
> >
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Vladimir Strigun <vs...@gmail.com>.
The numbers that I published was received on P4 under Windows +
server.emconf +Harmony-1980. Unfortunately I haven't run Dacapo under
x86_64, but I hope we could receive almost the same range (10-20 %
slower that Sun) with the mentioned configuration.
Thanks,
Vladimir.
On 11/23/06, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> I reviewed - looks like Robin is seeing DRLVM get an aggregate
> performance of about 35% of whatever he's measuring against.
>
> geir
>
>
> Geir Magnusson Jr. wrote:
> >
> >
> > Stefano Mazzocchi wrote:
> >> Geir Magnusson Jr. wrote:
> >>>
> >>> Stefano Mazzocchi wrote:
> >>>> Sergey Kuksenko wrote:
> >>>>
> >>>>> Lets do the simplest thing fist. :)
> >>>>> We can do it. We only need to specify a set of workloads.
> >>>> I've tried running dacapo with 10 warming stages and we are constantly
> >>>> around 25% speed against the leading JVM (which is always sun5 or
> >>>> sun6),
> >>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> >>>> results shortly.
> >>> I don't understand this at all. It wasn't but a few weeks ago when
> >>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> >>> 110% of Sun5.
> >>
> >> on x86_64?
> >
> > That's true. It was x86.
> >
> > But the numbers that Robin is reporting aren't great either, are they?
> >
> > geir
> >
> >>
>
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
I reviewed - looks like Robin is seeing DRLVM get an aggregate
performance of about 35% of whatever he's measuring against.
geir
Geir Magnusson Jr. wrote:
>
>
> Stefano Mazzocchi wrote:
>> Geir Magnusson Jr. wrote:
>>>
>>> Stefano Mazzocchi wrote:
>>>> Sergey Kuksenko wrote:
>>>>
>>>>> Lets do the simplest thing fist. :)
>>>>> We can do it. We only need to specify a set of workloads.
>>>> I've tried running dacapo with 10 warming stages and we are constantly
>>>> around 25% speed against the leading JVM (which is always sun5 or
>>>> sun6),
>>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>>>> results shortly.
>>> I don't understand this at all. It wasn't but a few weeks ago when
>>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
>>> 110% of Sun5.
>>
>> on x86_64?
>
> That's true. It was x86.
>
> But the numbers that Robin is reporting aren't great either, are they?
>
> geir
>
>>
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Stefano Mazzocchi wrote:
> Geir Magnusson Jr. wrote:
>>
>> Stefano Mazzocchi wrote:
>>> Sergey Kuksenko wrote:
>>>
>>>> Lets do the simplest thing fist. :)
>>>> We can do it. We only need to specify a set of workloads.
>>> I've tried running dacapo with 10 warming stages and we are constantly
>>> around 25% speed against the leading JVM (which is always sun5 or sun6),
>>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>>> results shortly.
>> I don't understand this at all. It wasn't but a few weeks ago when
>> someone was reporting decapo numbers that ranged from 90% of Sun5 to
>> 110% of Sun5.
>
> on x86_64?
That's true. It was x86.
But the numbers that Robin is reporting aren't great either, are they?
geir
>
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Geir Magnusson Jr. wrote:
>
>
> Stefano Mazzocchi wrote:
>> Sergey Kuksenko wrote:
>>
>>> Lets do the simplest thing fist. :)
>>> We can do it. We only need to specify a set of workloads.
>>
>> I've tried running dacapo with 10 warming stages and we are constantly
>> around 25% speed against the leading JVM (which is always sun5 or sun6),
>> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
>> results shortly.
>
> I don't understand this at all. It wasn't but a few weeks ago when
> someone was reporting decapo numbers that ranged from 90% of Sun5 to
> 110% of Sun5.
on x86_64?
--
Stefano.
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Stefano Mazzocchi wrote:
> Sergey Kuksenko wrote:
>
>> Lets do the simplest thing fist. :)
>> We can do it. We only need to specify a set of workloads.
>
> I've tried running dacapo with 10 warming stages and we are constantly
> around 25% speed against the leading JVM (which is always sun5 or sun6),
> bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
> results shortly.
I don't understand this at all. It wasn't but a few weeks ago when
someone was reporting decapo numbers that ranged from 90% of Sun5 to
110% of Sun5.
geir
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Sergey Kuksenko wrote:
> Lets do the simplest thing fist. :)
> We can do it. We only need to specify a set of workloads.
I've tried running dacapo with 10 warming stages and we are constantly
around 25% speed against the leading JVM (which is always sun5 or sun6),
bea5 is around 80% and ibm5 is around 70%, I'll have more detailed
results shortly.
Robin, any chance you guys can add 'skymark2' to the dacapo benchmarks?
I think that would be helpful.
Also, a great thing would be for dacapo to output an XML serialization
of the benchmark results.
It also appears that we can't run some of the dacapo tests, most notably
antlr (jvm fails with no error and just terminates right during the
first warmup) and chart (because it requires a DISPLAY to be
available... can we run harmony headless yet?)
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Sergey Kuksenko <se...@gmail.com>.
Stefano,
On 11/21/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> Sergey Kuksenko wrote:
> > Stefano,
> > Trying to get the potential of Harmony I've quickly checked SciMak on
> tuned
> > Harmony release build and compared it with BEA & SUN.
>
> > When I looked into Harmony OOB I've found that all hot methods of
> SciMark
> > are compiled by JET (not recompiled by the optimizing JIT compiler). The
> > way
> > our DRLVM currrently recognises hot path is not sutable for SciMark
> becasue
> > of short run.
>
> Hmmm, what do you mean by "short run"? the entire app runs for a short
> amount of time total or each hot method runs for a short amount of time
> not enough to have it recognized as "hot"?
SciMark performs default measurement in 2 seconds range (for each
subbenchmark).
Top subbenmarks methods were not called frequently:
FFT was called ~8200 times.
SOR was called ~ 15 times.
Monte Carlo was called ~26 times.
Sparse matmult was called ~17 times.
LU was called ~2000 times.
It is not enough to have it recognized as "hot" for current DRLVM.
For example. Default measurement time of SciMark is 2 seconds.
This time may be set as parameter. I've set 30 seconds and got the following
results.
BEA(OOB):
Composite Score (default, 2 secs): 435
Composite Score (30 secs): 438
SUN(OOB):
Composite Score (default, 2 secs): 229
Composite Score (30 secs): 225
Harmony(OOB):
Composite Score (default, 2 secs): 109
Composite Score (30 secs): 189
Thus the first problem here - is default Harmony configuration, we
should extend recompilation and only after that we can look into real
performance.
> We need to tune DRLVM options to get better results.
> > Tuned options give good SciMark score improvement (109->181).
>
> Well, to be fair, all the other JVM could probably do the same.
Other VM already did it. :)
The first problem is recompilation.
> Which moves Harmony performance close to what Sun OOB shows.
>
> excuse my ignorance, but what's OOB? (google define says "out of
> business" or "order of battle"... not sure they apply here ;-)
"out of the box" - by default, no VM options.
> Our client (default) compilation path was tuned a long time ago and it
> > probably makes sense to have another round. What we initially did was
> > running some script executing the given set of workloads trying to find
> the
> > best configuration for our VM. Having said that I suggest we choose the
> > right set of applications/benchmarks, so we can start our tuning once
> > again.
>
> Maybe it's the analog microelectonic guy in me talking, but every time I
> hear something like "let's get reasonable defaults", I think of
> introducing a variation and a feedback to reach a local minimum and
> stabilize the system.
>
> I know very little about how DRLVM works, but would it be feasible to
> start with such "reasonable defaults" and introduce a random variability
> to the way the JIT works alongside a very simple method profiler and see
> if the performance increase? think of you trying out different things
> and see if they work better... but done by the JVM as it runs.
You are absolutely right.
But it will have a low impact to short benchmarks (like SciMark) because
profiler need time.
Also my current suggestion is change (improve) "starting point" that also
may help for further profiling.
I suggested to specify a set of workloads which looks like "reasonable"
and find a common set of DRLVM options. These options can't be optimal for
each separate application, but they will improve overall performance (for
selected set).
I think it is possible to get overall improvement because current default
DRLVM options were set a long time ago.
Keep in mind I'm a total newbie in virtual machine design (or CPU
> architectures for that matter, despite my degree in microelectronics..
> well, to be fair, I was doing analog not digital circuits) so bear with
> me if I'm saying stupid things :-)
>
> > Currently we have in mind the following list:
> > - HWA (Hello World Application)
> > - SciMark
> > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and
> xalan)
> > - Anything else?
> >
> > What do you think about this? Any additions to the list? Comments?
> > Questions?
>
> The problem I have with this is that I feel that each one of such
> scenario might require different tuning parameters... and if that is the
> case, you end up with the 'short blanket' problem: you improve here and
> you decrease there.
You are right.
It is impossible to find the best options for all applications.
But I am afraid that current DRLVM default options are obsolete and I hope
there is a place for overall improvement.
An 'adaptive' scenario, on the other hand, would allow us to:
>
> 1) avoid trying to find the optimal defaults (since we can't possible
> test every scenario that will be useful in a way that is consistent with
> real world usage)
>
> 2) avoid the blanket problem, each VM can adapt to the scenario of use
>
> 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> changes and 'retune' itself if the environment changes.
>
> Of course, there is a price to pay in such 'fedback variability' systems
> since they have to find the minima over and over again.
>
> So, another solution is to have a JVM "tuning parameters discovery mode"
> that you can run and you turn such "parameter finding" autoprofiling
> on... and the JVM dumps the tuning results for you on disk which you can
> later use to initialize the JVM on your own.
>
> Not sure how feasible or complicated to write this is, but wow does this
> sound on paper?
Yes. Adaptive tuning is a mandatory part of VM and it is more important then
my suggestion.
But it is not so easy as starting point changing.
Lets do the simplest thing fist. :)
We can do it. We only need to specify a set of workloads.
Best regards,
---
Sergey Kuksenko.
Intel Enterprise Solutions Software Division.
> Thanks,
> > ---
> > Sergey Kuksenko
> > Intel Enterprise Solutions Software Division
> >
> >
> > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> >>
> >> Alexey Varlamov wrote:
> >> > Stefano,
> >> >
> >> > It is a bit unfair to compare *debug* build of Harmony with other
> >> > release versions :)
> >>
> >> I'm simulating what a journalist with a developer could do.
> >>
> >> If there is a way to make it compile in 'release mode' (if such a thing
> >> exists), I'll be very glad to redo the benchmarks.
> >>
> >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> >> > switches)?
> >>
> >> Right. No switches. I'm simulating what users do when they get the JVM:
> >> they run "java"... and if it's now fast enough they buy a new box.
> >>
> >> Having command line tuning parameters is mostly useless since most
> >> people don't know the internals of a JVM well enough to guess what
> >> parameters to tune anyway.
> >>
> >> So, what people will do once they get an harmony snapshot is "java
> >> my.class.Name <http://my.class.name/>" and see the results.
> >>
> >> I want to simulate that and compare it to the same exact experience
> they
> >> will get with other virtual machines for a variety of common scenarios
> >> (number crunching, xml processing, http serving, database load, etc...)
> >>
> >> I will focus on the server because that's there the apache action (and
> >> my personal interest) is.
> >>
> >> So, like I said, if there are 'compile time' switches that I can use to
> >> turn 'release mode' on, please tell me and I'll re-do the tests.
> >>
> >> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> >> >> There are lies, damn lies and benchmarks.... which don't really tell
> >> you
> >> >> if an implementation of a program is *faster* but at least it tells
> >> you
> >> >> where you're at.
> >> >>
> >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM,
> I
> >> >> was able to start running some benchmarks.
> >> >>
> >> >> The machine is the following:
> >> >>
> >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep
> 16
> >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >> >>
> >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> >> bogomips 6410.31 (per CPU)
> >> >>
> >> >> There is nothing else running on the machine (load is 0.04 at the
> time
> >> >> of testing).
> >> >>
> >> >> The various virtual machines tested are:
> >> >>
> >> >> harmony
> >> >> -------
> >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache
> Software
> >> >> Foundation or its licensors, as applicable.
> >> >> java version " 1.5.0"
> >> >> pre-alpha : not complete or compatible
> >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >> >>
> >> >> sun5
> >> >> ---
> >> >> java version "1.5.0_09 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03
> )
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >> >>
> >> >> sun6
> >> >> ----
> >> >> java version " 1.6.0-rc"
> >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >> >>
> >> >> ibm
> >> >> ---
> >> >> java version " 1.5.0"
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> >> pxa64dev-20061002a (SR3) )
> >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> >> j9vmxa6423-20061001 (JIT enabled)
> >> >> J9VM - 20060915_08260_LHdSMr
> >> >> JIT - 20060908_1811_r8
> >> >> GC - 20060906_AA)
> >> >> JCL - 20061002
> >> >>
> >> >> bea
> >> >> ---
> >> >> java version "1.5.0_06 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05
> )
> >> >> BEA JRockit(R) (build
> >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >> >>
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >> >>
> >> >> command: java jnt.scimark2.commandline
> >> >>
> >> >> NOTE: bigger number is better
> >> >>
> >> >> Sun6
> >> >> Composite Score: 364.5832265230057
> >> >> FFT (1024): 220.8458713892794
> >> >> SOR (100x100): 696.1542342357722
> >> >> Monte Carlo : 149.37978088875656
> >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> >> LU (100x100): 430.1617273683819
> >> >>
> >> >> BEA
> >> >> Composite Score: 359.13480378697835
> >> >> FFT (1024): 303.8746880751562
> >> >> SOR (100x100): 454.25628897202307
> >> >> Monte Carlo : 93.23913192138497
> >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> >> LU (100x100): 413.8627835924175
> >> >>
> >> >> Sun5
> >> >> Composite Score: 332.84987587548574
> >> >> FFT (1024): 216.5144595799027
> >> >> SOR (100x100): 689.429322146947
> >> >> Monte Carlo : 25.791262124978065
> >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> >> LU (100x100): 414.99493895566377
> >> >>
> >> >> IBM
> >> >> Composite Score: 259.8249218693683
> >> >> FFT (1024): 296.8415012789055
> >> >> SOR (100x100): 428.974881649179
> >> >> Monte Carlo : 89.15159857584082
> >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> >> LU (100x100): 339.8042037225181
> >> >>
> >> >> Harmony
> >> >> Composite Score: 113.65082278962575
> >> >> FFT (1024): 203.76641991778123
> >> >> SOR (100x100): 224.37761309236748
> >> >> Monte Carlo : 9.063866256533116
> >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> >> LU (100x100): 65.6410280487242
> >> >>
> >> >> In this test harmony is clearly lagging behind... at about 30%
> >> >> performance of the best JVM, it's a little crappy. Please note how
> >> FFT's
> >> >> performance is not so bad awhile monte carlo is pretty bad compared
> to
> >> >> BEA or IBM.
> >> >>
> >> >> Overall, it seems like there is some serious work to do here to
> catch
> >> up.
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 2: Dhrystones
> >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> >> )
> >> >>
> >> >> command: java dhry 100000000
> >> >>
> >> >> NOTE: bigger is better
> >> >>
> >> >> NB: I modified the code to accept the count at input from the
> command
> >> >> line!
> >> >>
> >> >> sun6: 8552856 dhrystones/sec
> >> >> sun5: 6605892
> >> >> bea: 5678914
> >> >> harmony: 669734
> >> >> ibm: 501562
> >> >>
> >> >> The performance here is horrific but what's surprising is that J9 is
> >> >> even worse. No idea what's going on but it seems like something is
> not
> >> >> working as it should (in both harmony and J9)
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >> >>
> >> >> command: java Sieve 30
> >> >>
> >> >> NB: I modified the test to run for a configurable amount of seconds.
> >> >>
> >> >> sun6 8545 sieves/sec
> >> >> sun5 8364
> >> >> bea 6174
> >> >> harmony 1836
> >> >> ibm 225
> >> >>
> >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> >> lagging behind.
> >> >>
> >> >> Stay tuned for more tests.
> >> >>
> >> >> --
> >> >> Stefano.
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Stefano.
> >>
> >>
> >
>
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Pavel Ozhdikhin <pa...@gmail.com>.
> Currently we have in mind the following list:
> - HWA (Hello World Application)
> - SciMark
> - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> - Anything else?
Smoke tests and kernel tests - this is the applications we run most often!
Relax, it was a joke. :)
I would add Eclipse-based scenarios or Eclipse unit tests representing usual
developer's acrivities - coding, building etc. In the default DRLVM mode we
should also account application startup time for big apps such as Eclipse
again.
Thanks,
Pavel
On 11/21/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> Sergey Kuksenko wrote:
> > Stefano,
> > Trying to get the potential of Harmony I've quickly checked SciMak on
> tuned
> > Harmony release build and compared it with BEA & SUN.
>
> Sergey,
>
> many thanks for doing this.
>
> > Hardware: P4 Xeon 3GHz
> > Windows XP SP2 (It's another platform, but I hope the key things are
> still
> > the same).
> >
> > BEA -
> > java version "1.5.0_06"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > BEA JRockit(R) (build R26.3.0-32-58710-1.5.0_06-20060308-2022-win-ia32,
> )
> >
> > SUN -
> > java version "1.5.0_06"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)
> >
> >
> > Harmony -
> > Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> > Foundation or its licensors, as applicable.
> > java version "1.5.0"
> > pre-alpha : not complete or compatible
> > svn = 475925, (Nov 17 2006), Windows/ia32/msvc 1310, release build
> >
> > I've got the following results
> >
> > BEA (out of the box):
> >
> > Composite Score: 435.9674695335291
> > FFT (1024): 295.33366058958575
> > SOR (100x100): 474.15229982839213
> > Monte Carlo : 111.56918839504195
> > Sparse matmult (N=1000, nz=5000): 551.8821052631578
> > LU (100x100): 746.9000935914679
> > ----
> >
> > Sun (out of the box):
> >
> > Composite Score: 229.70779446543412
> > FFT (1024): 104.92303791891565
> > SOR (100x100): 400.44785722405015
> > Monte Carlo : 13.257380552894444
> > Sparse matmult (N=1000, nz=5000): 160.07814989061512
> > LU (100x100): 469.8325467406951
> >
> > ---
> >
> > Harmony (out of the box):
> >
> > Composite Score: 109.43208528481887
> > FFT (1024): 51.30119529411764
> > SOR (100x100): 257.9591618631154
> > Monte Carlo : 17.04568642272773
> > Sparse matmult (N=1000, nz=5000): 129.4666069618598
> > LU (100x100): 91.38777588227376
> > ----
> >
> > Harmony (tuned options, server path):
> >
> > Composite Score: 181.54555681031619
> > FFT (1024): 91.22597999162443
> > SOR (100x100): 329.8450882375011
> > Monte Carlo : 42.51432538579417
> > Sparse matmult (N=1000, nz=5000): 260.58050602943024
> > LU (100x100): 183.56188440723088
>
> that's pretty good.
>
> > ------
> >
> > When I looked into Harmony OOB I've found that all hot methods of
> SciMark
> > are compiled by JET (not recompiled by the optimizing JIT compiler). The
> > way
> > our DRLVM currrently recognises hot path is not sutable for SciMark
> becasue
> > of short run.
>
> Hmmm, what do you mean by "short run"? the entire app runs for a short
> amount of time total or each hot method runs for a short amount of time
> not enough to have it recognized as "hot"?
>
> > We need to tune DRLVM options to get better results.
> > Tuned options give good SciMark score improvement (109->181).
>
> Well, to be fair, all the other JVM could probably do the same.
>
> > Which moves Harmony performance close to what Sun OOB shows.
>
> excuse my ignorance, but what's OOB? (google define says "out of
> business" or "order of battle"... not sure they apply here ;-)
>
> > Our client (default) compilation path was tuned a long time ago and it
> > probably makes sense to have another round. What we initially did was
> > running some script executing the given set of workloads trying to find
> the
> > best configuration for our VM. Having said that I suggest we choose the
> > right set of applications/benchmarks, so we can start our tuning once
> > again.
>
> Maybe it's the analog microelectonic guy in me talking, but every time I
> hear something like "let's get reasonable defaults", I think of
> introducing a variation and a feedback to reach a local minimum and
> stabilize the system.
>
> I know very little about how DRLVM works, but would it be feasible to
> start with such "reasonable defaults" and introduce a random variability
> to the way the JIT works alongside a very simple method profiler and see
> if the performance increase? think of you trying out different things
> and see if they work better... but done by the JVM as it runs.
>
> Keep in mind I'm a total newbie in virtual machine design (or CPU
> architectures for that matter, despite my degree in microelectronics..
> well, to be fair, I was doing analog not digital circuits) so bear with
> me if I'm saying stupid things :-)
>
> > Currently we have in mind the following list:
> > - HWA (Hello World Application)
> > - SciMark
> > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and
> xalan)
> > - Anything else?
> >
> > What do you think about this? Any additions to the list? Comments?
> > Questions?
>
> The problem I have with this is that I feel that each one of such
> scenario might require different tuning parameters... and if that is the
> case, you end up with the 'short blanket' problem: you improve here and
> you decrease there.
>
> An 'adaptive' scenario, on the other hand, would allow us to:
>
> 1) avoid trying to find the optimal defaults (since we can't possible
> test every scenario that will be useful in a way that is consistent with
> real world usage)
>
> 2) avoid the blanket problem, each VM can adapt to the scenario of use
>
> 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> changes and 'retune' itself if the environment changes.
>
> Of course, there is a price to pay in such 'fedback variability' systems
> since they have to find the minima over and over again.
>
> So, another solution is to have a JVM "tuning parameters discovery mode"
> that you can run and you turn such "parameter finding" autoprofiling
> on... and the JVM dumps the tuning results for you on disk which you can
> later use to initialize the JVM on your own.
>
> Not sure how feasible or complicated to write this is, but wow does this
> sound on paper?
>
>
> > Thanks,
> > ---
> > Sergey Kuksenko
> > Intel Enterprise Solutions Software Division
> >
> >
> > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> >>
> >> Alexey Varlamov wrote:
> >> > Stefano,
> >> >
> >> > It is a bit unfair to compare *debug* build of Harmony with other
> >> > release versions :)
> >>
> >> I'm simulating what a journalist with a developer could do.
> >>
> >> If there is a way to make it compile in 'release mode' (if such a thing
> >> exists), I'll be very glad to redo the benchmarks.
> >>
> >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> >> > switches)?
> >>
> >> Right. No switches. I'm simulating what users do when they get the JVM:
> >> they run "java"... and if it's now fast enough they buy a new box.
> >>
> >> Having command line tuning parameters is mostly useless since most
> >> people don't know the internals of a JVM well enough to guess what
> >> parameters to tune anyway.
> >>
> >> So, what people will do once they get an harmony snapshot is "java
> >> my.class.Name <http://my.class.name/>" and see the results.
> >>
> >> I want to simulate that and compare it to the same exact experience
> they
> >> will get with other virtual machines for a variety of common scenarios
> >> (number crunching, xml processing, http serving, database load, etc...)
> >>
> >> I will focus on the server because that's there the apache action (and
> >> my personal interest) is.
> >>
> >> So, like I said, if there are 'compile time' switches that I can use to
> >> turn 'release mode' on, please tell me and I'll re-do the tests.
> >>
> >> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> >> >> There are lies, damn lies and benchmarks.... which don't really tell
> >> you
> >> >> if an implementation of a program is *faster* but at least it tells
> >> you
> >> >> where you're at.
> >> >>
> >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM,
> I
> >> >> was able to start running some benchmarks.
> >> >>
> >> >> The machine is the following:
> >> >>
> >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep
> 16
> >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >> >>
> >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> >> bogomips 6410.31 (per CPU)
> >> >>
> >> >> There is nothing else running on the machine (load is 0.04 at the
> time
> >> >> of testing).
> >> >>
> >> >> The various virtual machines tested are:
> >> >>
> >> >> harmony
> >> >> -------
> >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache
> Software
> >> >> Foundation or its licensors, as applicable.
> >> >> java version " 1.5.0"
> >> >> pre-alpha : not complete or compatible
> >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >> >>
> >> >> sun5
> >> >> ---
> >> >> java version "1.5.0_09 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03
> )
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >> >>
> >> >> sun6
> >> >> ----
> >> >> java version " 1.6.0-rc"
> >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >> >>
> >> >> ibm
> >> >> ---
> >> >> java version " 1.5.0"
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> >> pxa64dev-20061002a (SR3) )
> >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> >> j9vmxa6423-20061001 (JIT enabled)
> >> >> J9VM - 20060915_08260_LHdSMr
> >> >> JIT - 20060908_1811_r8
> >> >> GC - 20060906_AA)
> >> >> JCL - 20061002
> >> >>
> >> >> bea
> >> >> ---
> >> >> java version "1.5.0_06 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05
> )
> >> >> BEA JRockit(R) (build
> >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >> >>
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >> >>
> >> >> command: java jnt.scimark2.commandline
> >> >>
> >> >> NOTE: bigger number is better
> >> >>
> >> >> Sun6
> >> >> Composite Score: 364.5832265230057
> >> >> FFT (1024): 220.8458713892794
> >> >> SOR (100x100): 696.1542342357722
> >> >> Monte Carlo : 149.37978088875656
> >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> >> LU (100x100): 430.1617273683819
> >> >>
> >> >> BEA
> >> >> Composite Score: 359.13480378697835
> >> >> FFT (1024): 303.8746880751562
> >> >> SOR (100x100): 454.25628897202307
> >> >> Monte Carlo : 93.23913192138497
> >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> >> LU (100x100): 413.8627835924175
> >> >>
> >> >> Sun5
> >> >> Composite Score: 332.84987587548574
> >> >> FFT (1024): 216.5144595799027
> >> >> SOR (100x100): 689.429322146947
> >> >> Monte Carlo : 25.791262124978065
> >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> >> LU (100x100): 414.99493895566377
> >> >>
> >> >> IBM
> >> >> Composite Score: 259.8249218693683
> >> >> FFT (1024): 296.8415012789055
> >> >> SOR (100x100): 428.974881649179
> >> >> Monte Carlo : 89.15159857584082
> >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> >> LU (100x100): 339.8042037225181
> >> >>
> >> >> Harmony
> >> >> Composite Score: 113.65082278962575
> >> >> FFT (1024): 203.76641991778123
> >> >> SOR (100x100): 224.37761309236748
> >> >> Monte Carlo : 9.063866256533116
> >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> >> LU (100x100): 65.6410280487242
> >> >>
> >> >> In this test harmony is clearly lagging behind... at about 30%
> >> >> performance of the best JVM, it's a little crappy. Please note how
> >> FFT's
> >> >> performance is not so bad awhile monte carlo is pretty bad compared
> to
> >> >> BEA or IBM.
> >> >>
> >> >> Overall, it seems like there is some serious work to do here to
> catch
> >> up.
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 2: Dhrystones
> >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> >> )
> >> >>
> >> >> command: java dhry 100000000
> >> >>
> >> >> NOTE: bigger is better
> >> >>
> >> >> NB: I modified the code to accept the count at input from the
> command
> >> >> line!
> >> >>
> >> >> sun6: 8552856 dhrystones/sec
> >> >> sun5: 6605892
> >> >> bea: 5678914
> >> >> harmony: 669734
> >> >> ibm: 501562
> >> >>
> >> >> The performance here is horrific but what's surprising is that J9 is
> >> >> even worse. No idea what's going on but it seems like something is
> not
> >> >> working as it should (in both harmony and J9)
> >> >>
> >> >>
> >>
> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >> >>
> >> >> command: java Sieve 30
> >> >>
> >> >> NB: I modified the test to run for a configurable amount of seconds.
> >> >>
> >> >> sun6 8545 sieves/sec
> >> >> sun5 8364
> >> >> bea 6174
> >> >> harmony 1836
> >> >> ibm 225
> >> >>
> >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> >> lagging behind.
> >> >>
> >> >> Stay tuned for more tests.
> >> >>
> >> >> --
> >> >> Stefano.
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Stefano.
> >>
> >>
> >
>
>
> --
> Stefano.
>
>
Re: [performance] DaCapo benchmarks
Posted by Robin Garner <ro...@anu.edu.au>.
I've also added performance results to the DaCapo regression tests. If
you page down on the front page
http://cs.anu.edu.au/people/Robin.Garner/dacapo/regression/
You'll see a bunch of graphs, comparing the performance of JikesRVM with
DRLVM. The numbers are all relative to the best figure from any of the
commercial VMs I had available.
cheers
Stefano Mazzocchi wrote:
> Sergey Kuksenko wrote:
>> Stefano,
>> Trying to get the potential of Harmony I've quickly checked SciMak on tuned
>> Harmony release build and compared it with BEA & SUN.
>
> Sergey,
>
> many thanks for doing this.
>
>> Hardware: P4 Xeon 3GHz
>> Windows XP SP2 (It's another platform, but I hope the key things are still
>> the same).
>>
>> BEA -
>> java version "1.5.0_06"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> BEA JRockit(R) (build R26.3.0-32-58710-1.5.0_06-20060308-2022-win-ia32, )
>>
>> SUN -
>> java version "1.5.0_06"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)
>>
>>
>> Harmony -
>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> Foundation or its licensors, as applicable.
>> java version "1.5.0"
>> pre-alpha : not complete or compatible
>> svn = 475925, (Nov 17 2006), Windows/ia32/msvc 1310, release build
>>
>> I've got the following results
>>
>> BEA (out of the box):
>>
>> Composite Score: 435.9674695335291
>> FFT (1024): 295.33366058958575
>> SOR (100x100): 474.15229982839213
>> Monte Carlo : 111.56918839504195
>> Sparse matmult (N=1000, nz=5000): 551.8821052631578
>> LU (100x100): 746.9000935914679
>> ----
>>
>> Sun (out of the box):
>>
>> Composite Score: 229.70779446543412
>> FFT (1024): 104.92303791891565
>> SOR (100x100): 400.44785722405015
>> Monte Carlo : 13.257380552894444
>> Sparse matmult (N=1000, nz=5000): 160.07814989061512
>> LU (100x100): 469.8325467406951
>>
>> ---
>>
>> Harmony (out of the box):
>>
>> Composite Score: 109.43208528481887
>> FFT (1024): 51.30119529411764
>> SOR (100x100): 257.9591618631154
>> Monte Carlo : 17.04568642272773
>> Sparse matmult (N=1000, nz=5000): 129.4666069618598
>> LU (100x100): 91.38777588227376
>> ----
>>
>> Harmony (tuned options, server path):
>>
>> Composite Score: 181.54555681031619
>> FFT (1024): 91.22597999162443
>> SOR (100x100): 329.8450882375011
>> Monte Carlo : 42.51432538579417
>> Sparse matmult (N=1000, nz=5000): 260.58050602943024
>> LU (100x100): 183.56188440723088
>
> that's pretty good.
>
>> ------
>>
>> When I looked into Harmony OOB I've found that all hot methods of SciMark
>> are compiled by JET (not recompiled by the optimizing JIT compiler). The
>> way
>> our DRLVM currrently recognises hot path is not sutable for SciMark becasue
>> of short run.
>
> Hmmm, what do you mean by "short run"? the entire app runs for a short
> amount of time total or each hot method runs for a short amount of time
> not enough to have it recognized as "hot"?
>
>> We need to tune DRLVM options to get better results.
>> Tuned options give good SciMark score improvement (109->181).
>
> Well, to be fair, all the other JVM could probably do the same.
>
>> Which moves Harmony performance close to what Sun OOB shows.
>
> excuse my ignorance, but what's OOB? (google define says "out of
> business" or "order of battle"... not sure they apply here ;-)
>
>> Our client (default) compilation path was tuned a long time ago and it
>> probably makes sense to have another round. What we initially did was
>> running some script executing the given set of workloads trying to find the
>> best configuration for our VM. Having said that I suggest we choose the
>> right set of applications/benchmarks, so we can start our tuning once
>> again.
>
> Maybe it's the analog microelectonic guy in me talking, but every time I
> hear something like "let's get reasonable defaults", I think of
> introducing a variation and a feedback to reach a local minimum and
> stabilize the system.
>
> I know very little about how DRLVM works, but would it be feasible to
> start with such "reasonable defaults" and introduce a random variability
> to the way the JIT works alongside a very simple method profiler and see
> if the performance increase? think of you trying out different things
> and see if they work better... but done by the JVM as it runs.
>
> Keep in mind I'm a total newbie in virtual machine design (or CPU
> architectures for that matter, despite my degree in microelectronics..
> well, to be fair, I was doing analog not digital circuits) so bear with
> me if I'm saying stupid things :-)
>
>> Currently we have in mind the following list:
>> - HWA (Hello World Application)
>> - SciMark
>> - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
>> - Anything else?
>>
>> What do you think about this? Any additions to the list? Comments?
>> Questions?
>
> The problem I have with this is that I feel that each one of such
> scenario might require different tuning parameters... and if that is the
> case, you end up with the 'short blanket' problem: you improve here and
> you decrease there.
>
> An 'adaptive' scenario, on the other hand, would allow us to:
>
> 1) avoid trying to find the optimal defaults (since we can't possible
> test every scenario that will be useful in a way that is consistent with
> real world usage)
>
> 2) avoid the blanket problem, each VM can adapt to the scenario of use
>
> 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> changes and 'retune' itself if the environment changes.
>
> Of course, there is a price to pay in such 'fedback variability' systems
> since they have to find the minima over and over again.
>
> So, another solution is to have a JVM "tuning parameters discovery mode"
> that you can run and you turn such "parameter finding" autoprofiling
> on... and the JVM dumps the tuning results for you on disk which you can
> later use to initialize the JVM on your own.
>
> Not sure how feasible or complicated to write this is, but wow does this
> sound on paper?
>
>
>> Thanks,
>> ---
>> Sergey Kuksenko
>> Intel Enterprise Solutions Software Division
>>
>>
>> On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>>> Alexey Varlamov wrote:
>>>> Stefano,
>>>>
>>>> It is a bit unfair to compare *debug* build of Harmony with other
>>>> release versions :)
>>> I'm simulating what a journalist with a developer could do.
>>>
>>> If there is a way to make it compile in 'release mode' (if such a thing
>>> exists), I'll be very glad to redo the benchmarks.
>>>
>>>> I suppose all VMs where run in default mode (i.e. no special cmd-line
>>>> switches)?
>>> Right. No switches. I'm simulating what users do when they get the JVM:
>>> they run "java"... and if it's now fast enough they buy a new box.
>>>
>>> Having command line tuning parameters is mostly useless since most
>>> people don't know the internals of a JVM well enough to guess what
>>> parameters to tune anyway.
>>>
>>> So, what people will do once they get an harmony snapshot is "java
>>> my.class.Name <http://my.class.name/>" and see the results.
>>>
>>> I want to simulate that and compare it to the same exact experience they
>>> will get with other virtual machines for a variety of common scenarios
>>> (number crunching, xml processing, http serving, database load, etc...)
>>>
>>> I will focus on the server because that's there the apache action (and
>>> my personal interest) is.
>>>
>>> So, like I said, if there are 'compile time' switches that I can use to
>>> turn 'release mode' on, please tell me and I'll re-do the tests.
>>>
>>>> 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
>>>>> There are lies, damn lies and benchmarks.... which don't really tell
>>> you
>>>>> if an implementation of a program is *faster* but at least it tells
>>> you
>>>>> where you're at.
>>>>>
>>>>> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>>>>> was able to start running some benchmarks.
>>>>>
>>>>> The machine is the following:
>>>>>
>>>>> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>>>>> 01:50:50 UTC 2006 x86_64 GNU/Linux
>>>>>
>>>>> dual Intel(R) Pentium(R) D CPU 3.20GHz
>>>>> bogomips 6410.31 (per CPU)
>>>>>
>>>>> There is nothing else running on the machine (load is 0.04 at the time
>>>>> of testing).
>>>>>
>>>>> The various virtual machines tested are:
>>>>>
>>>>> harmony
>>>>> -------
>>>>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>>>>> Foundation or its licensors, as applicable.
>>>>> java version " 1.5.0"
>>>>> pre-alpha : not complete or compatible
>>>>> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>>>>>
>>>>> sun5
>>>>> ---
>>>>> java version "1.5.0_09 "
>>>>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>>>>>
>>>>> sun6
>>>>> ----
>>>>> java version " 1.6.0-rc"
>>>>> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>>>>> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>>>>>
>>>>> ibm
>>>>> ---
>>>>> java version " 1.5.0"
>>>>> Java(TM) 2 Runtime Environment, Standard Edition (build
>>>>> pxa64dev-20061002a (SR3) )
>>>>> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>>>>> j9vmxa6423-20061001 (JIT enabled)
>>>>> J9VM - 20060915_08260_LHdSMr
>>>>> JIT - 20060908_1811_r8
>>>>> GC - 20060906_AA)
>>>>> JCL - 20061002
>>>>>
>>>>> bea
>>>>> ---
>>>>> java version "1.5.0_06 "
>>>>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>>>>> BEA JRockit(R) (build
>>>>> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>>>>>
>>>>>
>>>>>
>>> --------------------------------------------------------------------------
>>>
>>>>>
>>>>> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>>>>>
>>>>> command: java jnt.scimark2.commandline
>>>>>
>>>>> NOTE: bigger number is better
>>>>>
>>>>> Sun6
>>>>> Composite Score: 364.5832265230057
>>>>> FFT (1024): 220.8458713892794
>>>>> SOR (100x100): 696.1542342357722
>>>>> Monte Carlo : 149.37978088875656
>>>>> Sparse matmult (N=1000, nz=5000): 326.37451873283845
>>>>> LU (100x100): 430.1617273683819
>>>>>
>>>>> BEA
>>>>> Composite Score: 359.13480378697835
>>>>> FFT (1024): 303.8746880751562
>>>>> SOR (100x100): 454.25628897202307
>>>>> Monte Carlo : 93.23913192138497
>>>>> Sparse matmult (N=1000, nz=5000): 530.44112637391
>>>>> LU (100x100): 413.8627835924175
>>>>>
>>>>> Sun5
>>>>> Composite Score: 332.84987587548574
>>>>> FFT (1024): 216.5144595799027
>>>>> SOR (100x100): 689.429322146947
>>>>> Monte Carlo : 25.791262124978065
>>>>> Sparse matmult (N=1000, nz=5000): 317.5193965699373
>>>>> LU (100x100): 414.99493895566377
>>>>>
>>>>> IBM
>>>>> Composite Score: 259.8249218693683
>>>>> FFT (1024): 296.8415012789055
>>>>> SOR (100x100): 428.974881649179
>>>>> Monte Carlo : 89.15159857584082
>>>>> Sparse matmult (N=1000, nz=5000): 144.3524241203982
>>>>> LU (100x100): 339.8042037225181
>>>>>
>>>>> Harmony
>>>>> Composite Score: 113.65082278962575
>>>>> FFT (1024): 203.76641991778123
>>>>> SOR (100x100): 224.37761309236748
>>>>> Monte Carlo : 9.063866256533116
>>>>> Sparse matmult (N=1000, nz=5000): 65.4051866327227
>>>>> LU (100x100): 65.6410280487242
>>>>>
>>>>> In this test harmony is clearly lagging behind... at about 30%
>>>>> performance of the best JVM, it's a little crappy. Please note how
>>> FFT's
>>>>> performance is not so bad awhile monte carlo is pretty bad compared to
>>>>> BEA or IBM.
>>>>>
>>>>> Overall, it seems like there is some serious work to do here to catch
>>> up.
>>>>>
>>> --------------------------------------------------------------------------
>>>
>>>>>
>>>>> Test 2: Dhrystones
>>> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
>>> )
>>>>> command: java dhry 100000000
>>>>>
>>>>> NOTE: bigger is better
>>>>>
>>>>> NB: I modified the code to accept the count at input from the command
>>>>> line!
>>>>>
>>>>> sun6: 8552856 dhrystones/sec
>>>>> sun5: 6605892
>>>>> bea: 5678914
>>>>> harmony: 669734
>>>>> ibm: 501562
>>>>>
>>>>> The performance here is horrific but what's surprising is that J9 is
>>>>> even worse. No idea what's going on but it seems like something is not
>>>>> working as it should (in both harmony and J9)
>>>>>
>>>>>
>>> --------------------------------------------------------------------------
>>>
>>>>>
>>>>> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>>>>>
>>>>> command: java Sieve 30
>>>>>
>>>>> NB: I modified the test to run for a configurable amount of seconds.
>>>>>
>>>>> sun6 8545 sieves/sec
>>>>> sun5 8364
>>>>> bea 6174
>>>>> harmony 1836
>>>>> ibm 225
>>>>>
>>>>> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>>>>> lagging behind.
>>>>>
>>>>> Stay tuned for more tests.
>>>>>
>>>>> --
>>>>> Stefano.
>>>>>
>>>>>
>>>
>>> --
>>> Stefano.
>>>
>>>
>
>
--
Robin Garner
Dept. of Computer Science
Australian National University
http://cs.anu.edu.au/people/Robin.Garner/
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x22A day of Apache Harmony Alexey Varlamov wrote:
> [snip]
> > We can try to tune DRLVM default mode for HelloWorld + Eclipse +
> > SciMark + DaCapo, would it be enough for users to be happy with
> > Harmony on their first glance? I dunno, actually. IMHO, almost all
> > java users taking care of performance know about client and server
> > modes and what applications are best for each mode.
>
> Err, SciMark looks quite specific in this list, and I suspect it will
> provoke that "short blanket" scenario. Probably we should not include
> it for the default mode tuning.
What can we do:
* agree that SciMark is not for DRLVM default mode (=do nothing)
* take SciMark into account, make a weighted metric (with SciMark at
low priority, of course)
We should be "reasonably" fast, is 4x slowdown on SciMark reasonable?
Tuning decisions are always not easy...
> As a side note, any user cares about performance but not all dare to
> tune it.
Thank you for the side note)) There is going to be the "short blanket"
story forever or at least for a while. And options are all we can do
for those who care. I think, we all know that.
> This is the main motivation for measuring OOB at the first place...
Now I know, OOB=OutOfBox :)
> >
> > > > Currently we have in mind the following list:
> > > > - HWA (Hello World Application)
> > > > - SciMark
> > > > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > > > - Anything else?
> > > >
> > > > What do you think about this? Any additions to the list? Comments?
> > > > Questions?
> > >
> > > The problem I have with this is that I feel that each one of such
> > > scenario might require different tuning parameters... and if that is the
> > > case, you end up with the 'short blanket' problem: you improve here and
> > > you decrease there.
> > >
> > > An 'adaptive' scenario, on the other hand, would allow us to:
> > >
> > > 1) avoid trying to find the optimal defaults (since we can't possible
> > > test every scenario that will be useful in a way that is consistent with
> > > real world usage)
> > >
> > > 2) avoid the blanket problem, each VM can adapt to the scenario of use
> > >
> > > 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> > > changes and 'retune' itself if the environment changes.
> > >
> > > Of course, there is a price to pay in such 'fedback variability' systems
> > > since they have to find the minima over and over again.
> > >
> > > So, another solution is to have a JVM "tuning parameters discovery mode"
> > > that you can run and you turn such "parameter finding" autoprofiling
> > > on... and the JVM dumps the tuning results for you on disk which you can
> > > later use to initialize the JVM on your own.
> > >
> > > Not sure how feasible or complicated to write this is, but wow does this
> > > sound on paper?
> > >
> > >
> > > > Thanks,
> > > > ---
> > > > Sergey Kuksenko
> > > > Intel Enterprise Solutions Software Division
> > > >
> > > >
> > > > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> > > >>
> > > >> Alexey Varlamov wrote:
> > > >> > Stefano,
> > > >> >
> > > >> > It is a bit unfair to compare *debug* build of Harmony with other
> > > >> > release versions :)
> > > >>
> > > >> I'm simulating what a journalist with a developer could do.
> > > >>
> > > >> If there is a way to make it compile in 'release mode' (if such a thing
> > > >> exists), I'll be very glad to redo the benchmarks.
> > > >>
> > > >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> > > >> > switches)?
> > > >>
> > > >> Right. No switches. I'm simulating what users do when they get the JVM:
> > > >> they run "java"... and if it's now fast enough they buy a new box.
> > > >>
> > > >> Having command line tuning parameters is mostly useless since most
> > > >> people don't know the internals of a JVM well enough to guess what
> > > >> parameters to tune anyway.
> > > >>
> > > >> So, what people will do once they get an harmony snapshot is "java
> > > >> my.class.Name <http://my.class.name/>" and see the results.
> > > >>
> > > >> I want to simulate that and compare it to the same exact experience they
> > > >> will get with other virtual machines for a variety of common scenarios
> > > >> (number crunching, xml processing, http serving, database load, etc...)
> > > >>
> > > >> I will focus on the server because that's there the apache action (and
> > > >> my personal interest) is.
> > > >>
> > > >> So, like I said, if there are 'compile time' switches that I can use to
> > > >> turn 'release mode' on, please tell me and I'll re-do the tests.
> > > >>
> > > >> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> > > >> >> There are lies, damn lies and benchmarks.... which don't really tell
> > > >> you
> > > >> >> if an implementation of a program is *faster* but at least it tells
> > > >> you
> > > >> >> where you're at.
> > > >> >>
> > > >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> > > >> >> was able to start running some benchmarks.
> > > >> >>
> > > >> >> The machine is the following:
> > > >> >>
> > > >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> > > >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> > > >> >>
> > > >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> > > >> >> bogomips 6410.31 (per CPU)
> > > >> >>
> > > >> >> There is nothing else running on the machine (load is 0.04 at the time
> > > >> >> of testing).
> > > >> >>
> > > >> >> The various virtual machines tested are:
> > > >> >>
> > > >> >> harmony
> > > >> >> -------
> > > >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> > > >> >> Foundation or its licensors, as applicable.
> > > >> >> java version " 1.5.0"
> > > >> >> pre-alpha : not complete or compatible
> > > >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> > > >> >>
> > > >> >> sun5
> > > >> >> ---
> > > >> >> java version "1.5.0_09 "
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> > > >> >>
> > > >> >> sun6
> > > >> >> ----
> > > >> >> java version " 1.6.0-rc"
> > > >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> > > >> >>
> > > >> >> ibm
> > > >> >> ---
> > > >> >> java version " 1.5.0"
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> > > >> >> pxa64dev-20061002a (SR3) )
> > > >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > > >> >> j9vmxa6423-20061001 (JIT enabled)
> > > >> >> J9VM - 20060915_08260_LHdSMr
> > > >> >> JIT - 20060908_1811_r8
> > > >> >> GC - 20060906_AA)
> > > >> >> JCL - 20061002
> > > >> >>
> > > >> >> bea
> > > >> >> ---
> > > >> >> java version "1.5.0_06 "
> > > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > > >> >> BEA JRockit(R) (build
> > > >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> > > >> >>
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> > > >> >>
> > > >> >> command: java jnt.scimark2.commandline
> > > >> >>
> > > >> >> NOTE: bigger number is better
> > > >> >>
> > > >> >> Sun6
> > > >> >> Composite Score: 364.5832265230057
> > > >> >> FFT (1024): 220.8458713892794
> > > >> >> SOR (100x100): 696.1542342357722
> > > >> >> Monte Carlo : 149.37978088875656
> > > >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > > >> >> LU (100x100): 430.1617273683819
> > > >> >>
> > > >> >> BEA
> > > >> >> Composite Score: 359.13480378697835
> > > >> >> FFT (1024): 303.8746880751562
> > > >> >> SOR (100x100): 454.25628897202307
> > > >> >> Monte Carlo : 93.23913192138497
> > > >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> > > >> >> LU (100x100): 413.8627835924175
> > > >> >>
> > > >> >> Sun5
> > > >> >> Composite Score: 332.84987587548574
> > > >> >> FFT (1024): 216.5144595799027
> > > >> >> SOR (100x100): 689.429322146947
> > > >> >> Monte Carlo : 25.791262124978065
> > > >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > > >> >> LU (100x100): 414.99493895566377
> > > >> >>
> > > >> >> IBM
> > > >> >> Composite Score: 259.8249218693683
> > > >> >> FFT (1024): 296.8415012789055
> > > >> >> SOR (100x100): 428.974881649179
> > > >> >> Monte Carlo : 89.15159857584082
> > > >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > > >> >> LU (100x100): 339.8042037225181
> > > >> >>
> > > >> >> Harmony
> > > >> >> Composite Score: 113.65082278962575
> > > >> >> FFT (1024): 203.76641991778123
> > > >> >> SOR (100x100): 224.37761309236748
> > > >> >> Monte Carlo : 9.063866256533116
> > > >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > > >> >> LU (100x100): 65.6410280487242
> > > >> >>
> > > >> >> In this test harmony is clearly lagging behind... at about 30%
> > > >> >> performance of the best JVM, it's a little crappy. Please note how
> > > >> FFT's
> > > >> >> performance is not so bad awhile monte carlo is pretty bad compared to
> > > >> >> BEA or IBM.
> > > >> >>
> > > >> >> Overall, it seems like there is some serious work to do here to catch
> > > >> up.
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test 2: Dhrystones
> > > >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> > > >> )
> > > >> >>
> > > >> >> command: java dhry 100000000
> > > >> >>
> > > >> >> NOTE: bigger is better
> > > >> >>
> > > >> >> NB: I modified the code to accept the count at input from the command
> > > >> >> line!
> > > >> >>
> > > >> >> sun6: 8552856 dhrystones/sec
> > > >> >> sun5: 6605892
> > > >> >> bea: 5678914
> > > >> >> harmony: 669734
> > > >> >> ibm: 501562
> > > >> >>
> > > >> >> The performance here is horrific but what's surprising is that J9 is
> > > >> >> even worse. No idea what's going on but it seems like something is not
> > > >> >> working as it should (in both harmony and J9)
> > > >> >>
> > > >> >>
> > > >> --------------------------------------------------------------------------
> > > >>
> > > >> >>
> > > >> >>
> > > >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> > > >> >>
> > > >> >> command: java Sieve 30
> > > >> >>
> > > >> >> NB: I modified the test to run for a configurable amount of seconds.
> > > >> >>
> > > >> >> sun6 8545 sieves/sec
> > > >> >> sun5 8364
> > > >> >> bea 6174
> > > >> >> harmony 1836
> > > >> >> ibm 225
> > > >> >>
> > > >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> > > >> >> lagging behind.
> > > >> >>
> > > >> >> Stay tuned for more tests.
> > > >> >>
> > > >> >> --
> > > >> >> Stefano.
> > > >> >>
> > > >> >>
> > > >>
> > > >>
> > > >> --
> > > >> Stefano.
> > > >>
> > > >>
> > > >
> > >
> > >
> > > --
> > > Stefano.
> > >
> > >
> >
> > --
> > Egor Pasko
> >
> >
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Alexey Varlamov <al...@gmail.com>.
[snip]
> We can try to tune DRLVM default mode for HelloWorld + Eclipse +
> SciMark + DaCapo, would it be enough for users to be happy with
> Harmony on their first glance? I dunno, actually. IMHO, almost all
> java users taking care of performance know about client and server
> modes and what applications are best for each mode.
Err, SciMark looks quite specific in this list, and I suspect it will
provoke that "short blanket" scenario. Probably we should not include
it for the default mode tuning.
As a side note, any user cares about performance but not all dare to
tune it. This is the main motivation for measuring OOB at the first
place...
>
> > > Currently we have in mind the following list:
> > > - HWA (Hello World Application)
> > > - SciMark
> > > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > > - Anything else?
> > >
> > > What do you think about this? Any additions to the list? Comments?
> > > Questions?
> >
> > The problem I have with this is that I feel that each one of such
> > scenario might require different tuning parameters... and if that is the
> > case, you end up with the 'short blanket' problem: you improve here and
> > you decrease there.
> >
> > An 'adaptive' scenario, on the other hand, would allow us to:
> >
> > 1) avoid trying to find the optimal defaults (since we can't possible
> > test every scenario that will be useful in a way that is consistent with
> > real world usage)
> >
> > 2) avoid the blanket problem, each VM can adapt to the scenario of use
> >
> > 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> > changes and 'retune' itself if the environment changes.
> >
> > Of course, there is a price to pay in such 'fedback variability' systems
> > since they have to find the minima over and over again.
> >
> > So, another solution is to have a JVM "tuning parameters discovery mode"
> > that you can run and you turn such "parameter finding" autoprofiling
> > on... and the JVM dumps the tuning results for you on disk which you can
> > later use to initialize the JVM on your own.
> >
> > Not sure how feasible or complicated to write this is, but wow does this
> > sound on paper?
> >
> >
> > > Thanks,
> > > ---
> > > Sergey Kuksenko
> > > Intel Enterprise Solutions Software Division
> > >
> > >
> > > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> > >>
> > >> Alexey Varlamov wrote:
> > >> > Stefano,
> > >> >
> > >> > It is a bit unfair to compare *debug* build of Harmony with other
> > >> > release versions :)
> > >>
> > >> I'm simulating what a journalist with a developer could do.
> > >>
> > >> If there is a way to make it compile in 'release mode' (if such a thing
> > >> exists), I'll be very glad to redo the benchmarks.
> > >>
> > >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> > >> > switches)?
> > >>
> > >> Right. No switches. I'm simulating what users do when they get the JVM:
> > >> they run "java"... and if it's now fast enough they buy a new box.
> > >>
> > >> Having command line tuning parameters is mostly useless since most
> > >> people don't know the internals of a JVM well enough to guess what
> > >> parameters to tune anyway.
> > >>
> > >> So, what people will do once they get an harmony snapshot is "java
> > >> my.class.Name <http://my.class.name/>" and see the results.
> > >>
> > >> I want to simulate that and compare it to the same exact experience they
> > >> will get with other virtual machines for a variety of common scenarios
> > >> (number crunching, xml processing, http serving, database load, etc...)
> > >>
> > >> I will focus on the server because that's there the apache action (and
> > >> my personal interest) is.
> > >>
> > >> So, like I said, if there are 'compile time' switches that I can use to
> > >> turn 'release mode' on, please tell me and I'll re-do the tests.
> > >>
> > >> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> > >> >> There are lies, damn lies and benchmarks.... which don't really tell
> > >> you
> > >> >> if an implementation of a program is *faster* but at least it tells
> > >> you
> > >> >> where you're at.
> > >> >>
> > >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> > >> >> was able to start running some benchmarks.
> > >> >>
> > >> >> The machine is the following:
> > >> >>
> > >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> > >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> > >> >>
> > >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> > >> >> bogomips 6410.31 (per CPU)
> > >> >>
> > >> >> There is nothing else running on the machine (load is 0.04 at the time
> > >> >> of testing).
> > >> >>
> > >> >> The various virtual machines tested are:
> > >> >>
> > >> >> harmony
> > >> >> -------
> > >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> > >> >> Foundation or its licensors, as applicable.
> > >> >> java version " 1.5.0"
> > >> >> pre-alpha : not complete or compatible
> > >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> > >> >>
> > >> >> sun5
> > >> >> ---
> > >> >> java version "1.5.0_09 "
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> > >> >>
> > >> >> sun6
> > >> >> ----
> > >> >> java version " 1.6.0-rc"
> > >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> > >> >>
> > >> >> ibm
> > >> >> ---
> > >> >> java version " 1.5.0"
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> > >> >> pxa64dev-20061002a (SR3) )
> > >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > >> >> j9vmxa6423-20061001 (JIT enabled)
> > >> >> J9VM - 20060915_08260_LHdSMr
> > >> >> JIT - 20060908_1811_r8
> > >> >> GC - 20060906_AA)
> > >> >> JCL - 20061002
> > >> >>
> > >> >> bea
> > >> >> ---
> > >> >> java version "1.5.0_06 "
> > >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > >> >> BEA JRockit(R) (build
> > >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> > >> >>
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> > >> >>
> > >> >> command: java jnt.scimark2.commandline
> > >> >>
> > >> >> NOTE: bigger number is better
> > >> >>
> > >> >> Sun6
> > >> >> Composite Score: 364.5832265230057
> > >> >> FFT (1024): 220.8458713892794
> > >> >> SOR (100x100): 696.1542342357722
> > >> >> Monte Carlo : 149.37978088875656
> > >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > >> >> LU (100x100): 430.1617273683819
> > >> >>
> > >> >> BEA
> > >> >> Composite Score: 359.13480378697835
> > >> >> FFT (1024): 303.8746880751562
> > >> >> SOR (100x100): 454.25628897202307
> > >> >> Monte Carlo : 93.23913192138497
> > >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> > >> >> LU (100x100): 413.8627835924175
> > >> >>
> > >> >> Sun5
> > >> >> Composite Score: 332.84987587548574
> > >> >> FFT (1024): 216.5144595799027
> > >> >> SOR (100x100): 689.429322146947
> > >> >> Monte Carlo : 25.791262124978065
> > >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > >> >> LU (100x100): 414.99493895566377
> > >> >>
> > >> >> IBM
> > >> >> Composite Score: 259.8249218693683
> > >> >> FFT (1024): 296.8415012789055
> > >> >> SOR (100x100): 428.974881649179
> > >> >> Monte Carlo : 89.15159857584082
> > >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > >> >> LU (100x100): 339.8042037225181
> > >> >>
> > >> >> Harmony
> > >> >> Composite Score: 113.65082278962575
> > >> >> FFT (1024): 203.76641991778123
> > >> >> SOR (100x100): 224.37761309236748
> > >> >> Monte Carlo : 9.063866256533116
> > >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > >> >> LU (100x100): 65.6410280487242
> > >> >>
> > >> >> In this test harmony is clearly lagging behind... at about 30%
> > >> >> performance of the best JVM, it's a little crappy. Please note how
> > >> FFT's
> > >> >> performance is not so bad awhile monte carlo is pretty bad compared to
> > >> >> BEA or IBM.
> > >> >>
> > >> >> Overall, it seems like there is some serious work to do here to catch
> > >> up.
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test 2: Dhrystones
> > >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> > >> )
> > >> >>
> > >> >> command: java dhry 100000000
> > >> >>
> > >> >> NOTE: bigger is better
> > >> >>
> > >> >> NB: I modified the code to accept the count at input from the command
> > >> >> line!
> > >> >>
> > >> >> sun6: 8552856 dhrystones/sec
> > >> >> sun5: 6605892
> > >> >> bea: 5678914
> > >> >> harmony: 669734
> > >> >> ibm: 501562
> > >> >>
> > >> >> The performance here is horrific but what's surprising is that J9 is
> > >> >> even worse. No idea what's going on but it seems like something is not
> > >> >> working as it should (in both harmony and J9)
> > >> >>
> > >> >>
> > >> --------------------------------------------------------------------------
> > >>
> > >> >>
> > >> >>
> > >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> > >> >>
> > >> >> command: java Sieve 30
> > >> >>
> > >> >> NB: I modified the test to run for a configurable amount of seconds.
> > >> >>
> > >> >> sun6 8545 sieves/sec
> > >> >> sun5 8364
> > >> >> bea 6174
> > >> >> harmony 1836
> > >> >> ibm 225
> > >> >>
> > >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> > >> >> lagging behind.
> > >> >>
> > >> >> Stay tuned for more tests.
> > >> >>
> > >> >> --
> > >> >> Stefano.
> > >> >>
> > >> >>
> > >>
> > >>
> > >> --
> > >> Stefano.
> > >>
> > >>
> > >
> >
> >
> > --
> > Stefano.
> >
> >
>
> --
> Egor Pasko
>
>
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x228 day of Apache Harmony Stefano Mazzocchi wrote:
> > Our client (default) compilation path was tuned a long time ago and it
> > probably makes sense to have another round. What we initially did was
> > running some script executing the given set of workloads trying to find the
> > best configuration for our VM. Having said that I suggest we choose the
> > right set of applications/benchmarks, so we can start our tuning once
> > again.
>
> Maybe it's the analog microelectonic guy in me talking, but every time I
> hear something like "let's get reasonable defaults", I think of
> introducing a variation and a feedback to reach a local minimum and
> stabilize the system.
>
> I know very little about how DRLVM works, but would it be feasible to
> start with such "reasonable defaults" and introduce a random variability
> to the way the JIT works alongside a very simple method profiler and see
> if the performance increase? think of you trying out different things
> and see if they work better... but done by the JVM as it runs.
>
> Keep in mind I'm a total newbie in virtual machine design (or CPU
> architectures for that matter, despite my degree in microelectronics..
> well, to be fair, I was doing analog not digital circuits) so bear with
> me if I'm saying stupid things :-)
not stupid)) Searching optimal parameters automatically is a good
idea. First of all, we need to find out, what the function is that we
want to minimize. That is, what benchmarks we should rely on when
composing the set of parameters for best-of-all default mode. Then a
genetic algorithm may suit for searching optimal parameters.
AFAIR, DRLVM default mode was tuned to
* HelloWorld program
* Eclipse startup
* some long running benchmarks
In general, there is a simple controversy: fast startup and fast
execution. We want both, everywhere. That is why we have
heuristic-based recompilation. But, sadly, there is no best-for-all
set of parameters yet, and won't be in the nearest future.
We can try to tune DRLVM default mode for HelloWorld + Eclipse +
SciMark + DaCapo, would it be enough for users to be happy with
Harmony on their first glance? I dunno, actually. IMHO, almost all
java users taking care of performance know about client and server
modes and what applications are best for each mode.
> > Currently we have in mind the following list:
> > - HWA (Hello World Application)
> > - SciMark
> > - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> > - Anything else?
> >
> > What do you think about this? Any additions to the list? Comments?
> > Questions?
>
> The problem I have with this is that I feel that each one of such
> scenario might require different tuning parameters... and if that is the
> case, you end up with the 'short blanket' problem: you improve here and
> you decrease there.
>
> An 'adaptive' scenario, on the other hand, would allow us to:
>
> 1) avoid trying to find the optimal defaults (since we can't possible
> test every scenario that will be useful in a way that is consistent with
> real world usage)
>
> 2) avoid the blanket problem, each VM can adapt to the scenario of use
>
> 3) avoid the 'stiffness' problem, each VM can adapt to machine resource
> changes and 'retune' itself if the environment changes.
>
> Of course, there is a price to pay in such 'fedback variability' systems
> since they have to find the minima over and over again.
>
> So, another solution is to have a JVM "tuning parameters discovery mode"
> that you can run and you turn such "parameter finding" autoprofiling
> on... and the JVM dumps the tuning results for you on disk which you can
> later use to initialize the JVM on your own.
>
> Not sure how feasible or complicated to write this is, but wow does this
> sound on paper?
>
>
> > Thanks,
> > ---
> > Sergey Kuksenko
> > Intel Enterprise Solutions Software Division
> >
> >
> > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> >>
> >> Alexey Varlamov wrote:
> >> > Stefano,
> >> >
> >> > It is a bit unfair to compare *debug* build of Harmony with other
> >> > release versions :)
> >>
> >> I'm simulating what a journalist with a developer could do.
> >>
> >> If there is a way to make it compile in 'release mode' (if such a thing
> >> exists), I'll be very glad to redo the benchmarks.
> >>
> >> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> >> > switches)?
> >>
> >> Right. No switches. I'm simulating what users do when they get the JVM:
> >> they run "java"... and if it's now fast enough they buy a new box.
> >>
> >> Having command line tuning parameters is mostly useless since most
> >> people don't know the internals of a JVM well enough to guess what
> >> parameters to tune anyway.
> >>
> >> So, what people will do once they get an harmony snapshot is "java
> >> my.class.Name <http://my.class.name/>" and see the results.
> >>
> >> I want to simulate that and compare it to the same exact experience they
> >> will get with other virtual machines for a variety of common scenarios
> >> (number crunching, xml processing, http serving, database load, etc...)
> >>
> >> I will focus on the server because that's there the apache action (and
> >> my personal interest) is.
> >>
> >> So, like I said, if there are 'compile time' switches that I can use to
> >> turn 'release mode' on, please tell me and I'll re-do the tests.
> >>
> >> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> >> >> There are lies, damn lies and benchmarks.... which don't really tell
> >> you
> >> >> if an implementation of a program is *faster* but at least it tells
> >> you
> >> >> where you're at.
> >> >>
> >> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> >> >> was able to start running some benchmarks.
> >> >>
> >> >> The machine is the following:
> >> >>
> >> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> >> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >> >>
> >> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> >> bogomips 6410.31 (per CPU)
> >> >>
> >> >> There is nothing else running on the machine (load is 0.04 at the time
> >> >> of testing).
> >> >>
> >> >> The various virtual machines tested are:
> >> >>
> >> >> harmony
> >> >> -------
> >> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> >> Foundation or its licensors, as applicable.
> >> >> java version " 1.5.0"
> >> >> pre-alpha : not complete or compatible
> >> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >> >>
> >> >> sun5
> >> >> ---
> >> >> java version "1.5.0_09 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >> >>
> >> >> sun6
> >> >> ----
> >> >> java version " 1.6.0-rc"
> >> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >> >>
> >> >> ibm
> >> >> ---
> >> >> java version " 1.5.0"
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> >> pxa64dev-20061002a (SR3) )
> >> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> >> j9vmxa6423-20061001 (JIT enabled)
> >> >> J9VM - 20060915_08260_LHdSMr
> >> >> JIT - 20060908_1811_r8
> >> >> GC - 20060906_AA)
> >> >> JCL - 20061002
> >> >>
> >> >> bea
> >> >> ---
> >> >> java version "1.5.0_06 "
> >> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> >> >> BEA JRockit(R) (build
> >> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >> >>
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >> >>
> >> >> command: java jnt.scimark2.commandline
> >> >>
> >> >> NOTE: bigger number is better
> >> >>
> >> >> Sun6
> >> >> Composite Score: 364.5832265230057
> >> >> FFT (1024): 220.8458713892794
> >> >> SOR (100x100): 696.1542342357722
> >> >> Monte Carlo : 149.37978088875656
> >> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> >> LU (100x100): 430.1617273683819
> >> >>
> >> >> BEA
> >> >> Composite Score: 359.13480378697835
> >> >> FFT (1024): 303.8746880751562
> >> >> SOR (100x100): 454.25628897202307
> >> >> Monte Carlo : 93.23913192138497
> >> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> >> LU (100x100): 413.8627835924175
> >> >>
> >> >> Sun5
> >> >> Composite Score: 332.84987587548574
> >> >> FFT (1024): 216.5144595799027
> >> >> SOR (100x100): 689.429322146947
> >> >> Monte Carlo : 25.791262124978065
> >> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> >> LU (100x100): 414.99493895566377
> >> >>
> >> >> IBM
> >> >> Composite Score: 259.8249218693683
> >> >> FFT (1024): 296.8415012789055
> >> >> SOR (100x100): 428.974881649179
> >> >> Monte Carlo : 89.15159857584082
> >> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> >> LU (100x100): 339.8042037225181
> >> >>
> >> >> Harmony
> >> >> Composite Score: 113.65082278962575
> >> >> FFT (1024): 203.76641991778123
> >> >> SOR (100x100): 224.37761309236748
> >> >> Monte Carlo : 9.063866256533116
> >> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> >> LU (100x100): 65.6410280487242
> >> >>
> >> >> In this test harmony is clearly lagging behind... at about 30%
> >> >> performance of the best JVM, it's a little crappy. Please note how
> >> FFT's
> >> >> performance is not so bad awhile monte carlo is pretty bad compared to
> >> >> BEA or IBM.
> >> >>
> >> >> Overall, it seems like there is some serious work to do here to catch
> >> up.
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 2: Dhrystones
> >> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> >> )
> >> >>
> >> >> command: java dhry 100000000
> >> >>
> >> >> NOTE: bigger is better
> >> >>
> >> >> NB: I modified the code to accept the count at input from the command
> >> >> line!
> >> >>
> >> >> sun6: 8552856 dhrystones/sec
> >> >> sun5: 6605892
> >> >> bea: 5678914
> >> >> harmony: 669734
> >> >> ibm: 501562
> >> >>
> >> >> The performance here is horrific but what's surprising is that J9 is
> >> >> even worse. No idea what's going on but it seems like something is not
> >> >> working as it should (in both harmony and J9)
> >> >>
> >> >>
> >> --------------------------------------------------------------------------
> >>
> >> >>
> >> >>
> >> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >> >>
> >> >> command: java Sieve 30
> >> >>
> >> >> NB: I modified the test to run for a configurable amount of seconds.
> >> >>
> >> >> sun6 8545 sieves/sec
> >> >> sun5 8364
> >> >> bea 6174
> >> >> harmony 1836
> >> >> ibm 225
> >> >>
> >> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> >> lagging behind.
> >> >>
> >> >> Stay tuned for more tests.
> >> >>
> >> >> --
> >> >> Stefano.
> >> >>
> >> >>
> >>
> >>
> >> --
> >> Stefano.
> >>
> >>
> >
>
>
> --
> Stefano.
>
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Sergey Kuksenko wrote:
> Stefano,
> Trying to get the potential of Harmony I've quickly checked SciMak on tuned
> Harmony release build and compared it with BEA & SUN.
Sergey,
many thanks for doing this.
> Hardware: P4 Xeon 3GHz
> Windows XP SP2 (It's another platform, but I hope the key things are still
> the same).
>
> BEA -
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> BEA JRockit(R) (build R26.3.0-32-58710-1.5.0_06-20060308-2022-win-ia32, )
>
> SUN -
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)
>
>
> Harmony -
> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> Foundation or its licensors, as applicable.
> java version "1.5.0"
> pre-alpha : not complete or compatible
> svn = 475925, (Nov 17 2006), Windows/ia32/msvc 1310, release build
>
> I've got the following results
>
> BEA (out of the box):
>
> Composite Score: 435.9674695335291
> FFT (1024): 295.33366058958575
> SOR (100x100): 474.15229982839213
> Monte Carlo : 111.56918839504195
> Sparse matmult (N=1000, nz=5000): 551.8821052631578
> LU (100x100): 746.9000935914679
> ----
>
> Sun (out of the box):
>
> Composite Score: 229.70779446543412
> FFT (1024): 104.92303791891565
> SOR (100x100): 400.44785722405015
> Monte Carlo : 13.257380552894444
> Sparse matmult (N=1000, nz=5000): 160.07814989061512
> LU (100x100): 469.8325467406951
>
> ---
>
> Harmony (out of the box):
>
> Composite Score: 109.43208528481887
> FFT (1024): 51.30119529411764
> SOR (100x100): 257.9591618631154
> Monte Carlo : 17.04568642272773
> Sparse matmult (N=1000, nz=5000): 129.4666069618598
> LU (100x100): 91.38777588227376
> ----
>
> Harmony (tuned options, server path):
>
> Composite Score: 181.54555681031619
> FFT (1024): 91.22597999162443
> SOR (100x100): 329.8450882375011
> Monte Carlo : 42.51432538579417
> Sparse matmult (N=1000, nz=5000): 260.58050602943024
> LU (100x100): 183.56188440723088
that's pretty good.
> ------
>
> When I looked into Harmony OOB I've found that all hot methods of SciMark
> are compiled by JET (not recompiled by the optimizing JIT compiler). The
> way
> our DRLVM currrently recognises hot path is not sutable for SciMark becasue
> of short run.
Hmmm, what do you mean by "short run"? the entire app runs for a short
amount of time total or each hot method runs for a short amount of time
not enough to have it recognized as "hot"?
> We need to tune DRLVM options to get better results.
> Tuned options give good SciMark score improvement (109->181).
Well, to be fair, all the other JVM could probably do the same.
> Which moves Harmony performance close to what Sun OOB shows.
excuse my ignorance, but what's OOB? (google define says "out of
business" or "order of battle"... not sure they apply here ;-)
> Our client (default) compilation path was tuned a long time ago and it
> probably makes sense to have another round. What we initially did was
> running some script executing the given set of workloads trying to find the
> best configuration for our VM. Having said that I suggest we choose the
> right set of applications/benchmarks, so we can start our tuning once
> again.
Maybe it's the analog microelectonic guy in me talking, but every time I
hear something like "let's get reasonable defaults", I think of
introducing a variation and a feedback to reach a local minimum and
stabilize the system.
I know very little about how DRLVM works, but would it be feasible to
start with such "reasonable defaults" and introduce a random variability
to the way the JIT works alongside a very simple method profiler and see
if the performance increase? think of you trying out different things
and see if they work better... but done by the JVM as it runs.
Keep in mind I'm a total newbie in virtual machine design (or CPU
architectures for that matter, despite my degree in microelectronics..
well, to be fair, I was doing analog not digital circuits) so bear with
me if I'm saying stupid things :-)
> Currently we have in mind the following list:
> - HWA (Hello World Application)
> - SciMark
> - Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
> - Anything else?
>
> What do you think about this? Any additions to the list? Comments?
> Questions?
The problem I have with this is that I feel that each one of such
scenario might require different tuning parameters... and if that is the
case, you end up with the 'short blanket' problem: you improve here and
you decrease there.
An 'adaptive' scenario, on the other hand, would allow us to:
1) avoid trying to find the optimal defaults (since we can't possible
test every scenario that will be useful in a way that is consistent with
real world usage)
2) avoid the blanket problem, each VM can adapt to the scenario of use
3) avoid the 'stiffness' problem, each VM can adapt to machine resource
changes and 'retune' itself if the environment changes.
Of course, there is a price to pay in such 'fedback variability' systems
since they have to find the minima over and over again.
So, another solution is to have a JVM "tuning parameters discovery mode"
that you can run and you turn such "parameter finding" autoprofiling
on... and the JVM dumps the tuning results for you on disk which you can
later use to initialize the JVM on your own.
Not sure how feasible or complicated to write this is, but wow does this
sound on paper?
> Thanks,
> ---
> Sergey Kuksenko
> Intel Enterprise Solutions Software Division
>
>
> On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>>
>> Alexey Varlamov wrote:
>> > Stefano,
>> >
>> > It is a bit unfair to compare *debug* build of Harmony with other
>> > release versions :)
>>
>> I'm simulating what a journalist with a developer could do.
>>
>> If there is a way to make it compile in 'release mode' (if such a thing
>> exists), I'll be very glad to redo the benchmarks.
>>
>> > I suppose all VMs where run in default mode (i.e. no special cmd-line
>> > switches)?
>>
>> Right. No switches. I'm simulating what users do when they get the JVM:
>> they run "java"... and if it's now fast enough they buy a new box.
>>
>> Having command line tuning parameters is mostly useless since most
>> people don't know the internals of a JVM well enough to guess what
>> parameters to tune anyway.
>>
>> So, what people will do once they get an harmony snapshot is "java
>> my.class.Name <http://my.class.name/>" and see the results.
>>
>> I want to simulate that and compare it to the same exact experience they
>> will get with other virtual machines for a variety of common scenarios
>> (number crunching, xml processing, http serving, database load, etc...)
>>
>> I will focus on the server because that's there the apache action (and
>> my personal interest) is.
>>
>> So, like I said, if there are 'compile time' switches that I can use to
>> turn 'release mode' on, please tell me and I'll re-do the tests.
>>
>> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
>> >> There are lies, damn lies and benchmarks.... which don't really tell
>> you
>> >> if an implementation of a program is *faster* but at least it tells
>> you
>> >> where you're at.
>> >>
>> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>> >> was able to start running some benchmarks.
>> >>
>> >> The machine is the following:
>> >>
>> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
>> >>
>> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
>> >> bogomips 6410.31 (per CPU)
>> >>
>> >> There is nothing else running on the machine (load is 0.04 at the time
>> >> of testing).
>> >>
>> >> The various virtual machines tested are:
>> >>
>> >> harmony
>> >> -------
>> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> >> Foundation or its licensors, as applicable.
>> >> java version " 1.5.0"
>> >> pre-alpha : not complete or compatible
>> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>> >>
>> >> sun5
>> >> ---
>> >> java version "1.5.0_09 "
>> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>> >>
>> >> sun6
>> >> ----
>> >> java version " 1.6.0-rc"
>> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>> >>
>> >> ibm
>> >> ---
>> >> java version " 1.5.0"
>> >> Java(TM) 2 Runtime Environment, Standard Edition (build
>> >> pxa64dev-20061002a (SR3) )
>> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>> >> j9vmxa6423-20061001 (JIT enabled)
>> >> J9VM - 20060915_08260_LHdSMr
>> >> JIT - 20060908_1811_r8
>> >> GC - 20060906_AA)
>> >> JCL - 20061002
>> >>
>> >> bea
>> >> ---
>> >> java version "1.5.0_06 "
>> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> >> BEA JRockit(R) (build
>> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>> >>
>> >>
>> >>
>> --------------------------------------------------------------------------
>>
>> >>
>> >>
>> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>> >>
>> >> command: java jnt.scimark2.commandline
>> >>
>> >> NOTE: bigger number is better
>> >>
>> >> Sun6
>> >> Composite Score: 364.5832265230057
>> >> FFT (1024): 220.8458713892794
>> >> SOR (100x100): 696.1542342357722
>> >> Monte Carlo : 149.37978088875656
>> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
>> >> LU (100x100): 430.1617273683819
>> >>
>> >> BEA
>> >> Composite Score: 359.13480378697835
>> >> FFT (1024): 303.8746880751562
>> >> SOR (100x100): 454.25628897202307
>> >> Monte Carlo : 93.23913192138497
>> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
>> >> LU (100x100): 413.8627835924175
>> >>
>> >> Sun5
>> >> Composite Score: 332.84987587548574
>> >> FFT (1024): 216.5144595799027
>> >> SOR (100x100): 689.429322146947
>> >> Monte Carlo : 25.791262124978065
>> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
>> >> LU (100x100): 414.99493895566377
>> >>
>> >> IBM
>> >> Composite Score: 259.8249218693683
>> >> FFT (1024): 296.8415012789055
>> >> SOR (100x100): 428.974881649179
>> >> Monte Carlo : 89.15159857584082
>> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
>> >> LU (100x100): 339.8042037225181
>> >>
>> >> Harmony
>> >> Composite Score: 113.65082278962575
>> >> FFT (1024): 203.76641991778123
>> >> SOR (100x100): 224.37761309236748
>> >> Monte Carlo : 9.063866256533116
>> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
>> >> LU (100x100): 65.6410280487242
>> >>
>> >> In this test harmony is clearly lagging behind... at about 30%
>> >> performance of the best JVM, it's a little crappy. Please note how
>> FFT's
>> >> performance is not so bad awhile monte carlo is pretty bad compared to
>> >> BEA or IBM.
>> >>
>> >> Overall, it seems like there is some serious work to do here to catch
>> up.
>> >>
>> >>
>> --------------------------------------------------------------------------
>>
>> >>
>> >>
>> >> Test 2: Dhrystones
>> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
>> )
>> >>
>> >> command: java dhry 100000000
>> >>
>> >> NOTE: bigger is better
>> >>
>> >> NB: I modified the code to accept the count at input from the command
>> >> line!
>> >>
>> >> sun6: 8552856 dhrystones/sec
>> >> sun5: 6605892
>> >> bea: 5678914
>> >> harmony: 669734
>> >> ibm: 501562
>> >>
>> >> The performance here is horrific but what's surprising is that J9 is
>> >> even worse. No idea what's going on but it seems like something is not
>> >> working as it should (in both harmony and J9)
>> >>
>> >>
>> --------------------------------------------------------------------------
>>
>> >>
>> >>
>> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>> >>
>> >> command: java Sieve 30
>> >>
>> >> NB: I modified the test to run for a configurable amount of seconds.
>> >>
>> >> sun6 8545 sieves/sec
>> >> sun5 8364
>> >> bea 6174
>> >> harmony 1836
>> >> ibm 225
>> >>
>> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>> >> lagging behind.
>> >>
>> >> Stay tuned for more tests.
>> >>
>> >> --
>> >> Stefano.
>> >>
>> >>
>>
>>
>> --
>> Stefano.
>>
>>
>
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Sergey Kuksenko <se...@gmail.com>.
Stefano,
Trying to get the potential of Harmony I've quickly checked SciMak on tuned
Harmony release build and compared it with BEA & SUN.
Hardware: P4 Xeon 3GHz
Windows XP SP2 (It's another platform, but I hope the key things are still
the same).
BEA -
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
BEA JRockit(R) (build R26.3.0-32-58710-1.5.0_06-20060308-2022-win-ia32, )
SUN -
java version "1.5.0_06"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
Java HotSpot(TM) Client VM (build 1.5.0_06-b05, mixed mode)
Harmony -
Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
Foundation or its licensors, as applicable.
java version "1.5.0"
pre-alpha : not complete or compatible
svn = 475925, (Nov 17 2006), Windows/ia32/msvc 1310, release build
I've got the following results
BEA (out of the box):
Composite Score: 435.9674695335291
FFT (1024): 295.33366058958575
SOR (100x100): 474.15229982839213
Monte Carlo : 111.56918839504195
Sparse matmult (N=1000, nz=5000): 551.8821052631578
LU (100x100): 746.9000935914679
----
Sun (out of the box):
Composite Score: 229.70779446543412
FFT (1024): 104.92303791891565
SOR (100x100): 400.44785722405015
Monte Carlo : 13.257380552894444
Sparse matmult (N=1000, nz=5000): 160.07814989061512
LU (100x100): 469.8325467406951
---
Harmony (out of the box):
Composite Score: 109.43208528481887
FFT (1024): 51.30119529411764
SOR (100x100): 257.9591618631154
Monte Carlo : 17.04568642272773
Sparse matmult (N=1000, nz=5000): 129.4666069618598
LU (100x100): 91.38777588227376
----
Harmony (tuned options, server path):
Composite Score: 181.54555681031619
FFT (1024): 91.22597999162443
SOR (100x100): 329.8450882375011
Monte Carlo : 42.51432538579417
Sparse matmult (N=1000, nz=5000): 260.58050602943024
LU (100x100): 183.56188440723088
------
When I looked into Harmony OOB I've found that all hot methods of SciMark
are compiled by JET (not recompiled by the optimizing JIT compiler). The way
our DRLVM currrently recognises hot path is not sutable for SciMark becasue
of short run. We need to tune DRLVM options to get better results.
Tuned options give good SciMark score improvement (109->181).
Which moves Harmony performance close to what Sun OOB shows.
Our client (default) compilation path was tuned a long time ago and it
probably makes sense to have another round. What we initially did was
running some script executing the given set of workloads trying to find the
best configuration for our VM. Having said that I suggest we choose the
right set of applications/benchmarks, so we can start our tuning once again.
Currently we have in mind the following list:
- HWA (Hello World Application)
- SciMark
- Dacapo (reasonable set of benchmarks, like fop, hsqldb, chart and xalan)
- Anything else?
What do you think about this? Any additions to the list? Comments?
Questions?
Thanks,
---
Sergey Kuksenko
Intel Enterprise Solutions Software Division
On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> Alexey Varlamov wrote:
> > Stefano,
> >
> > It is a bit unfair to compare *debug* build of Harmony with other
> > release versions :)
>
> I'm simulating what a journalist with a developer could do.
>
> If there is a way to make it compile in 'release mode' (if such a thing
> exists), I'll be very glad to redo the benchmarks.
>
> > I suppose all VMs where run in default mode (i.e. no special cmd-line
> > switches)?
>
> Right. No switches. I'm simulating what users do when they get the JVM:
> they run "java"... and if it's now fast enough they buy a new box.
>
> Having command line tuning parameters is mostly useless since most
> people don't know the internals of a JVM well enough to guess what
> parameters to tune anyway.
>
> So, what people will do once they get an harmony snapshot is "java
> my.class.Name <http://my.class.name/>" and see the results.
>
> I want to simulate that and compare it to the same exact experience they
> will get with other virtual machines for a variety of common scenarios
> (number crunching, xml processing, http serving, database load, etc...)
>
> I will focus on the server because that's there the apache action (and
> my personal interest) is.
>
> So, like I said, if there are 'compile time' switches that I can use to
> turn 'release mode' on, please tell me and I'll re-do the tests.
>
> > 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> >> There are lies, damn lies and benchmarks.... which don't really tell
> you
> >> if an implementation of a program is *faster* but at least it tells you
> >> where you're at.
> >>
> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> >> was able to start running some benchmarks.
> >>
> >> The machine is the following:
> >>
> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >>
> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> bogomips 6410.31 (per CPU)
> >>
> >> There is nothing else running on the machine (load is 0.04 at the time
> >> of testing).
> >>
> >> The various virtual machines tested are:
> >>
> >> harmony
> >> -------
> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> Foundation or its licensors, as applicable.
> >> java version " 1.5.0"
> >> pre-alpha : not complete or compatible
> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >>
> >> sun5
> >> ---
> >> java version "1.5.0_09 "
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >>
> >> sun6
> >> ----
> >> java version " 1.6.0-rc"
> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >>
> >> ibm
> >> ---
> >> java version " 1.5.0"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> pxa64dev-20061002a (SR3) )
> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> j9vmxa6423-20061001 (JIT enabled)
> >> J9VM - 20060915_08260_LHdSMr
> >> JIT - 20060908_1811_r8
> >> GC - 20060906_AA)
> >> JCL - 20061002
> >>
> >> bea
> >> ---
> >> java version "1.5.0_06 "
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> >> BEA JRockit(R) (build
> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >>
> >>
> >>
> --------------------------------------------------------------------------
> >>
> >>
> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >>
> >> command: java jnt.scimark2.commandline
> >>
> >> NOTE: bigger number is better
> >>
> >> Sun6
> >> Composite Score: 364.5832265230057
> >> FFT (1024): 220.8458713892794
> >> SOR (100x100): 696.1542342357722
> >> Monte Carlo : 149.37978088875656
> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> LU (100x100): 430.1617273683819
> >>
> >> BEA
> >> Composite Score: 359.13480378697835
> >> FFT (1024): 303.8746880751562
> >> SOR (100x100): 454.25628897202307
> >> Monte Carlo : 93.23913192138497
> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> LU (100x100): 413.8627835924175
> >>
> >> Sun5
> >> Composite Score: 332.84987587548574
> >> FFT (1024): 216.5144595799027
> >> SOR (100x100): 689.429322146947
> >> Monte Carlo : 25.791262124978065
> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> LU (100x100): 414.99493895566377
> >>
> >> IBM
> >> Composite Score: 259.8249218693683
> >> FFT (1024): 296.8415012789055
> >> SOR (100x100): 428.974881649179
> >> Monte Carlo : 89.15159857584082
> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> LU (100x100): 339.8042037225181
> >>
> >> Harmony
> >> Composite Score: 113.65082278962575
> >> FFT (1024): 203.76641991778123
> >> SOR (100x100): 224.37761309236748
> >> Monte Carlo : 9.063866256533116
> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> LU (100x100): 65.6410280487242
> >>
> >> In this test harmony is clearly lagging behind... at about 30%
> >> performance of the best JVM, it's a little crappy. Please note how
> FFT's
> >> performance is not so bad awhile monte carlo is pretty bad compared to
> >> BEA or IBM.
> >>
> >> Overall, it seems like there is some serious work to do here to catch
> up.
> >>
> >>
> --------------------------------------------------------------------------
> >>
> >>
> >> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/
> )
> >>
> >> command: java dhry 100000000
> >>
> >> NOTE: bigger is better
> >>
> >> NB: I modified the code to accept the count at input from the command
> >> line!
> >>
> >> sun6: 8552856 dhrystones/sec
> >> sun5: 6605892
> >> bea: 5678914
> >> harmony: 669734
> >> ibm: 501562
> >>
> >> The performance here is horrific but what's surprising is that J9 is
> >> even worse. No idea what's going on but it seems like something is not
> >> working as it should (in both harmony and J9)
> >>
> >>
> --------------------------------------------------------------------------
> >>
> >>
> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >>
> >> command: java Sieve 30
> >>
> >> NB: I modified the test to run for a configurable amount of seconds.
> >>
> >> sun6 8545 sieves/sec
> >> sun5 8364
> >> bea 6174
> >> harmony 1836
> >> ibm 225
> >>
> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> lagging behind.
> >>
> >> Stay tuned for more tests.
> >>
> >> --
> >> Stefano.
> >>
> >>
>
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Stefano Mazzocchi wrote:
> Alexey Varlamov wrote:
>> Stefano,
>>
>> It is a bit unfair to compare *debug* build of Harmony with other
>> release versions :)
>
> I'm simulating what a journalist with a developer could do.
Our snapshots are built in release mode.
>
> If there is a way to make it compile in 'release mode' (if such a thing
> exists), I'll be very glad to redo the benchmarks.
export BUILD_CFG=release
sh build.sh clean
sh build.sh
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Alexey Varlamov wrote:
> Stefano,
>
> It is a bit unfair to compare *debug* build of Harmony with other
> release versions :)
I'm simulating what a journalist with a developer could do.
If there is a way to make it compile in 'release mode' (if such a thing
exists), I'll be very glad to redo the benchmarks.
> I suppose all VMs where run in default mode (i.e. no special cmd-line
> switches)?
Right. No switches. I'm simulating what users do when they get the JVM:
they run "java"... and if it's now fast enough they buy a new box.
Having command line tuning parameters is mostly useless since most
people don't know the internals of a JVM well enough to guess what
parameters to tune anyway.
So, what people will do once they get an harmony snapshot is "java
my.class.Name" and see the results.
I want to simulate that and compare it to the same exact experience they
will get with other virtual machines for a variety of common scenarios
(number crunching, xml processing, http serving, database load, etc...)
I will focus on the server because that's there the apache action (and
my personal interest) is.
So, like I said, if there are 'compile time' switches that I can use to
turn 'release mode' on, please tell me and I'll re-do the tests.
> 2006/11/17, Stefano Mazzocchi <st...@apache.org>:
>> There are lies, damn lies and benchmarks.... which don't really tell you
>> if an implementation of a program is *faster* but at least it tells you
>> where you're at.
>>
>> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>> was able to start running some benchmarks.
>>
>> The machine is the following:
>>
>> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>> 01:50:50 UTC 2006 x86_64 GNU/Linux
>>
>> dual Intel(R) Pentium(R) D CPU 3.20GHz
>> bogomips 6410.31 (per CPU)
>>
>> There is nothing else running on the machine (load is 0.04 at the time
>> of testing).
>>
>> The various virtual machines tested are:
>>
>> harmony
>> -------
>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> Foundation or its licensors, as applicable.
>> java version "1.5.0"
>> pre-alpha : not complete or compatible
>> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>>
>> sun5
>> ---
>> java version "1.5.0_09"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>>
>> sun6
>> ----
>> java version "1.6.0-rc"
>> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>>
>> ibm
>> ---
>> java version "1.5.0"
>> Java(TM) 2 Runtime Environment, Standard Edition (build
>> pxa64dev-20061002a (SR3) )
>> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>> j9vmxa6423-20061001 (JIT enabled)
>> J9VM - 20060915_08260_LHdSMr
>> JIT - 20060908_1811_r8
>> GC - 20060906_AA)
>> JCL - 20061002
>>
>> bea
>> ---
>> java version "1.5.0_06"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> BEA JRockit(R) (build
>> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>>
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>>
>> command: java jnt.scimark2.commandline
>>
>> NOTE: bigger number is better
>>
>> Sun6
>> Composite Score: 364.5832265230057
>> FFT (1024): 220.8458713892794
>> SOR (100x100): 696.1542342357722
>> Monte Carlo : 149.37978088875656
>> Sparse matmult (N=1000, nz=5000): 326.37451873283845
>> LU (100x100): 430.1617273683819
>>
>> BEA
>> Composite Score: 359.13480378697835
>> FFT (1024): 303.8746880751562
>> SOR (100x100): 454.25628897202307
>> Monte Carlo : 93.23913192138497
>> Sparse matmult (N=1000, nz=5000): 530.44112637391
>> LU (100x100): 413.8627835924175
>>
>> Sun5
>> Composite Score: 332.84987587548574
>> FFT (1024): 216.5144595799027
>> SOR (100x100): 689.429322146947
>> Monte Carlo : 25.791262124978065
>> Sparse matmult (N=1000, nz=5000): 317.5193965699373
>> LU (100x100): 414.99493895566377
>>
>> IBM
>> Composite Score: 259.8249218693683
>> FFT (1024): 296.8415012789055
>> SOR (100x100): 428.974881649179
>> Monte Carlo : 89.15159857584082
>> Sparse matmult (N=1000, nz=5000): 144.3524241203982
>> LU (100x100): 339.8042037225181
>>
>> Harmony
>> Composite Score: 113.65082278962575
>> FFT (1024): 203.76641991778123
>> SOR (100x100): 224.37761309236748
>> Monte Carlo : 9.063866256533116
>> Sparse matmult (N=1000, nz=5000): 65.4051866327227
>> LU (100x100): 65.6410280487242
>>
>> In this test harmony is clearly lagging behind... at about 30%
>> performance of the best JVM, it's a little crappy. Please note how FFT's
>> performance is not so bad awhile monte carlo is pretty bad compared to
>> BEA or IBM.
>>
>> Overall, it seems like there is some serious work to do here to catch up.
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>>
>> command: java dhry 100000000
>>
>> NOTE: bigger is better
>>
>> NB: I modified the code to accept the count at input from the command
>> line!
>>
>> sun6: 8552856 dhrystones/sec
>> sun5: 6605892
>> bea: 5678914
>> harmony: 669734
>> ibm: 501562
>>
>> The performance here is horrific but what's surprising is that J9 is
>> even worse. No idea what's going on but it seems like something is not
>> working as it should (in both harmony and J9)
>>
>> --------------------------------------------------------------------------
>>
>>
>> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>>
>> command: java Sieve 30
>>
>> NB: I modified the test to run for a configurable amount of seconds.
>>
>> sun6 8545 sieves/sec
>> sun5 8364
>> bea 6174
>> harmony 1836
>> ibm 225
>>
>> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>> lagging behind.
>>
>> Stay tuned for more tests.
>>
>> --
>> Stefano.
>>
>>
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Alexey Varlamov <al...@gmail.com>.
Stefano,
It is a bit unfair to compare *debug* build of Harmony with other
release versions :)
I suppose all VMs where run in default mode (i.e. no special cmd-line switches)?
2006/11/17, Stefano Mazzocchi <st...@apache.org>:
> There are lies, damn lies and benchmarks.... which don't really tell you
> if an implementation of a program is *faster* but at least it tells you
> where you're at.
>
> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> was able to start running some benchmarks.
>
> The machine is the following:
>
> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> 01:50:50 UTC 2006 x86_64 GNU/Linux
>
> dual Intel(R) Pentium(R) D CPU 3.20GHz
> bogomips 6410.31 (per CPU)
>
> There is nothing else running on the machine (load is 0.04 at the time
> of testing).
>
> The various virtual machines tested are:
>
> harmony
> -------
> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> Foundation or its licensors, as applicable.
> java version "1.5.0"
> pre-alpha : not complete or compatible
> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>
> sun5
> ---
> java version "1.5.0_09"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>
> sun6
> ----
> java version "1.6.0-rc"
> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>
> ibm
> ---
> java version "1.5.0"
> Java(TM) 2 Runtime Environment, Standard Edition (build
> pxa64dev-20061002a (SR3) )
> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> j9vmxa6423-20061001 (JIT enabled)
> J9VM - 20060915_08260_LHdSMr
> JIT - 20060908_1811_r8
> GC - 20060906_AA)
> JCL - 20061002
>
> bea
> ---
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> BEA JRockit(R) (build
> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>
>
> --------------------------------------------------------------------------
>
> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>
> command: java jnt.scimark2.commandline
>
> NOTE: bigger number is better
>
> Sun6
> Composite Score: 364.5832265230057
> FFT (1024): 220.8458713892794
> SOR (100x100): 696.1542342357722
> Monte Carlo : 149.37978088875656
> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> LU (100x100): 430.1617273683819
>
> BEA
> Composite Score: 359.13480378697835
> FFT (1024): 303.8746880751562
> SOR (100x100): 454.25628897202307
> Monte Carlo : 93.23913192138497
> Sparse matmult (N=1000, nz=5000): 530.44112637391
> LU (100x100): 413.8627835924175
>
> Sun5
> Composite Score: 332.84987587548574
> FFT (1024): 216.5144595799027
> SOR (100x100): 689.429322146947
> Monte Carlo : 25.791262124978065
> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> LU (100x100): 414.99493895566377
>
> IBM
> Composite Score: 259.8249218693683
> FFT (1024): 296.8415012789055
> SOR (100x100): 428.974881649179
> Monte Carlo : 89.15159857584082
> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> LU (100x100): 339.8042037225181
>
> Harmony
> Composite Score: 113.65082278962575
> FFT (1024): 203.76641991778123
> SOR (100x100): 224.37761309236748
> Monte Carlo : 9.063866256533116
> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> LU (100x100): 65.6410280487242
>
> In this test harmony is clearly lagging behind... at about 30%
> performance of the best JVM, it's a little crappy. Please note how FFT's
> performance is not so bad awhile monte carlo is pretty bad compared to
> BEA or IBM.
>
> Overall, it seems like there is some serious work to do here to catch up.
>
> --------------------------------------------------------------------------
>
> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>
> command: java dhry 100000000
>
> NOTE: bigger is better
>
> NB: I modified the code to accept the count at input from the command line!
>
> sun6: 8552856 dhrystones/sec
> sun5: 6605892
> bea: 5678914
> harmony: 669734
> ibm: 501562
>
> The performance here is horrific but what's surprising is that J9 is
> even worse. No idea what's going on but it seems like something is not
> working as it should (in both harmony and J9)
>
> --------------------------------------------------------------------------
>
> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>
> command: java Sieve 30
>
> NB: I modified the test to run for a configurable amount of seconds.
>
> sun6 8545 sieves/sec
> sun5 8364
> bea 6174
> harmony 1836
> ibm 225
>
> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> lagging behind.
>
> Stay tuned for more tests.
>
> --
> Stefano.
>
>
Re: [performance] a few early benchmarks
Posted by Andrew Zhang <zh...@gmail.com>.
On 11/19/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> Andrew Zhang wrote:
> > On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> >> harmony
> >> -------
> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> Foundation or its licensors, as applicable.
> >> java version "1.5.0"
> >> pre-alpha : not complete or compatible
> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >
> > How about the performance of Harmony classlib + IBM VME? It may help us
> to
> > find out how many spaces are there for us to improve classlib code. :)
>
> How about pressuring the IBM guys to release a J9 on x86_64 that works
> with harmony? ;-)
We are looking forawrd to... :-)
As soon as they make it available, I'll be more than happy to include
> those numbers in the test as well.
>
> --
> Stefano.
>
>
--
Best regards,
Andrew Zhang
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Andrew Zhang wrote:
> On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>> harmony
>> -------
>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> Foundation or its licensors, as applicable.
>> java version "1.5.0"
>> pre-alpha : not complete or compatible
>> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>
> How about the performance of Harmony classlib + IBM VME? It may help us to
> find out how many spaces are there for us to improve classlib code. :)
How about pressuring the IBM guys to release a J9 on x86_64 that works
with harmony? ;-)
As soon as they make it available, I'll be more than happy to include
those numbers in the test as well.
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Alexei Fedotov wrote:
> Stefano,
> That is great. Thank you for a nice comparison!
My pleasure, really. I'm a sucker for stats, charts and maps ;-)
> Andrew Zhang wrote,
>> How about the performance of Harmony classlib + IBM VME?
>
> What does IBM/J9 mean in your list? Does it use Harmony class libraries?
No, IBM/J9 uses its own classlib.
> On 11/18/06, Andrew Zhang <zh...@gmail.com> wrote:
>> On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>> >
>> > There are lies, damn lies and benchmarks.... which don't really tell
>> you
>> > if an implementation of a program is *faster* but at least it tells you
>> > where you're at.
>> >
>> > So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>> > was able to start running some benchmarks.
>> >
>> > The machine is the following:
>> >
>> > Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>> > 01:50:50 UTC 2006 x86_64 GNU/Linux
>> >
>> > dual Intel(R) Pentium(R) D CPU 3.20GHz
>> > bogomips 6410.31 (per CPU)
>> >
>> > There is nothing else running on the machine (load is 0.04 at the time
>> > of testing).
>> >
>> > The various virtual machines tested are:
>> >
>> > harmony
>> > -------
>> > Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> > Foundation or its licensors, as applicable.
>> > java version "1.5.0"
>> > pre-alpha : not complete or compatible
>> > svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>>
>>
>> How about the performance of Harmony classlib + IBM VME? It may help
>> us to
>> find out how many spaces are there for us to improve classlib code. :)
>>
>> sun5
>> > ---
>> > java version "1.5.0_09"
>> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>> > Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>> >
>> > sun6
>> > ----
>> > java version "1.6.0-rc"
>> > Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>> > Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>> >
>> > ibm
>> > ---
>> > java version "1.5.0"
>> > Java(TM) 2 Runtime Environment, Standard Edition (build
>> > pxa64dev-20061002a (SR3) )
>> > IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>> > j9vmxa6423-20061001 (JIT enabled)
>> > J9VM - 20060915_08260_LHdSMr
>> > JIT - 20060908_1811_r8
>> > GC - 20060906_AA)
>> > JCL - 20061002
>> >
>> > bea
>> > ---
>> > java version "1.5.0_06"
>> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> > BEA JRockit(R) (build
>> > R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>> >
>> >
>> >
>> --------------------------------------------------------------------------
>>
>> >
>> > Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>> >
>> > command: java jnt.scimark2.commandline
>> >
>> > NOTE: bigger number is better
>> >
>> > Sun6
>> > Composite Score: 364.5832265230057
>> > FFT (1024): 220.8458713892794
>> > SOR (100x100): 696.1542342357722
>> > Monte Carlo : 149.37978088875656
>> > Sparse matmult (N=1000, nz=5000): 326.37451873283845
>> > LU (100x100): 430.1617273683819
>> >
>> > BEA
>> > Composite Score: 359.13480378697835
>> > FFT (1024): 303.8746880751562
>> > SOR (100x100): 454.25628897202307
>> > Monte Carlo : 93.23913192138497
>> > Sparse matmult (N=1000, nz=5000): 530.44112637391
>> > LU (100x100): 413.8627835924175
>> >
>> > Sun5
>> > Composite Score: 332.84987587548574
>> > FFT (1024): 216.5144595799027
>> > SOR (100x100): 689.429322146947
>> > Monte Carlo : 25.791262124978065
>> > Sparse matmult (N=1000, nz=5000): 317.5193965699373
>> > LU (100x100): 414.99493895566377
>> >
>> > IBM
>> > Composite Score: 259.8249218693683
>> > FFT (1024): 296.8415012789055
>> > SOR (100x100): 428.974881649179
>> > Monte Carlo : 89.15159857584082
>> > Sparse matmult (N=1000, nz=5000): 144.3524241203982
>> > LU (100x100): 339.8042037225181
>> >
>> > Harmony
>> > Composite Score: 113.65082278962575
>> > FFT (1024): 203.76641991778123
>> > SOR (100x100): 224.37761309236748
>> > Monte Carlo : 9.063866256533116
>> > Sparse matmult (N=1000, nz=5000): 65.4051866327227
>> > LU (100x100): 65.6410280487242
>> >
>> > In this test harmony is clearly lagging behind... at about 30%
>> > performance of the best JVM, it's a little crappy. Please note how
>> FFT's
>> > performance is not so bad awhile monte carlo is pretty bad compared to
>> > BEA or IBM.
>> >
>> > Overall, it seems like there is some serious work to do here to
>> catch up.
>> >
>> >
>> --------------------------------------------------------------------------
>>
>> >
>> > Test 2: Dhrystones
>> (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>> >
>> > command: java dhry 100000000
>> >
>> > NOTE: bigger is better
>> >
>> > NB: I modified the code to accept the count at input from the command
>> > line!
>> >
>> > sun6: 8552856 dhrystones/sec
>> > sun5: 6605892
>> > bea: 5678914
>> > harmony: 669734
>> > ibm: 501562
>> >
>> > The performance here is horrific but what's surprising is that J9 is
>> > even worse. No idea what's going on but it seems like something is not
>> > working as it should (in both harmony and J9)
>> >
>> >
>> --------------------------------------------------------------------------
>>
>> >
>> > Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>> >
>> > command: java Sieve 30
>> >
>> > NB: I modified the test to run for a configurable amount of seconds.
>> >
>> > sun6 8545 sieves/sec
>> > sun5 8364
>> > bea 6174
>> > harmony 1836
>> > ibm 225
>> >
>> > IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>> > lagging behind.
>> >
>> > Stay tuned for more tests.
>> >
>> > --
>> > Stefano.
>> >
>> >
>>
>>
>> --
>> Best regards,
>> Andrew Zhang
>>
>>
>
>
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Alexei Fedotov <al...@gmail.com>.
Stefano,
That is great. Thank you for a nice comparison!
Andrew Zhang wrote,
> How about the performance of Harmony classlib + IBM VME?
What does IBM/J9 mean in your list? Does it use Harmony class libraries?
On 11/18/06, Andrew Zhang <zh...@gmail.com> wrote:
> On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
> >
> > There are lies, damn lies and benchmarks.... which don't really tell you
> > if an implementation of a program is *faster* but at least it tells you
> > where you're at.
> >
> > So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> > was able to start running some benchmarks.
> >
> > The machine is the following:
> >
> > Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> > 01:50:50 UTC 2006 x86_64 GNU/Linux
> >
> > dual Intel(R) Pentium(R) D CPU 3.20GHz
> > bogomips 6410.31 (per CPU)
> >
> > There is nothing else running on the machine (load is 0.04 at the time
> > of testing).
> >
> > The various virtual machines tested are:
> >
> > harmony
> > -------
> > Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> > Foundation or its licensors, as applicable.
> > java version "1.5.0"
> > pre-alpha : not complete or compatible
> > svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>
>
> How about the performance of Harmony classlib + IBM VME? It may help us to
> find out how many spaces are there for us to improve classlib code. :)
>
> sun5
> > ---
> > java version "1.5.0_09"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> > Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >
> > sun6
> > ----
> > java version "1.6.0-rc"
> > Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> > Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >
> > ibm
> > ---
> > java version "1.5.0"
> > Java(TM) 2 Runtime Environment, Standard Edition (build
> > pxa64dev-20061002a (SR3) )
> > IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> > j9vmxa6423-20061001 (JIT enabled)
> > J9VM - 20060915_08260_LHdSMr
> > JIT - 20060908_1811_r8
> > GC - 20060906_AA)
> > JCL - 20061002
> >
> > bea
> > ---
> > java version "1.5.0_06"
> > Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> > BEA JRockit(R) (build
> > R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >
> >
> > --------------------------------------------------------------------------
> >
> > Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >
> > command: java jnt.scimark2.commandline
> >
> > NOTE: bigger number is better
> >
> > Sun6
> > Composite Score: 364.5832265230057
> > FFT (1024): 220.8458713892794
> > SOR (100x100): 696.1542342357722
> > Monte Carlo : 149.37978088875656
> > Sparse matmult (N=1000, nz=5000): 326.37451873283845
> > LU (100x100): 430.1617273683819
> >
> > BEA
> > Composite Score: 359.13480378697835
> > FFT (1024): 303.8746880751562
> > SOR (100x100): 454.25628897202307
> > Monte Carlo : 93.23913192138497
> > Sparse matmult (N=1000, nz=5000): 530.44112637391
> > LU (100x100): 413.8627835924175
> >
> > Sun5
> > Composite Score: 332.84987587548574
> > FFT (1024): 216.5144595799027
> > SOR (100x100): 689.429322146947
> > Monte Carlo : 25.791262124978065
> > Sparse matmult (N=1000, nz=5000): 317.5193965699373
> > LU (100x100): 414.99493895566377
> >
> > IBM
> > Composite Score: 259.8249218693683
> > FFT (1024): 296.8415012789055
> > SOR (100x100): 428.974881649179
> > Monte Carlo : 89.15159857584082
> > Sparse matmult (N=1000, nz=5000): 144.3524241203982
> > LU (100x100): 339.8042037225181
> >
> > Harmony
> > Composite Score: 113.65082278962575
> > FFT (1024): 203.76641991778123
> > SOR (100x100): 224.37761309236748
> > Monte Carlo : 9.063866256533116
> > Sparse matmult (N=1000, nz=5000): 65.4051866327227
> > LU (100x100): 65.6410280487242
> >
> > In this test harmony is clearly lagging behind... at about 30%
> > performance of the best JVM, it's a little crappy. Please note how FFT's
> > performance is not so bad awhile monte carlo is pretty bad compared to
> > BEA or IBM.
> >
> > Overall, it seems like there is some serious work to do here to catch up.
> >
> > --------------------------------------------------------------------------
> >
> > Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
> >
> > command: java dhry 100000000
> >
> > NOTE: bigger is better
> >
> > NB: I modified the code to accept the count at input from the command
> > line!
> >
> > sun6: 8552856 dhrystones/sec
> > sun5: 6605892
> > bea: 5678914
> > harmony: 669734
> > ibm: 501562
> >
> > The performance here is horrific but what's surprising is that J9 is
> > even worse. No idea what's going on but it seems like something is not
> > working as it should (in both harmony and J9)
> >
> > --------------------------------------------------------------------------
> >
> > Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >
> > command: java Sieve 30
> >
> > NB: I modified the test to run for a configurable amount of seconds.
> >
> > sun6 8545 sieves/sec
> > sun5 8364
> > bea 6174
> > harmony 1836
> > ibm 225
> >
> > IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> > lagging behind.
> >
> > Stay tuned for more tests.
> >
> > --
> > Stefano.
> >
> >
>
>
> --
> Best regards,
> Andrew Zhang
>
>
--
Thank you,
Alexei
Re: [performance] a few early benchmarks
Posted by Andrew Zhang <zh...@gmail.com>.
On 11/17/06, Stefano Mazzocchi <st...@apache.org> wrote:
>
> There are lies, damn lies and benchmarks.... which don't really tell you
> if an implementation of a program is *faster* but at least it tells you
> where you're at.
>
> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> was able to start running some benchmarks.
>
> The machine is the following:
>
> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> 01:50:50 UTC 2006 x86_64 GNU/Linux
>
> dual Intel(R) Pentium(R) D CPU 3.20GHz
> bogomips 6410.31 (per CPU)
>
> There is nothing else running on the machine (load is 0.04 at the time
> of testing).
>
> The various virtual machines tested are:
>
> harmony
> -------
> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> Foundation or its licensors, as applicable.
> java version "1.5.0"
> pre-alpha : not complete or compatible
> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
How about the performance of Harmony classlib + IBM VME? It may help us to
find out how many spaces are there for us to improve classlib code. :)
sun5
> ---
> java version "1.5.0_09"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>
> sun6
> ----
> java version "1.6.0-rc"
> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>
> ibm
> ---
> java version "1.5.0"
> Java(TM) 2 Runtime Environment, Standard Edition (build
> pxa64dev-20061002a (SR3) )
> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> j9vmxa6423-20061001 (JIT enabled)
> J9VM - 20060915_08260_LHdSMr
> JIT - 20060908_1811_r8
> GC - 20060906_AA)
> JCL - 20061002
>
> bea
> ---
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> BEA JRockit(R) (build
> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>
>
> --------------------------------------------------------------------------
>
> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>
> command: java jnt.scimark2.commandline
>
> NOTE: bigger number is better
>
> Sun6
> Composite Score: 364.5832265230057
> FFT (1024): 220.8458713892794
> SOR (100x100): 696.1542342357722
> Monte Carlo : 149.37978088875656
> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> LU (100x100): 430.1617273683819
>
> BEA
> Composite Score: 359.13480378697835
> FFT (1024): 303.8746880751562
> SOR (100x100): 454.25628897202307
> Monte Carlo : 93.23913192138497
> Sparse matmult (N=1000, nz=5000): 530.44112637391
> LU (100x100): 413.8627835924175
>
> Sun5
> Composite Score: 332.84987587548574
> FFT (1024): 216.5144595799027
> SOR (100x100): 689.429322146947
> Monte Carlo : 25.791262124978065
> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> LU (100x100): 414.99493895566377
>
> IBM
> Composite Score: 259.8249218693683
> FFT (1024): 296.8415012789055
> SOR (100x100): 428.974881649179
> Monte Carlo : 89.15159857584082
> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> LU (100x100): 339.8042037225181
>
> Harmony
> Composite Score: 113.65082278962575
> FFT (1024): 203.76641991778123
> SOR (100x100): 224.37761309236748
> Monte Carlo : 9.063866256533116
> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> LU (100x100): 65.6410280487242
>
> In this test harmony is clearly lagging behind... at about 30%
> performance of the best JVM, it's a little crappy. Please note how FFT's
> performance is not so bad awhile monte carlo is pretty bad compared to
> BEA or IBM.
>
> Overall, it seems like there is some serious work to do here to catch up.
>
> --------------------------------------------------------------------------
>
> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>
> command: java dhry 100000000
>
> NOTE: bigger is better
>
> NB: I modified the code to accept the count at input from the command
> line!
>
> sun6: 8552856 dhrystones/sec
> sun5: 6605892
> bea: 5678914
> harmony: 669734
> ibm: 501562
>
> The performance here is horrific but what's surprising is that J9 is
> even worse. No idea what's going on but it seems like something is not
> working as it should (in both harmony and J9)
>
> --------------------------------------------------------------------------
>
> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>
> command: java Sieve 30
>
> NB: I modified the test to run for a configurable amount of seconds.
>
> sun6 8545 sieves/sec
> sun5 8364
> bea 6174
> harmony 1836
> ibm 225
>
> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> lagging behind.
>
> Stay tuned for more tests.
>
> --
> Stefano.
>
>
--
Best regards,
Andrew Zhang
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
> Egor Pasko wrote:
> > Stefano,
> >
> > thank you for pointing peroformance issues out!
>
> you're welcome, but I have to warn you, I've barely started... you might
> get sick of my performance regression charts in the future ;-)
I cannot wait to get the best from your charts :)
> > Although that was a debug build, it uncovers that Harmony is not
> > properly tuned for FP-intensive workloads as SciMark. One of the
> > possible reasons of this (known:) limitation is also raised in the
> > thread "[drlvm][jit][ia-32]register-based fast calling convention",
> > where Alex vounteers to help with more intelligent XMM register
> > usage. I consider this a quite-high priority, fully support Alex, hope
> > for the best, ready to help, etc etc.
>
> Please understand that I'm not pointing fingers nor trying to insult
> people and their work (or pride).
I understand. I am just thinking in technical terms making assumptions
on the root causes, giving idea on the benchmark that should be
sensitive to what Alex is doing (so, he is free from writing a
benchmark on his own).
I am not trying to convince anyone that we are doing the good work. I
am thinking how to make the work even better. And your reports help here.
> Rather the opposite: there is nothing like hard facts and evolution
> charts to make developers focus. As much as japi makes it rewarding to
> work on the API because we approach the mythical 100%, by showing the
> performance comparisons to the state of the art, we understand how far
> behind (or ahead!) we are.
>
> I fully believe in the power of open development over closed
> development. It's faster, more efficient, more innovative and overall,
> more effective.
>
> Open development is the reason why Harmony exists, not open source.
>
> Open source is a necessary condition, but hardly a sufficient one. Sun
> will realize that the hard way.
That's something to write in my blog :) AFAIR, Sun wants to have
comitters from the outside-sun-world. That's pretty good in terms of
openness.
> This means that Harmony has the full potential to blow all other JVMs
> out of the water in terms of reliability, portability and performance...
> just look at the development momentum and forecast what can happen in a
> a year or so.
>
> So, no, don't take this the wrong way: it's not saying "bad guys, no
> donut", it's saying "keep it up and here's how fast you're going, just FYI".
Yes, thank you! I understand.
> Keep up the good work. I'm just building tools to give you (and others
> around us!) a condensed view of the status and evolution of the project
> in all of the aspects that matter (compatibility, reliability,
> performance, quality, and momentum)
>
> > On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
> >> There are lies, damn lies and benchmarks.... which don't really tell you
> >> if an implementation of a program is *faster* but at least it tells you
> >> where you're at.
> >>
> >> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> >> was able to start running some benchmarks.
> >>
> >> The machine is the following:
> >>
> >> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> >> 01:50:50 UTC 2006 x86_64 GNU/Linux
> >>
> >> dual Intel(R) Pentium(R) D CPU 3.20GHz
> >> bogomips 6410.31 (per CPU)
> >>
> >> There is nothing else running on the machine (load is 0.04 at the time
> >> of testing).
> >>
> >> The various virtual machines tested are:
> >>
> >> harmony
> >> -------
> >> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> >> Foundation or its licensors, as applicable.
> >> java version "1.5.0"
> >> pre-alpha : not complete or compatible
> >> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
> >>
> >> sun5
> >> ---
> >> java version "1.5.0_09"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
> >>
> >> sun6
> >> ----
> >> java version "1.6.0-rc"
> >> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> >> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
> >>
> >> ibm
> >> ---
> >> java version "1.5.0"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build
> >> pxa64dev-20061002a (SR3) )
> >> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> >> j9vmxa6423-20061001 (JIT enabled)
> >> J9VM - 20060915_08260_LHdSMr
> >> JIT - 20060908_1811_r8
> >> GC - 20060906_AA)
> >> JCL - 20061002
> >>
> >> bea
> >> ---
> >> java version "1.5.0_06"
> >> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> >> BEA JRockit(R) (build
> >> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
> >>
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
> >>
> >> command: java jnt.scimark2.commandline
> >>
> >> NOTE: bigger number is better
> >>
> >> Sun6
> >> Composite Score: 364.5832265230057
> >> FFT (1024): 220.8458713892794
> >> SOR (100x100): 696.1542342357722
> >> Monte Carlo : 149.37978088875656
> >> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> >> LU (100x100): 430.1617273683819
> >>
> >> BEA
> >> Composite Score: 359.13480378697835
> >> FFT (1024): 303.8746880751562
> >> SOR (100x100): 454.25628897202307
> >> Monte Carlo : 93.23913192138497
> >> Sparse matmult (N=1000, nz=5000): 530.44112637391
> >> LU (100x100): 413.8627835924175
> >>
> >> Sun5
> >> Composite Score: 332.84987587548574
> >> FFT (1024): 216.5144595799027
> >> SOR (100x100): 689.429322146947
> >> Monte Carlo : 25.791262124978065
> >> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> >> LU (100x100): 414.99493895566377
> >>
> >> IBM
> >> Composite Score: 259.8249218693683
> >> FFT (1024): 296.8415012789055
> >> SOR (100x100): 428.974881649179
> >> Monte Carlo : 89.15159857584082
> >> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> >> LU (100x100): 339.8042037225181
> >>
> >> Harmony
> >> Composite Score: 113.65082278962575
> >> FFT (1024): 203.76641991778123
> >> SOR (100x100): 224.37761309236748
> >> Monte Carlo : 9.063866256533116
> >> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> >> LU (100x100): 65.6410280487242
> >>
> >> In this test harmony is clearly lagging behind... at about 30%
> >> performance of the best JVM, it's a little crappy. Please note how FFT's
> >> performance is not so bad awhile monte carlo is pretty bad compared to
> >> BEA or IBM.
> >>
> >> Overall, it seems like there is some serious work to do here to catch up.
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
> >>
> >> command: java dhry 100000000
> >>
> >> NOTE: bigger is better
> >>
> >> NB: I modified the code to accept the count at input from the command line!
> >>
> >> sun6: 8552856 dhrystones/sec
> >> sun5: 6605892
> >> bea: 5678914
> >> harmony: 669734
> >> ibm: 501562
> >>
> >> The performance here is horrific but what's surprising is that J9 is
> >> even worse. No idea what's going on but it seems like something is not
> >> working as it should (in both harmony and J9)
> >>
> >> --------------------------------------------------------------------------
> >>
> >> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
> >>
> >> command: java Sieve 30
> >>
> >> NB: I modified the test to run for a configurable amount of seconds.
> >>
> >> sun6 8545 sieves/sec
> >> sun5 8364
> >> bea 6174
> >> harmony 1836
> >> ibm 225
> >>
> >> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> >> lagging behind.
> >>
> >> Stay tuned for more tests.
> >>
> >> --
> >> Stefano.
> >>
> >>
> >
>
>
> --
> Stefano.
>
>
--
Egor Pasko
Re: [performance] a few early benchmarks
Posted by Stefano Mazzocchi <st...@apache.org>.
Egor Pasko wrote:
> Stefano,
>
> thank you for pointing peroformance issues out!
you're welcome, but I have to warn you, I've barely started... you might
get sick of my performance regression charts in the future ;-)
> Although that was a debug build, it uncovers that Harmony is not
> properly tuned for FP-intensive workloads as SciMark. One of the
> possible reasons of this (known:) limitation is also raised in the
> thread "[drlvm][jit][ia-32]register-based fast calling convention",
> where Alex vounteers to help with more intelligent XMM register
> usage. I consider this a quite-high priority, fully support Alex, hope
> for the best, ready to help, etc etc.
Please understand that I'm not pointing fingers nor trying to insult
people and their work (or pride).
Rather the opposite: there is nothing like hard facts and evolution
charts to make developers focus. As much as japi makes it rewarding to
work on the API because we approach the mythical 100%, by showing the
performance comparisons to the state of the art, we understand how far
behind (or ahead!) we are.
I fully believe in the power of open development over closed
development. It's faster, more efficient, more innovative and overall,
more effective.
Open development is the reason why Harmony exists, not open source.
Open source is a necessary condition, but hardly a sufficient one. Sun
will realize that the hard way.
This means that Harmony has the full potential to blow all other JVMs
out of the water in terms of reliability, portability and performance...
just look at the development momentum and forecast what can happen in a
a year or so.
So, no, don't take this the wrong way: it's not saying "bad guys, no
donut", it's saying "keep it up and here's how fast you're going, just FYI".
Keep up the good work. I'm just building tools to give you (and others
around us!) a condensed view of the status and evolution of the project
in all of the aspects that matter (compatibility, reliability,
performance, quality, and momentum)
> On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
>> There are lies, damn lies and benchmarks.... which don't really tell you
>> if an implementation of a program is *faster* but at least it tells you
>> where you're at.
>>
>> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
>> was able to start running some benchmarks.
>>
>> The machine is the following:
>>
>> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
>> 01:50:50 UTC 2006 x86_64 GNU/Linux
>>
>> dual Intel(R) Pentium(R) D CPU 3.20GHz
>> bogomips 6410.31 (per CPU)
>>
>> There is nothing else running on the machine (load is 0.04 at the time
>> of testing).
>>
>> The various virtual machines tested are:
>>
>> harmony
>> -------
>> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
>> Foundation or its licensors, as applicable.
>> java version "1.5.0"
>> pre-alpha : not complete or compatible
>> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>>
>> sun5
>> ---
>> java version "1.5.0_09"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>>
>> sun6
>> ----
>> java version "1.6.0-rc"
>> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
>> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>>
>> ibm
>> ---
>> java version "1.5.0"
>> Java(TM) 2 Runtime Environment, Standard Edition (build
>> pxa64dev-20061002a (SR3) )
>> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
>> j9vmxa6423-20061001 (JIT enabled)
>> J9VM - 20060915_08260_LHdSMr
>> JIT - 20060908_1811_r8
>> GC - 20060906_AA)
>> JCL - 20061002
>>
>> bea
>> ---
>> java version "1.5.0_06"
>> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
>> BEA JRockit(R) (build
>> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>>
>>
>> --------------------------------------------------------------------------
>>
>> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>>
>> command: java jnt.scimark2.commandline
>>
>> NOTE: bigger number is better
>>
>> Sun6
>> Composite Score: 364.5832265230057
>> FFT (1024): 220.8458713892794
>> SOR (100x100): 696.1542342357722
>> Monte Carlo : 149.37978088875656
>> Sparse matmult (N=1000, nz=5000): 326.37451873283845
>> LU (100x100): 430.1617273683819
>>
>> BEA
>> Composite Score: 359.13480378697835
>> FFT (1024): 303.8746880751562
>> SOR (100x100): 454.25628897202307
>> Monte Carlo : 93.23913192138497
>> Sparse matmult (N=1000, nz=5000): 530.44112637391
>> LU (100x100): 413.8627835924175
>>
>> Sun5
>> Composite Score: 332.84987587548574
>> FFT (1024): 216.5144595799027
>> SOR (100x100): 689.429322146947
>> Monte Carlo : 25.791262124978065
>> Sparse matmult (N=1000, nz=5000): 317.5193965699373
>> LU (100x100): 414.99493895566377
>>
>> IBM
>> Composite Score: 259.8249218693683
>> FFT (1024): 296.8415012789055
>> SOR (100x100): 428.974881649179
>> Monte Carlo : 89.15159857584082
>> Sparse matmult (N=1000, nz=5000): 144.3524241203982
>> LU (100x100): 339.8042037225181
>>
>> Harmony
>> Composite Score: 113.65082278962575
>> FFT (1024): 203.76641991778123
>> SOR (100x100): 224.37761309236748
>> Monte Carlo : 9.063866256533116
>> Sparse matmult (N=1000, nz=5000): 65.4051866327227
>> LU (100x100): 65.6410280487242
>>
>> In this test harmony is clearly lagging behind... at about 30%
>> performance of the best JVM, it's a little crappy. Please note how FFT's
>> performance is not so bad awhile monte carlo is pretty bad compared to
>> BEA or IBM.
>>
>> Overall, it seems like there is some serious work to do here to catch up.
>>
>> --------------------------------------------------------------------------
>>
>> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>>
>> command: java dhry 100000000
>>
>> NOTE: bigger is better
>>
>> NB: I modified the code to accept the count at input from the command line!
>>
>> sun6: 8552856 dhrystones/sec
>> sun5: 6605892
>> bea: 5678914
>> harmony: 669734
>> ibm: 501562
>>
>> The performance here is horrific but what's surprising is that J9 is
>> even worse. No idea what's going on but it seems like something is not
>> working as it should (in both harmony and J9)
>>
>> --------------------------------------------------------------------------
>>
>> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>>
>> command: java Sieve 30
>>
>> NB: I modified the test to run for a configurable amount of seconds.
>>
>> sun6 8545 sieves/sec
>> sun5 8364
>> bea 6174
>> harmony 1836
>> ibm 225
>>
>> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
>> lagging behind.
>>
>> Stay tuned for more tests.
>>
>> --
>> Stefano.
>>
>>
>
--
Stefano.
Re: [performance] a few early benchmarks
Posted by Egor Pasko <eg...@gmail.com>.
Stefano,
thank you for pointing peroformance issues out!
Although that was a debug build, it uncovers that Harmony is not
properly tuned for FP-intensive workloads as SciMark. One of the
possible reasons of this (known:) limitation is also raised in the
thread "[drlvm][jit][ia-32]register-based fast calling convention",
where Alex vounteers to help with more intelligent XMM register
usage. I consider this a quite-high priority, fully support Alex, hope
for the best, ready to help, etc etc.
On the 0x224 day of Apache Harmony Stefano Mazzocchi wrote:
> There are lies, damn lies and benchmarks.... which don't really tell you
> if an implementation of a program is *faster* but at least it tells you
> where you're at.
>
> So, as Geir managed to get the DSO linking problem go away in DRLVM, I
> was able to start running some benchmarks.
>
> The machine is the following:
>
> Linux harmony-em64t 2.6.15-27-amd64-generic #1 SMP PREEMPT Sat Sep 16
> 01:50:50 UTC 2006 x86_64 GNU/Linux
>
> dual Intel(R) Pentium(R) D CPU 3.20GHz
> bogomips 6410.31 (per CPU)
>
> There is nothing else running on the machine (load is 0.04 at the time
> of testing).
>
> The various virtual machines tested are:
>
> harmony
> -------
> Apache Harmony Launcher : (c) Copyright 1991, 2006 The Apache Software
> Foundation or its licensors, as applicable.
> java version "1.5.0"
> pre-alpha : not complete or compatible
> svn = r476006, (Nov 16 2006), Linux/em64t/gcc 4.0.3, debug build
>
> sun5
> ---
> java version "1.5.0_09"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_09-b03)
> Java HotSpot(TM) 64-Bit Server VM (build 1.5.0_09-b03, mixed mode)
>
> sun6
> ----
> java version "1.6.0-rc"
> Java(TM) SE Runtime Environment (build 1.6.0-rc-b104)
> Java HotSpot(TM) 64-Bit Server VM (build 1.6.0-rc-b104, mixed mode)
>
> ibm
> ---
> java version "1.5.0"
> Java(TM) 2 Runtime Environment, Standard Edition (build
> pxa64dev-20061002a (SR3) )
> IBM J9 VM (build 2.3, J2RE 1.5.0 IBM J9 2.3 Linux amd64-64
> j9vmxa6423-20061001 (JIT enabled)
> J9VM - 20060915_08260_LHdSMr
> JIT - 20060908_1811_r8
> GC - 20060906_AA)
> JCL - 20061002
>
> bea
> ---
> java version "1.5.0_06"
> Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_06-b05)
> BEA JRockit(R) (build
> R26.4.0-63-63688-1.5.0_06-20060626-2259-linux-x86_64, )
>
>
> --------------------------------------------------------------------------
>
> Test #1: java scimark2 (http://math.nist.gov/scimark2/)
>
> command: java jnt.scimark2.commandline
>
> NOTE: bigger number is better
>
> Sun6
> Composite Score: 364.5832265230057
> FFT (1024): 220.8458713892794
> SOR (100x100): 696.1542342357722
> Monte Carlo : 149.37978088875656
> Sparse matmult (N=1000, nz=5000): 326.37451873283845
> LU (100x100): 430.1617273683819
>
> BEA
> Composite Score: 359.13480378697835
> FFT (1024): 303.8746880751562
> SOR (100x100): 454.25628897202307
> Monte Carlo : 93.23913192138497
> Sparse matmult (N=1000, nz=5000): 530.44112637391
> LU (100x100): 413.8627835924175
>
> Sun5
> Composite Score: 332.84987587548574
> FFT (1024): 216.5144595799027
> SOR (100x100): 689.429322146947
> Monte Carlo : 25.791262124978065
> Sparse matmult (N=1000, nz=5000): 317.5193965699373
> LU (100x100): 414.99493895566377
>
> IBM
> Composite Score: 259.8249218693683
> FFT (1024): 296.8415012789055
> SOR (100x100): 428.974881649179
> Monte Carlo : 89.15159857584082
> Sparse matmult (N=1000, nz=5000): 144.3524241203982
> LU (100x100): 339.8042037225181
>
> Harmony
> Composite Score: 113.65082278962575
> FFT (1024): 203.76641991778123
> SOR (100x100): 224.37761309236748
> Monte Carlo : 9.063866256533116
> Sparse matmult (N=1000, nz=5000): 65.4051866327227
> LU (100x100): 65.6410280487242
>
> In this test harmony is clearly lagging behind... at about 30%
> performance of the best JVM, it's a little crappy. Please note how FFT's
> performance is not so bad awhile monte carlo is pretty bad compared to
> BEA or IBM.
>
> Overall, it seems like there is some serious work to do here to catch up.
>
> --------------------------------------------------------------------------
>
> Test 2: Dhrystones (http://www.c-creators.co.jp/okayan/DhrystoneApplet/)
>
> command: java dhry 100000000
>
> NOTE: bigger is better
>
> NB: I modified the code to accept the count at input from the command line!
>
> sun6: 8552856 dhrystones/sec
> sun5: 6605892
> bea: 5678914
> harmony: 669734
> ibm: 501562
>
> The performance here is horrific but what's surprising is that J9 is
> even worse. No idea what's going on but it seems like something is not
> working as it should (in both harmony and J9)
>
> --------------------------------------------------------------------------
>
> Test 3: Sieve (part of http://www.sax.de/~adlibit/tya18.tgz)
>
> command: java Sieve 30
>
> NB: I modified the test to run for a configurable amount of seconds.
>
> sun6 8545 sieves/sec
> sun5 8364
> bea 6174
> harmony 1836
> ibm 225
>
> IBM J9 clearly has something wrong on x86_64 but harmony is clearly
> lagging behind.
>
> Stay tuned for more tests.
>
> --
> Stefano.
>
>
--
Egor Pasko