Tests scores on http://harmonytest.org Was: [DRLVM] General stability

["Fedotov, Alexei A" <al...@intel.com>]

Geir,
I like the overall letter. 

Anton,

I have related question. How can we improve http://harmonytest.org to
make it possible to publish not just pass, fail, or error but numeric
test scores? 

How this is related to the letter? I believe that stress tests which
were mentioned in the letter may have scores in a way performance tests
do, see
http://incubator.apache.org/harmony/subcomponents/stresstest/index.html

BTW, Egor Pasko requested scores to report documentation quality.

With best regards,
Alexei Fedotov,
Intel Java & XML Engineering

>-----Original Message-----
>From: Geir Magnusson Jr. [mailto:geir@pobox.com]
>Sent: Thursday, November 09, 2006 4:51 AM
>To: harmony-dev@incubator.apache.org
>Subject: Re: [DRLVM] General stability
>
>
>
>Fedotov, Alexei A wrote:
>> Alexey Petrenko wrote,
>>> The only release I can imagine is Harmony Java5SE 100% compatible.
>>> To be Java5SE 100% compatible we need TCK first.
>>
>> +1
>>
>
>Yes - and I still think that talk of a release is a bit premature right
>now.
>
>The key things that I believe we need to focus on are
>
>  a) stability and
>
>  b) completeness.
>
>  c) reliability (which may be 'stability')
>
>(and not always in that order :)
>
>
>Things I'd like to see us do :
>
>1)  We need to drive to fully working unit tests for both DRLVM and
>classlib  (using DRLVM).  Great progress has been made in this area,
and
>  we should probably make this a "campaign" for DRLVM as we did for
>classlib.
>
>2) Add stress tests
>
>3) Get our CC-based build-test framework patched and running on as many
>platforms as possible, reporting breakage into the list.
>
>4) Identify problem areas and focus on them.  For example, threading in
>DRLVM...
>
>I do think of us having a 'zero regression' policy except in cases
where
>we make the explicit decision to break.  (like we did with TM, for
example)
>
>
>> I hesitate to say that again, but we also need to decide about VM we
>> will use for that release. I like the following mission: "Class
library
>> and DRLVM pass TCK on Ubuntu 6". I'm open for any other mission which
is
>> challenging, understandable and short enough.
>
>Well, we'll need Windows XP and RHEL as well.
>
>
>>
>> Writing down this mission certainly shouldn't inhibit individuals
from
>> achieving other goals at Harmony. But it would help the rest of
>> community to concentrate on the common task.
>>
>> 1.
>>
http://wiki.apache.org/harmony/Platforms_to_Run_Harmony_Development_Kit_
>> on
>>
>> With best regards,
>> Alexei Fedotov,
>> Intel Java & XML Engineering
>>
>>> -----Original Message-----
>>> From: Alexey Petrenko [mailto:alexey.a.petrenko@gmail.com]
>>> Sent: Wednesday, November 08, 2006 10:36 AM
>>> To: harmony-dev@incubator.apache.org
>>> Subject: Re: [DRLVM] General stability
>>>
>>> 2006/11/8, Mikhail Fursov <mi...@gmail.com>:
>>>> On 11/8/06, Alexey Petrenko <al...@gmail.com> wrote:
>>>>> Probably it's time to create some release plan :)
>>>>>
>>>> So let's start this discussion?
>>>> Good idea!
>>>> The only release I can imagine is Harmony Java5SE 100% compatible.
>>> To be Java5SE 100% compatible we need TCK first.
>>> So we could think about some less impressive goal for the first
release
>> :)
>>> SY, Alexey
>>

Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability

[Anton Luht <an...@gmail.com>]

Alexei,

> I like your approach to result comparison. 10% can be default value
> for some form field - anyone can change it if needed.

OK, let's try it and see if it satisfies the community/

> Probably more conventional would be to parse
> <system-out><![CDATA[]]></system-out> for some metric. Does the site
> already parse TEST-* files?

It parses TESTS-TestSuites.xml  - as far as I can see, it contains the
same information as TEST-* files. Is there a common format of output
of stress tests you're talking about?


>
> Thank you, Alexei
>
> On 11/10/06, Anton Luht <an...@gmail.com> wrote:
> > Hello, Alexei,
> >
> > > I have related question. How can we improve http://harmonytest.org to
> > > make it possible to publish not just pass, fail, or error but numeric
> > > test scores?
> >
> > Easily - test results in JUnit reports have 'time' property -
> > execution time in seconds. We can import and show them in the results.
> > What else is needed? Maybe add something like 'show regressions' to
> > the the 'compare runs' page? For example, show tests that increased
> > execution time by more than 10% sorted by increase rate desc?
> >
> > --
> > Regards,
> > Anton Luht,
> > Intel Java & XML Engineering
> >
>
>
> --
> Thank you,
> Alexei
>


-- 
Regards,
Anton Luht,
Intel Java & XML Engineering

Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability

[Alexei Fedotov <al...@gmail.com>]

Anton,

I like your approach to result comparison. 10% can be default value
for some form field - anyone can change it if needed.

As for test execution time reported by JUnit, it is applicable for
stress tests as well if we gradually increase a load over time. Though
using ttime field for stress tests and documentation rankings is not
quite conventional.

Probably more conventional would be to parse
<system-out><![CDATA[]]></system-out> for some metric. Does the site
already parse TEST-* files?

Thank you, Alexei

On 11/10/06, Anton Luht <an...@gmail.com> wrote:
> Hello, Alexei,
>
> > I have related question. How can we improve http://harmonytest.org to
> > make it possible to publish not just pass, fail, or error but numeric
> > test scores?
>
> Easily - test results in JUnit reports have 'time' property -
> execution time in seconds. We can import and show them in the results.
> What else is needed? Maybe add something like 'show regressions' to
> the the 'compare runs' page? For example, show tests that increased
> execution time by more than 10% sorted by increase rate desc?
>
> --
> Regards,
> Anton Luht,
> Intel Java & XML Engineering
>


-- 
Thank you,
Alexei

Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability

[Anton Luht <an...@gmail.com>]

Hello, Alexei,

> I have related question. How can we improve http://harmonytest.org to
> make it possible to publish not just pass, fail, or error but numeric
> test scores?

Easily - test results in JUnit reports have 'time' property -
execution time in seconds. We can import and show them in the results.
What else is needed? Maybe add something like 'show regressions' to
the the 'compare runs' page? For example, show tests that increased
execution time by more than 10% sorted by increase rate desc?

-- 
Regards,
Anton Luht,
Intel Java & XML Engineering