You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Anton Luht <an...@gmail.com> on 2006/11/10 08:44:20 UTC
Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability
Hello, Alexei,
> I have related question. How can we improve http://harmonytest.org to
> make it possible to publish not just pass, fail, or error but numeric
> test scores?
Easily - test results in JUnit reports have 'time' property -
execution time in seconds. We can import and show them in the results.
What else is needed? Maybe add something like 'show regressions' to
the the 'compare runs' page? For example, show tests that increased
execution time by more than 10% sorted by increase rate desc?
--
Regards,
Anton Luht,
Intel Java & XML Engineering
Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability
Posted by Anton Luht <an...@gmail.com>.
Alexei,
> I like your approach to result comparison. 10% can be default value
> for some form field - anyone can change it if needed.
OK, let's try it and see if it satisfies the community/
> Probably more conventional would be to parse
> <system-out><![CDATA[]]></system-out> for some metric. Does the site
> already parse TEST-* files?
It parses TESTS-TestSuites.xml - as far as I can see, it contains the
same information as TEST-* files. Is there a common format of output
of stress tests you're talking about?
>
> Thank you, Alexei
>
> On 11/10/06, Anton Luht <an...@gmail.com> wrote:
> > Hello, Alexei,
> >
> > > I have related question. How can we improve http://harmonytest.org to
> > > make it possible to publish not just pass, fail, or error but numeric
> > > test scores?
> >
> > Easily - test results in JUnit reports have 'time' property -
> > execution time in seconds. We can import and show them in the results.
> > What else is needed? Maybe add something like 'show regressions' to
> > the the 'compare runs' page? For example, show tests that increased
> > execution time by more than 10% sorted by increase rate desc?
> >
> > --
> > Regards,
> > Anton Luht,
> > Intel Java & XML Engineering
> >
>
>
> --
> Thank you,
> Alexei
>
--
Regards,
Anton Luht,
Intel Java & XML Engineering
Re: [testing] Tests scores on http://harmonytest.org Was: [DRLVM] General stability
Posted by Alexei Fedotov <al...@gmail.com>.
Anton,
I like your approach to result comparison. 10% can be default value
for some form field - anyone can change it if needed.
As for test execution time reported by JUnit, it is applicable for
stress tests as well if we gradually increase a load over time. Though
using ttime field for stress tests and documentation rankings is not
quite conventional.
Probably more conventional would be to parse
<system-out><![CDATA[]]></system-out> for some metric. Does the site
already parse TEST-* files?
Thank you, Alexei
On 11/10/06, Anton Luht <an...@gmail.com> wrote:
> Hello, Alexei,
>
> > I have related question. How can we improve http://harmonytest.org to
> > make it possible to publish not just pass, fail, or error but numeric
> > test scores?
>
> Easily - test results in JUnit reports have 'time' property -
> execution time in seconds. We can import and show them in the results.
> What else is needed? Maybe add something like 'show regressions' to
> the the 'compare runs' page? For example, show tests that increased
> execution time by more than 10% sorted by increase rate desc?
>
> --
> Regards,
> Anton Luht,
> Intel Java & XML Engineering
>
--
Thank you,
Alexei