You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Egor Pasko <eg...@gmail.com> on 2006/12/27 12:07:33 UTC

[drlvm][testing][jit] regular testing for JIT regressions on HUT

Folks,

I have a novell for you.

Preface:

today I ran Harmony Unit Tests on pure Jitrino.OPT and pure
Jitrino.JET again to compare. Results are not surprising)) So much to
fix))

Should I wake up regularly to run the tests ang ring the bell?  Maybe. That
makes me think of some automation/improvements to "the process". This item would
be interesting to JIT guys and especially 'build-test' and CC guys.

In brief:

    * JET is more stable than OPT, 24 tests passed on JET, but failed on OPT 
      (see below for the list of failed tests)

    * I encountered some problems running awt tests from command-line
      without ant 
      (wanna help? see below)

    * I am starting the new iteration of OPT fixing on HUT 
      (congratulate below)

    * Got some ideas how JIT bugs can be detected fast/automatically on
      HUT aiming at "no regression" 
      (participate .. below)

In depth:

--------------------------------------------------------------------------------
*24 tests passed on JET, but failed on OPT*
--------------------------------------------------------------------------------
      java.awt.BasicStrokeTest
      java.awt.geom.AffineTransformTest
      java.awt.geom.Arc2DTest
      java.awt.geom.CubicCurve2DTest
      java.awt.geom.Ellipse2DTest
      java.awt.geom.QuadCurve2DTest
      java.awt.geom.RoundRectangle2DTest
      javax.swing.plaf.UIResourceTest
      javax.swing.plaf.metal.MetalFileChooserUITest
      org.apache.harmony.luni.tests.java.lang.DoubleTest
      org.apache.harmony.luni.tests.java.lang.FloatTest
      org.apache.harmony.luni.tests.java.lang.LongTest
      org.apache.harmony.luni.tests.java.lang.MathTest
      org.apache.harmony.luni.tests.java.lang.StrictMathTest
      org.apache.harmony.luni.tests.java.lang.StringBuilderTest
      org.apache.harmony.luni.tests.java.util.ArraysTest
      org.apache.harmony.luni.tests.java.util.UUIDTest
      org.apache.harmony.sound.tests.javax.sound.midi.SequenceTest
      org.apache.harmony.tests.java.math.BigDecimalArithmeticTest
      org.apache.harmony.tests.java.math.BigDecimalCompareTest
      org.apache.harmony.tests.java.math.BigDecimalConstructorsTest
      org.apache.harmony.tests.java.math.BigDecimalScaleOperationsTest
      org.apache.harmony.tests.java.math.BigIntegerConvertTest
      org.apache.harmony.text.tests.java.text.ChoiceFormatTest
each means an near 100% *bug in OPT*, which is most likely not
reproduced on everyday runs, but is enough dangerous

--------------------------------------------------------------------------------
*I encountered some problems running awt tests from command-line*
--------------------------------------------------------------------------------
    To run the big pack of tests I use 'ant test'. To run on some specific
    mode, say -Xem:opt, I put "-Xem:opt" to
    deploy/jdk/jre/bin/default/harmonyvm.properties, but I do not like
    doing it // may influence some other tests running at the same time

    What I am used to do to investigate a failure on a classlib test:

    HARMONY  -Xbootclasspath/p:$classlib/depends/jars/junit_3.8.2/junit.jar:$classlib/deploy/build/test/support.jar:$classlib/modules/luni/bin/test junit.textui.TestRunner org.apache.harmony.luni.tests.java.lang.DoubleTest

    for luni it is OK:
    ..................................
    Time: 0.446

    OK (34 tests)

    for awt the thing is different:
    $HARMONY -Xbootclasspath/p:$classlib/depends/jars/junit_3.8.2/junit.jar:$classlib/deploy/build/test/support.jar:$classlib/modules/awt/bin/test junit.textui.TestRunner java.awt.geom.AffineTransformTest

    .F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.F.
    F.F.F.F.F.F.F.F.F.F.F.F.F.F.F
    Time: 0.023
    There were 55 failures:
    1) warning(junit.framework.TestSuite$1)junit.framework.AssertionFailedError: Exception in constructor: testCreate1 (junit.framework.Assertio
    nFailedError: Variable TEST_SRC_DIR not defined
            at java.awt.SerializeTestCase.getSerializePath(SerializeTestCase.java:46)
            at java.awt.geom.AffineTransformTest.<init>(AffineTransformTest.java:105)
            at java.lang.reflect.VMReflection.newClassInstance(Native Method)
            at java.lang.reflect.Constructor.newInstance(Unknown Source)
            at junit.runner.BaseTestRunner.getTest(BaseTestRunner.java:118)
    (... and a lot of stuff like that)

    what should I set to get rid of "Variable TEST_SRC_DIR not defined"?

--------------------------------------------------------------------------------
*I am starting the new iteration of OPT fixing on HUT*
--------------------------------------------------------------------------------

    It is more fun to fix this crap together. George? Mikhail Fursov? Pavel
    Ozhdikhin?

    I am too lazy to report them officially now. Some tests are obviously
    failing due to the same reason. I will be reporting a test to JIRA and
    [1] as soon as I start investigating. Suggestions welcome.

--------------------------------------------------------------------------------
*Got some ideas how JIT bugs can be detected*
--------------------------------------------------------------------------------

    Knowing bugs sooner is obviously better. Negative power of a bug grows
    exponentially over time. To detect bugs faster we have pre-commit
    tests, CC runs, extending/excluding activity is going on etc. Which is
    good.

    Bad news for Jitrino.OPT is that it is lacking regular testing with
    tests we already have. Most of the tests run in default mode, which is
    not OPT-active, JET is used most of the time.

    Good news for Jitrino.OPT is that many of it's bugs can be detected easily
    and almost 100% detected as OPT-specific *automatically*. To detect if a bug
    is OPT-specific a script can:
    * run all tests (for example, HUT) on JET
    * run the same on OPT
    * filter out intermittent failures (by rerunning failed tests several times)
    * if some test fails on OPT and passes on JET, report it

    that is almost 100% Jitrino.OPT bug searcher, which is a promising tool to
    detect Jitrino.OPT bugs early and *automatically*.

    Opinions?

-- 
Egor Pasko


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Egor Pasko <eg...@gmail.com>.
On the 0x24D day of Apache Harmony Alexey Varlamov wrote:
> 2006/12/28, Alexander Kleymenov <kl...@gmail.com>:
> > > BTW, Mikhail, are we going to integrate IR-level JIT regression tests
> > > ([1]) into the *DRLVM regression testing infrastructure* in the
> > > nearest future?
> >
> > Stop, guys, let's think. Regression test is a test checking for
> > particular problem reported and fixed before. So regression test and
> > JIRA report have one-to-one correspondence. What I see at HARMONY-1531
> > is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
> > So, I think, these tests should go to different place, not to regression
> > test suite. Right?
> 
> Alexander,
> I'm not expert in testing taxonomy, could you please share the full
> picture - which tests go where and when/how intended to be run, etc?
> 
> > > Are there any pending tasks to reach that point? IMHO, we should
> > > integrate/commit it and let go for constant improvement. That's quite
> > > enough for the start. (Not the highest priority, but gonnabe pretty
> > > useful). Please, tell me if I am missing something here.
> >
> > I'm agree with you. Tests first! Any tests increase the quality
> > of the product (if they do not do some intentional harm of course).
> 
> I got an impression that great diversity of test suites may come into
> conflict with this manifested goal - more frameworks to maintain, more
> steps to launch/track various tests, scattered reports, more complex
> desicion tree to add new tests, etc.
> Leaving aside official certification tests, I personally do not really
> care where from a test case is taken - from spec perseption of code
> implementer or from real-life user problem, the essence point is that
> it properly exercises particular bit of functionality. And further
> grading IMO should be practical tradeoff between
> importance/coverage/time to test, dividing pre-commit/pre-contribute
> and nightly or weekly CC runs.
> Ideally there should be just one entry point which takes in a
> config/testbase to be tested and a single consolidated test status and
> report.

"one entry point" for all tests sounds good

-- 
Egor Pasko


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Alexey Varlamov <al...@gmail.com>.
2006/12/28, Alexander Kleymenov <kl...@gmail.com>:
> > BTW, Mikhail, are we going to integrate IR-level JIT regression tests
> > ([1]) into the *DRLVM regression testing infrastructure* in the
> > nearest future?
>
> Stop, guys, let's think. Regression test is a test checking for
> particular problem reported and fixed before. So regression test and
> JIRA report have one-to-one correspondence. What I see at HARMONY-1531
> is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
> So, I think, these tests should go to different place, not to regression
> test suite. Right?

Alexander,
I'm not expert in testing taxonomy, could you please share the full
picture - which tests go where and when/how intended to be run, etc?

> > Are there any pending tasks to reach that point? IMHO, we should
> > integrate/commit it and let go for constant improvement. That's quite
> > enough for the start. (Not the highest priority, but gonnabe pretty
> > useful). Please, tell me if I am missing something here.
>
> I'm agree with you. Tests first! Any tests increase the quality
> of the product (if they do not do some intentional harm of course).

I got an impression that great diversity of test suites may come into
conflict with this manifested goal - more frameworks to maintain, more
steps to launch/track various tests, scattered reports, more complex
desicion tree to add new tests, etc.
Leaving aside official certification tests, I personally do not really
care where from a test case is taken - from spec perseption of code
implementer or from real-life user problem, the essence point is that
it properly exercises particular bit of functionality. And further
grading IMO should be practical tradeoff between
importance/coverage/time to test, dividing pre-commit/pre-contribute
and nightly or weekly CC runs.
Ideally there should be just one entry point which takes in a
config/testbase to be tested and a single consolidated test status and
report.

--
Alexey

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Tim Ellison <t....@gmail.com>.
Alexander Kleymenov wrote:
> Tim,
> 
> On 12/28/06, Tim Ellison <t....@gmail.com> wrote:
>> Alexander Kleymenov wrote:
>> > Stop, guys, let's think. Regression test is a test checking for
>> > particular problem reported and fixed before. So regression test and
>> > JIRA report have one-to-one correspondence. What I see at HARMONY-1531
>> > is a set of tests for Jitrino.OPT marked as "improvement", not as a
>> bug.
>> > So, I think, these tests should go to different place, not to
>> regression
>> > test suite. Right?
>>
>> While I understand what a regression test is, can you help me understand
>> why you need to treat them differently to the functional unit tests
>> written by developers when the originally authored the code?  i.e. they
>> all still need to pass, what do you use the regression classification
>> to do?
> 
> Regression test suite keeps us from going back to resolved problems,
> from quality degradation. It insures us to be on achieved quality level.
> While other test suites insures product works as expected by developer.
> 
> Sometimes it is needed to be sure that some fix does not roll to back
> the product. It could be easily and quickly done by running the
> regression test suite. And we don't have to run all the tests – it just
> could take a long time.
> 
> So with separate regression test suite we can perform fast pre-commit
> testing, and/or do ongoing automated code integrity testing (each 2
> hours for example).
> 
> In our case it is possible to include vm regression tests into CI CC
> right now. Other test packages will be included into another CCs.

I know what regression tests are, I'm trying to understand why you want
keep them separate.  So it's a time thing ... i.e. you would run all the
tests if you had the time?

Tim

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Alexander Kleymenov <kl...@gmail.com>.
Tim,

On 12/28/06, Tim Ellison <t....@gmail.com> wrote:
> Alexander Kleymenov wrote:
> > Stop, guys, let's think. Regression test is a test checking for
> > particular problem reported and fixed before. So regression test and
> > JIRA report have one-to-one correspondence. What I see at HARMONY-1531
> > is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
> > So, I think, these tests should go to different place, not to regression
> > test suite. Right?
>
> While I understand what a regression test is, can you help me understand
> why you need to treat them differently to the functional unit tests
> written by developers when the originally authored the code?  i.e. they
> all still need to pass, what do you use the regression classification to do?

Regression test suite keeps us from going back to resolved problems,
from quality degradation. It insures us to be on achieved quality level.
While other test suites insures product works as expected by developer.

Sometimes it is needed to be sure that some fix does not roll to back
the product. It could be easily and quickly done by running the
regression test suite. And we don't have to run all the tests – it just
could take a long time.

So with separate regression test suite we can perform fast pre-commit
testing, and/or do ongoing automated code integrity testing (each 2
hours for example).

In our case it is possible to include vm regression tests into CI CC
right now. Other test packages will be included into another CCs.

Thanks,
Alexander

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Tim Ellison <t....@gmail.com>.
Alexander Kleymenov wrote:
> Stop, guys, let's think. Regression test is a test checking for
> particular problem reported and fixed before. So regression test and
> JIRA report have one-to-one correspondence. What I see at HARMONY-1531
> is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
> So, I think, these tests should go to different place, not to regression
> test suite. Right?

While I understand what a regression test is, can you help me understand
why you need to treat them differently to the functional unit tests
written by developers when the originally authored the code?  i.e. they
all still need to pass, what do you use the regression classification to do?

Regards,
Tim


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Mikhail Fursov <mi...@gmail.com>.
All,
HARMONY-1531 was frozed for some period of time because
1) I was busy with much higher priority tasks (wait for a set of new JIRA
issues from me after holidays:) )
2) Noone from JIT gurus was interested/had a time to help

My plans for the the next month is to add unrolling optimization to Jitrino
(I have already wrote some code) and to port magics support for EM64T and
IPF.
I'm going to use internal Jitrino testing framework to test loop unrolling
code. I'm not sure that my efforts will be enough to integrate it. So the
main answer for the question about H1531 readiness: it will be ready as soon
as other JIT developers start using it.

On 28 Dec 2006 20:23:41 +0600, Egor Pasko <eg...@gmail.com> wrote:
>
> Well, actually, JIT IR-level tests are C-based too. Then, probably,
> c-unit is the best place?


The JIRA already contains JUnit integration for JIT internal tests. You can
add Cunit integration and put it into any test suite you want. I have no
preferences here.

I think, it makes sense to combine *all* types of tests in a single
> framefork with unified top-level running mechanisms as well as
> reporting mechanisms. When we have them all together we can cut them
> in pieces based on importance/time-to-run/catching-rate. Some pieces
> to run as pre-commit some in CC, some even more rare on snapshots, etc.
>
> > Now, that leaves smoke, kernel and regression.
> >
> > Smoke and kernel should be able to be combined.  Smoke just seems to
> > be tests where someone wanted to write their own testing framework.
>
> I like that
>
> > Regresssion -  I do understand where Tim is coming from in later
> > messages in this thread, but I don't actually mind slapping tests
> > that show reported bugs into a separate category called "regression",
> > because they probably are more complicated than unit tests or even
> > general functional tests, and a maybe poke at weird corner cases or
> > effects of multiple interacting systems.  It would be interesting to
> > see if our regression tests are finding things that unit tests should
> > have...
> >
> > geir
> >
>
> --
> Egor Pasko
>
>


-- 
Mikhail Fursov

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Egor Pasko <eg...@gmail.com>.
On the 0x24D day of Apache Harmony Geir Magnusson, Jr. wrote:
> On Dec 28, 2006, at 5:31 AM, Egor Pasko wrote:
> 
> >
> >> So, I think, these tests should go to different place, not to
> >> regression test suite. Right?
> >
> > Good point.
> >
> > Initially the intention was to call it JIT Unit Tests. But what can
> > stop us from tracking regressions with tests based on this idea?
> >
> > Let's look at the variants where to include the framework:
> > 1. smoke
> > 2. c-unit
> > 3. kernel
> > 4. regression
> > 5. harmony unit
> > 6. forgot something?
> > So, for the moment (4) seems most natural. Maybe, some tests to
> > include into (2) and some to (4). I do not insist, just want to know
> > your opinion for now.
> 
> TO be clear, 5 is "classlibrary tests", which I think should remain
> different.  

yep. I just tried to recall all test suites we have (and show that
it's pretty much for my small small brain organ)

> c-unit are also different, because it's a totally
> different testing framework (c-based). 

Well, actually, JIT IR-level tests are C-based too. Then, probably,
c-unit is the best place?

I think, it makes sense to combine *all* types of tests in a single
framefork with unified top-level running mechanisms as well as
reporting mechanisms. When we have them all together we can cut them
in pieces based on importance/time-to-run/catching-rate. Some pieces
to run as pre-commit some in CC, some even more rare on snapshots, etc.

> Now, that leaves smoke, kernel and regression.
> 
> Smoke and kernel should be able to be combined.  Smoke just seems to
> be tests where someone wanted to write their own testing framework.

I like that

> Regresssion -  I do understand where Tim is coming from in later
> messages in this thread, but I don't actually mind slapping tests
> that show reported bugs into a separate category called "regression",
> because they probably are more complicated than unit tests or even
> general functional tests, and a maybe poke at weird corner cases or
> effects of multiple interacting systems.  It would be interesting to
> see if our regression tests are finding things that unit tests should
> have...
> 
> geir
> 

-- 
Egor Pasko


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Dec 28, 2006, at 5:31 AM, Egor Pasko wrote:

>
>> So, I think, these tests should go to different place, not to
>> regression test suite. Right?
>
> Good point.
>
> Initially the intention was to call it JIT Unit Tests. But what can
> stop us from tracking regressions with tests based on this idea?
>
> Let's look at the variants where to include the framework:
> 1. smoke
> 2. c-unit
> 3. kernel
> 4. regression
> 5. harmony unit
> 6. forgot something?
> So, for the moment (4) seems most natural. Maybe, some tests to
> include into (2) and some to (4). I do not insist, just want to know
> your opinion for now.

TO be clear, 5 is "classlibrary tests", which I think should remain  
different.  c-unit are also different, because it's a totally  
different testing framework (c-based).  Now, that leaves smoke,  
kernel and regression.

Smoke and kernel should be able to be combined.  Smoke just seems to  
be tests where someone wanted to write their own testing framework.

Regresssion -  I do understand where Tim is coming from in later  
messages in this thread, but I don't actually mind slapping tests  
that show reported bugs into a separate category called "regression",  
because they probably are more complicated than unit tests or even  
general functional tests, and a maybe poke at weird corner cases or  
effects of multiple interacting systems.  It would be interesting to  
see if our regression tests are finding things that unit tests should  
have...

geir

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Egor Pasko <eg...@gmail.com>.
On the 0x24D day of Apache Harmony Alexander Kleymenov wrote:
> Hello,
> 
> > OK))
> >
> > BTW, Mikhail, are we going to integrate IR-level JIT regression tests
> > ([1]) into the *DRLVM regression testing infrastructure* in the
> > nearest future?
> 
> Stop, guys, let's think. Regression test is a test checking for
> particular problem reported and fixed before. So regression test and
> JIRA report have one-to-one correspondence. 

Agreed.

> What I see at HARMONY-1531 is a set of tests for Jitrino.OPT marked
> as "improvement", not as a bug. 

yes, it is an improvement *that allows* to run special JIT IR-level
tests to run. Basic idea came from the fact that some JIT bugs can NOT
be reproduced by a simple Java test, we have Jasmin to write more
JIT-oriented tests, but many issues can NOT be covered by Jasmin
coding too. For example, quality of Register Allocator and such.

We would like to have more fine-tuned tests that create certain IR
(=Intermediate Representation) sources and feed JIT with them via a
special interface, then check, how JIT optimized it.

That's a technical detail. Now back to politics:

> So, I think, these tests should go to different place, not to
> regression test suite. Right?

Good point.

Initially the intention was to call it JIT Unit Tests. But what can
stop us from tracking regressions with tests based on this idea?

Let's look at the variants where to include the framework:
1. smoke
2. c-unit
3. kernel
4. regression
5. harmony unit
6. forgot something?
So, for the moment (4) seems most natural. Maybe, some tests to
include into (2) and some to (4). I do not insist, just want to know
your opinion for now.

ps: gosh, what a lots of test suites! how canitbe?

> > Are there any pending tasks to reach that point? IMHO, we should
> > integrate/commit it and let go for constant improvement. That's quite
> > enough for the start. (Not the highest priority, but gonnabe pretty
> > useful). Please, tell me if I am missing something here.
> 
> I'm agree with you. Tests first! Any tests increase the quality
> of the product (if they do not do some intentional harm of course).
> 
> > Alexander Kleymenov, are you interested? BTW, do you have a
> > prioritized list of TODO items for DRLVM RTI?
> 
> No, I don't have such. I see some further works on reg test suite (such as
> old reg tests converting/integration), but I did not make any plans and
> prioritizations. I think we should have public list of such TODOs so
> everyone interesting could help us with it. Your thoughts?

Yeah, knowing the prioritized list is very important. Let's create it
and publish somewhere, say, on wiki to track things in a better way.

What I see immediately:
* exclude lists
* integration of HARMONY-1531 // we are discussing it
* integration to other test suites 
  (same report engine and such) // need to discuss
* what I missed

-- 
Egor Pasko


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Alexander Kleymenov <kl...@gmail.com>.
Hello,

> OK))
>
> BTW, Mikhail, are we going to integrate IR-level JIT regression tests
> ([1]) into the *DRLVM regression testing infrastructure* in the
> nearest future?

Stop, guys, let's think. Regression test is a test checking for
particular problem reported and fixed before. So regression test and
JIRA report have one-to-one correspondence. What I see at HARMONY-1531
is a set of tests for Jitrino.OPT marked as "improvement", not as a bug.
So, I think, these tests should go to different place, not to regression
test suite. Right?

> Are there any pending tasks to reach that point? IMHO, we should
> integrate/commit it and let go for constant improvement. That's quite
> enough for the start. (Not the highest priority, but gonnabe pretty
> useful). Please, tell me if I am missing something here.

I'm agree with you. Tests first! Any tests increase the quality
of the product (if they do not do some intentional harm of course).

> Alexander Kleymenov, are you interested? BTW, do you have a
> prioritized list of TODO items for DRLVM RTI?

No, I don't have such. I see some further works on reg test suite (such as
old reg tests converting/integration), but I did not make any plans and
prioritizations. I think we should have public list of such TODOs so
everyone interesting could help us with it. Your thoughts?

Thanks,
Alexander

Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Egor Pasko <eg...@gmail.com>.
On the 0x24C day of Apache Harmony Mikhail Fursov wrote:
> On 27 Dec 2006 17:07:33 +0600, Egor Pasko <eg...@gmail.com> wrote:
> >
> >     Opinions?
> 
> 
> Let's fix them all :)

OK))

BTW, Mikhail, are we going to integrate IR-level JIT regression tests
([1]) into the *DRLVM regression testing infrastructure* in the
nearest future?

Are there any pending tasks to reach that point? IMHO, we should
integrate/commit it and let go for constant improvement. That's quite
enough for the start. (Not the highest priority, but gonnabe pretty
useful). Please, tell me if I am missing something here.

Alexander Kleymenov, are you interested? BTW, do you have a
prioritized list of TODO items for DRLVM RTI?

[1] [drlvm][jit] Internal testing framework for Jitrino.OPT compiler
https://issues.apache.org/jira/browse/HARMONY-1531

-- 
Egor Pasko


Re: [drlvm][testing][jit] regular testing for JIT regressions on HUT

Posted by Mikhail Fursov <mi...@gmail.com>.
On 27 Dec 2006 17:07:33 +0600, Egor Pasko <eg...@gmail.com> wrote:
>
>     Opinions?


Let's fix them all :)

-- 
Mikhail Fursov