You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by Gregory Shimansky <gs...@gmail.com> on 2006/11/21 00:39:49 UTC
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Geir Magnusson Jr. wrote:
>
>
> Alexey Varlamov wrote:
>> Folks,
>>
>> I've just did a little step in improving pre-commit testing for DRLVM;
>> most important change is a move from "perTest" forking mode to "once"
>> (aka sameVM mode).
>> This reduces testing time drastically (~50%), but may introduce some
>> extra instability (like new intermittent failures or timeouts).
>
> Plus the inability to figure out what's screwing things up.
>
> I can't decide if I like this. On one hand, I like it because it's
> actually better to find side-effects - some test can pass, but still
> leave the VM in a broken state that another test will show.
I think the change has a good intension but it should've gone through
more testing. Preferably the fixes for problems on windows should've
been included into the patch. If the problems are too hard to fix, why
not open a discussion before changing the files in SVN?
> OTOH, it does remove the clarity of each test being a single, separate
> test. It conceptually mixes integration testing with unit testing.
>
> Can you please just add a switch? That way in the event of a failure,
> we can re-run with forking on, and therefore can tell if the crash is
> specifically due to the test that is crashing, or a side effect caused
> by something that came before.
I would also like a switch. I don't like it that a change which
knowingly introduces problems with acceptance tests is committed without
discussion. Should we discard kernel tests from drlvm commits until they
are fixed? I think that there was an agreement about no regression. So
now all commits to drlvm are blocked.
--
Gregory
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/21, Alexei Fedotov <al...@gmail.com>:
> Hello,
> If a developer runs all tests in one VM and they pass, that's great
> and he can commit. Can developer make a commit if tests pass in
> "perTest" mode and fail in "once" mode?
>
> Let's imagine following scenarios of cooperative development when
> pre-commit test suite demonstrates unpredictable behavior (such as
> "once" mode").
>
> Scenario A.
> 1. A developer runs tests one time, and tests failed.
> 2. He runs them again and tests passes.
> 3. He commit changes.
> Why do we need to exercise developers' patience? Simple tests are
> predictable - they either pass or fail.
>
> Scenario B.
> 1. Developer A pass tests and commit his changes.
> 2. Developer B checks out a fresh workspace, makes changes and runs
> tests - they failed.
> 3. He digs out the problem to the stage when he runs tests for the
> clean workspace - tests fail since he has slightly different
> configuration.
> Developer B suspects developer A to run tests multiple times, or skip
> acceptance test run at all. This destroys mutual trust. This makes me
> think that the simpler pre-commit tests are, the better.
So in extreme case, no testing at all is the best! ;)
Seriously, kernel tests are simple enough unit tests - if we cannot
even trust "sameVM" runs here, do we ever need to invent complex
testing scenarios involving big apps?
Sure, we can reverse the default to "perTest" mode, but won't you feel
deeply sorry for time wasted in such testing? Just to remind, tests
for debug version take up to 2h on Windows server.
>
> Thanks!
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Alexei Fedotov wrote:
> Hello,
> If a developer runs all tests in one VM and they pass, that's great
> and he can commit. Can developer make a commit if tests pass in
> "perTest" mode and fail in "once" mode?
>
> Let's imagine following scenarios of cooperative development when
> pre-commit test suite demonstrates unpredictable behavior (such as
> "once" mode").
>
> Scenario A.
> 1. A developer runs tests one time, and tests failed.
> 2. He runs them again and tests passes.
> 3. He commit changes.
> Why do we need to exercise developers' patience? Simple tests are
> predictable - they either pass or fail.
Well, no. If we've achieved stability, and are really being truthful
about "no regression", that committer shouldn't commit that. "Being
lucky" isn't a reason to commit changes.
>
> Scenario B.
> 1. Developer A pass tests and commit his changes.
> 2. Developer B checks out a fresh workspace, makes changes and runs
> tests - they failed.
> 3. He digs out the problem to the stage when he runs tests for the
> clean workspace - tests fail since he has slightly different
> configuration.
> Developer B suspects developer A to run tests multiple times, or skip
> acceptance test run at all. This destroys mutual trust. This makes me
> think that the simpler pre-commit tests are, the better.
I don't understand what simplifying the tests achieve. If developer B
has a problem, he or she should ask developer A about it.
>
> Thanks!
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexei Fedotov <al...@gmail.com>.
Hello,
If a developer runs all tests in one VM and they pass, that's great
and he can commit. Can developer make a commit if tests pass in
"perTest" mode and fail in "once" mode?
Let's imagine following scenarios of cooperative development when
pre-commit test suite demonstrates unpredictable behavior (such as
"once" mode").
Scenario A.
1. A developer runs tests one time, and tests failed.
2. He runs them again and tests passes.
3. He commit changes.
Why do we need to exercise developers' patience? Simple tests are
predictable - they either pass or fail.
Scenario B.
1. Developer A pass tests and commit his changes.
2. Developer B checks out a fresh workspace, makes changes and runs
tests - they failed.
3. He digs out the problem to the stage when he runs tests for the
clean workspace - tests fail since he has slightly different
configuration.
Developer B suspects developer A to run tests multiple times, or skip
acceptance test run at all. This destroys mutual trust. This makes me
think that the simpler pre-commit tests are, the better.
Thanks!
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Alexey Varlamov wrote:
> 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
>> Sorry - yes, you are right.
>>
>> Other things :
>>
>> 1) When running on x86_64 (where we have bugs), I noticed that the
>> kernel tests don't run to completion on failure. How can we get it to
>> do that? It would be nice to figure out what all the failures are.
>> Failfast is set to off. Maybe I have to run forkmode as whatever the
>> other value is. what is the other value?
> It must be VM crash, it may die silently and do not finish full suite.
> If it complained about invalid xml file while generating report, means
> that corresponding test killed VM.
> Please try with -Dkernel.test.forkmode=perTest to get whole picture.
"preTest" was what I was fishing for. I added a little bit of
documentation to the test.properties file.
>
>>
>> 2) Would it be hard to have an overall test report? Like :
>>
>> JVMTI (jit) - PASSED
>> JVMTI (int) - PASSED
>> c-unit - PASSED
>> smoke (jit) - PASSED
>> smoke (int) - PASSED
>> kernel (jit) - PASSED
>> kernel (opt) - PASSED
>> kernel (int) - PASSED
>>
>>
>> Should be easy - maybe we just echo the result of each stage to a file
>> and print the result. I'll poke at that.
> Yes, please :) Actually this is exactly how kernel.test summary works.
Ok - that will give people a better sense of what's going on, I think
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
> Sorry - yes, you are right.
>
> Other things :
>
> 1) When running on x86_64 (where we have bugs), I noticed that the
> kernel tests don't run to completion on failure. How can we get it to
> do that? It would be nice to figure out what all the failures are.
> Failfast is set to off. Maybe I have to run forkmode as whatever the
> other value is. what is the other value?
It must be VM crash, it may die silently and do not finish full suite.
If it complained about invalid xml file while generating report, means
that corresponding test killed VM.
Please try with -Dkernel.test.forkmode=perTest to get whole picture.
>
> 2) Would it be hard to have an overall test report? Like :
>
> JVMTI (jit) - PASSED
> JVMTI (int) - PASSED
> c-unit - PASSED
> smoke (jit) - PASSED
> smoke (int) - PASSED
> kernel (jit) - PASSED
> kernel (opt) - PASSED
> kernel (int) - PASSED
>
>
> Should be easy - maybe we just echo the result of each stage to a file
> and print the result. I'll poke at that.
Yes, please :) Actually this is exactly how kernel.test summary works.
>
> geir
>
> Alexey Varlamov wrote:
> > 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
> >> I just ran full test suite on ubuntu 6 x86, after goosing the timeout
> >> number by an order of magnitude (and committing it..)
> >>
> >> But...
> >>
> >> where did the jvmti tests go?
> >>
> >> They should run by default as well
> >
> > They do run, the first in order - so may be just scrolled high in console.
> >
> >>
> >> geir
> >>
> >>
> >> Alexey Varlamov wrote:
> >> > 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
> >> >>
> >> >>
> >> >> Geir Magnusson Jr. wrote:
> >> >> >
> >> >> >
> >> >> > Alexey Varlamov wrote:
> >> >> >> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> >> >> >>> Geir Magnusson Jr. wrote:
> >> >>
> >> >> >
> >> >> >> Generally I agree it is always better to discuss, and in fact
> >> >> >> hesitated to commit with default forking set to "once" - but
> >> decided
> >> >> >> this is right thing to do rigth now. The current trend to make
> >> drlvm
> >> >> >> truly standard VM for harmony, recently Alexei even proclaimed 100%
> >> >> >> HUT pass rate, neverthless drlvm is not yet reliable enough and we
> >> >> >> should keep the momentum and face possible problems with open eyes.
> >> >> >> Again, I'm sorry to bother you^H^H^H us VM guys, but believe
> >> this is
> >> >> >> neccessary evil now.
> >> >> >
> >> >> > The nicer way to do this is to introduce this change as an
> >> option, and
> >> >> > let people start using it...
> >> >> >
> >> >>
> >> >> So it doesn't appear that I'm beating up on you :) I want to say
> >> that i
> >> >> do like some of these changes. very much, actually, such as the test
> >> >> properties, and the summary info. And the fact that we have that
> >> switch
> >> >> and you just forgot to mention it... is great.
> >> >>
> >> >> It's just good to announce to people, to give them a heads-up and a
> >> >> chance to respond.
> >> >>
> >> >> It's also easier if you describe the changes in prose, rather than
> >> hope
> >> >> someone has the time and wherewithal to grok the svn diff.
> >> >
> >> > Thank you Geir! And thanks to all who cared and dared to speak up.
> >> > I see and I learn a lot.
> >> >
> >> > --
> >> > Alexey
> >> >
> >> >>
> >> >>
> >> >> geir
> >> >>
> >> >
> >>
> >
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Sorry - yes, you are right.
Other things :
1) When running on x86_64 (where we have bugs), I noticed that the
kernel tests don't run to completion on failure. How can we get it to
do that? It would be nice to figure out what all the failures are.
Failfast is set to off. Maybe I have to run forkmode as whatever the
other value is. what is the other value?
2) Would it be hard to have an overall test report? Like :
JVMTI (jit) - PASSED
JVMTI (int) - PASSED
c-unit - PASSED
smoke (jit) - PASSED
smoke (int) - PASSED
kernel (jit) - PASSED
kernel (opt) - PASSED
kernel (int) - PASSED
Should be easy - maybe we just echo the result of each stage to a file
and print the result. I'll poke at that.
geir
Alexey Varlamov wrote:
> 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
>> I just ran full test suite on ubuntu 6 x86, after goosing the timeout
>> number by an order of magnitude (and committing it..)
>>
>> But...
>>
>> where did the jvmti tests go?
>>
>> They should run by default as well
>
> They do run, the first in order - so may be just scrolled high in console.
>
>>
>> geir
>>
>>
>> Alexey Varlamov wrote:
>> > 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
>> >>
>> >>
>> >> Geir Magnusson Jr. wrote:
>> >> >
>> >> >
>> >> > Alexey Varlamov wrote:
>> >> >> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
>> >> >>> Geir Magnusson Jr. wrote:
>> >>
>> >> >
>> >> >> Generally I agree it is always better to discuss, and in fact
>> >> >> hesitated to commit with default forking set to "once" - but
>> decided
>> >> >> this is right thing to do rigth now. The current trend to make
>> drlvm
>> >> >> truly standard VM for harmony, recently Alexei even proclaimed 100%
>> >> >> HUT pass rate, neverthless drlvm is not yet reliable enough and we
>> >> >> should keep the momentum and face possible problems with open eyes.
>> >> >> Again, I'm sorry to bother you^H^H^H us VM guys, but believe
>> this is
>> >> >> neccessary evil now.
>> >> >
>> >> > The nicer way to do this is to introduce this change as an
>> option, and
>> >> > let people start using it...
>> >> >
>> >>
>> >> So it doesn't appear that I'm beating up on you :) I want to say
>> that i
>> >> do like some of these changes. very much, actually, such as the test
>> >> properties, and the summary info. And the fact that we have that
>> switch
>> >> and you just forgot to mention it... is great.
>> >>
>> >> It's just good to announce to people, to give them a heads-up and a
>> >> chance to respond.
>> >>
>> >> It's also easier if you describe the changes in prose, rather than
>> hope
>> >> someone has the time and wherewithal to grok the svn diff.
>> >
>> > Thank you Geir! And thanks to all who cared and dared to speak up.
>> > I see and I learn a lot.
>> >
>> > --
>> > Alexey
>> >
>> >>
>> >>
>> >> geir
>> >>
>> >
>>
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
> I just ran full test suite on ubuntu 6 x86, after goosing the timeout
> number by an order of magnitude (and committing it..)
>
> But...
>
> where did the jvmti tests go?
>
> They should run by default as well
They do run, the first in order - so may be just scrolled high in console.
>
> geir
>
>
> Alexey Varlamov wrote:
> > 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
> >>
> >>
> >> Geir Magnusson Jr. wrote:
> >> >
> >> >
> >> > Alexey Varlamov wrote:
> >> >> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> >> >>> Geir Magnusson Jr. wrote:
> >>
> >> >
> >> >> Generally I agree it is always better to discuss, and in fact
> >> >> hesitated to commit with default forking set to "once" - but decided
> >> >> this is right thing to do rigth now. The current trend to make drlvm
> >> >> truly standard VM for harmony, recently Alexei even proclaimed 100%
> >> >> HUT pass rate, neverthless drlvm is not yet reliable enough and we
> >> >> should keep the momentum and face possible problems with open eyes.
> >> >> Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
> >> >> neccessary evil now.
> >> >
> >> > The nicer way to do this is to introduce this change as an option, and
> >> > let people start using it...
> >> >
> >>
> >> So it doesn't appear that I'm beating up on you :) I want to say that i
> >> do like some of these changes. very much, actually, such as the test
> >> properties, and the summary info. And the fact that we have that switch
> >> and you just forgot to mention it... is great.
> >>
> >> It's just good to announce to people, to give them a heads-up and a
> >> chance to respond.
> >>
> >> It's also easier if you describe the changes in prose, rather than hope
> >> someone has the time and wherewithal to grok the svn diff.
> >
> > Thank you Geir! And thanks to all who cared and dared to speak up.
> > I see and I learn a lot.
> >
> > --
> > Alexey
> >
> >>
> >>
> >> geir
> >>
> >
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
I just ran full test suite on ubuntu 6 x86, after goosing the timeout
number by an order of magnitude (and committing it..)
But...
where did the jvmti tests go?
They should run by default as well
geir
Alexey Varlamov wrote:
> 2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
>>
>>
>> Geir Magnusson Jr. wrote:
>> >
>> >
>> > Alexey Varlamov wrote:
>> >> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
>> >>> Geir Magnusson Jr. wrote:
>>
>> >
>> >> Generally I agree it is always better to discuss, and in fact
>> >> hesitated to commit with default forking set to "once" - but decided
>> >> this is right thing to do rigth now. The current trend to make drlvm
>> >> truly standard VM for harmony, recently Alexei even proclaimed 100%
>> >> HUT pass rate, neverthless drlvm is not yet reliable enough and we
>> >> should keep the momentum and face possible problems with open eyes.
>> >> Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
>> >> neccessary evil now.
>> >
>> > The nicer way to do this is to introduce this change as an option, and
>> > let people start using it...
>> >
>>
>> So it doesn't appear that I'm beating up on you :) I want to say that i
>> do like some of these changes. very much, actually, such as the test
>> properties, and the summary info. And the fact that we have that switch
>> and you just forgot to mention it... is great.
>>
>> It's just good to announce to people, to give them a heads-up and a
>> chance to respond.
>>
>> It's also easier if you describe the changes in prose, rather than hope
>> someone has the time and wherewithal to grok the svn diff.
>
> Thank you Geir! And thanks to all who cared and dared to speak up.
> I see and I learn a lot.
>
> --
> Alexey
>
>>
>>
>> geir
>>
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/21, Geir Magnusson Jr. <ge...@pobox.com>:
>
>
> Geir Magnusson Jr. wrote:
> >
> >
> > Alexey Varlamov wrote:
> >> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> >>> Geir Magnusson Jr. wrote:
>
> >
> >> Generally I agree it is always better to discuss, and in fact
> >> hesitated to commit with default forking set to "once" - but decided
> >> this is right thing to do rigth now. The current trend to make drlvm
> >> truly standard VM for harmony, recently Alexei even proclaimed 100%
> >> HUT pass rate, neverthless drlvm is not yet reliable enough and we
> >> should keep the momentum and face possible problems with open eyes.
> >> Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
> >> neccessary evil now.
> >
> > The nicer way to do this is to introduce this change as an option, and
> > let people start using it...
> >
>
> So it doesn't appear that I'm beating up on you :) I want to say that i
> do like some of these changes. very much, actually, such as the test
> properties, and the summary info. And the fact that we have that switch
> and you just forgot to mention it... is great.
>
> It's just good to announce to people, to give them a heads-up and a
> chance to respond.
>
> It's also easier if you describe the changes in prose, rather than hope
> someone has the time and wherewithal to grok the svn diff.
Thank you Geir! And thanks to all who cared and dared to speak up.
I see and I learn a lot.
--
Alexey
>
>
> geir
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Geir Magnusson Jr. wrote:
>
>
> Alexey Varlamov wrote:
>> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
>>> Geir Magnusson Jr. wrote:
>
>> Generally I agree it is always better to discuss, and in fact
>> hesitated to commit with default forking set to "once" - but decided
>> this is right thing to do rigth now. The current trend to make drlvm
>> truly standard VM for harmony, recently Alexei even proclaimed 100%
>> HUT pass rate, neverthless drlvm is not yet reliable enough and we
>> should keep the momentum and face possible problems with open eyes.
>> Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
>> neccessary evil now.
>
> The nicer way to do this is to introduce this change as an option, and
> let people start using it...
>
So it doesn't appear that I'm beating up on you :) I want to say that i
do like some of these changes. very much, actually, such as the test
properties, and the summary info. And the fact that we have that switch
and you just forgot to mention it... is great.
It's just good to announce to people, to give them a heads-up and a
chance to respond.
It's also easier if you describe the changes in prose, rather than hope
someone has the time and wherewithal to grok the svn diff.
geir
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Alexey Varlamov wrote:
> 2006/11/21, Gregory Shimansky <gs...@gmail.com>:
>> Geir Magnusson Jr. wrote:
>> >
>> >
>> > Alexey Varlamov wrote:
>> >> Folks,
>> >>
>> >> I've just did a little step in improving pre-commit testing for DRLVM;
>> >> most important change is a move from "perTest" forking mode to "once"
>> >> (aka sameVM mode).
>> >> This reduces testing time drastically (~50%), but may introduce some
>> >> extra instability (like new intermittent failures or timeouts).
>> >
>> > Plus the inability to figure out what's screwing things up.
>> >
>> > I can't decide if I like this. On one hand, I like it because it's
>> > actually better to find side-effects - some test can pass, but still
>> > leave the VM in a broken state that another test will show.
>>
>> I think the change has a good intension but it should've gone through
>> more testing. Preferably the fixes for problems on windows should've
>> been included into the patch. If the problems are too hard to fix, why
>> not open a discussion before changing the files in SVN?
>
> I did test on SUSE9, Win2003 server and WinXP(laptop) - only spotted
> *intermittent* failures on WinXP. I assumed it maybe little bothering
> but really easy to switch; OTOH we better discover instabilities
> sooner if they are frequent.
>
>>
>> > OTOH, it does remove the clarity of each test being a single, separate
>> > test. It conceptually mixes integration testing with unit testing.
>> >
>> > Can you please just add a switch? That way in the event of a failure,
>> > we can re-run with forking on, and therefore can tell if the crash is
>> > specifically due to the test that is crashing, or a side effect caused
>> > by something that came before.
>>
>> I would also like a switch. I don't like it that a change which
>> knowingly introduces problems with acceptance tests is committed without
>> discussion. Should we discard kernel tests from drlvm commits until they
>> are fixed? I think that there was an agreement about no regression. So
>> now all commits to drlvm are blocked.
>
> But there *is* cmd-line switch, my bad that I didn't explained this
> more clearly. And I'm not sure we firmly adopted "zero regression"
> policy - at least intermittent failures in kernel tests did block
> nobody so far. Should we vote formally, BTW?
No. We've always been zero regression in classlib - we've never been
clean enough with DRLVM to even know it.
So do I don't think we should make formal votes about - lets just drive
forward to make it so.
> Generally I agree it is always better to discuss, and in fact
> hesitated to commit with default forking set to "once" - but decided
> this is right thing to do rigth now. The current trend to make drlvm
> truly standard VM for harmony, recently Alexei even proclaimed 100%
> HUT pass rate, neverthless drlvm is not yet reliable enough and we
> should keep the momentum and face possible problems with open eyes.
> Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
> neccessary evil now.
The nicer way to do this is to introduce this change as an option, and
let people start using it...
geir
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make: targets/kernel.test.xml test.properties
Posted by Alexey Varlamov <al...@gmail.com>.
2006/11/21, Gregory Shimansky <gs...@gmail.com>:
> Geir Magnusson Jr. wrote:
> >
> >
> > Alexey Varlamov wrote:
> >> Folks,
> >>
> >> I've just did a little step in improving pre-commit testing for DRLVM;
> >> most important change is a move from "perTest" forking mode to "once"
> >> (aka sameVM mode).
> >> This reduces testing time drastically (~50%), but may introduce some
> >> extra instability (like new intermittent failures or timeouts).
> >
> > Plus the inability to figure out what's screwing things up.
> >
> > I can't decide if I like this. On one hand, I like it because it's
> > actually better to find side-effects - some test can pass, but still
> > leave the VM in a broken state that another test will show.
>
> I think the change has a good intension but it should've gone through
> more testing. Preferably the fixes for problems on windows should've
> been included into the patch. If the problems are too hard to fix, why
> not open a discussion before changing the files in SVN?
I did test on SUSE9, Win2003 server and WinXP(laptop) - only spotted
*intermittent* failures on WinXP. I assumed it maybe little bothering
but really easy to switch; OTOH we better discover instabilities
sooner if they are frequent.
>
> > OTOH, it does remove the clarity of each test being a single, separate
> > test. It conceptually mixes integration testing with unit testing.
> >
> > Can you please just add a switch? That way in the event of a failure,
> > we can re-run with forking on, and therefore can tell if the crash is
> > specifically due to the test that is crashing, or a side effect caused
> > by something that came before.
>
> I would also like a switch. I don't like it that a change which
> knowingly introduces problems with acceptance tests is committed without
> discussion. Should we discard kernel tests from drlvm commits until they
> are fixed? I think that there was an agreement about no regression. So
> now all commits to drlvm are blocked.
But there *is* cmd-line switch, my bad that I didn't explained this
more clearly. And I'm not sure we firmly adopted "zero regression"
policy - at least intermittent failures in kernel tests did block
nobody so far. Should we vote formally, BTW?
Generally I agree it is always better to discuss, and in fact
hesitated to commit with default forking set to "once" - but decided
this is right thing to do rigth now. The current trend to make drlvm
truly standard VM for harmony, recently Alexei even proclaimed 100%
HUT pass rate, neverthless drlvm is not yet reliable enough and we
should keep the momentum and face possible problems with open eyes.
Again, I'm sorry to bother you^H^H^H us VM guys, but believe this is
neccessary evil now.
--
Alexey
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>
>
> Gregory Shimansky wrote:
>> Geir Magnusson Jr. wrote:
>>>
>>>
>>> Gregory Shimansky wrote:
>>>
>>>> I just complained that instead of breaking things like this, the new
>>>> mode could be introduced as an option to allow us to fix tests
>>>> first, then switch to it.
>>>
>>> I'm missing something though - what broke because of this change?
>>
>> Well I assumed that kernel tests crashes on windows XP are because of
>> the new mode when all tests are executed in the same VM. When test is
>> executed separately it doesn't crash. I am going to try to find the
>> problem now.
>
> If that's the case, I'd argue that we were already broken, and we just
> didn't know it.
Yes this is true.
But imagine that I've created a new test that fails on VM. Should I
immediately add it to acceptance tests, or should I fix the problem
(announce about it in case I cannot do it myself) and only then add this
test as a regression?
The same happened with kernel tests. The switch was done before the
problem was fixed. I think it is not right.
--
Gregory
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Gregory Shimansky wrote:
> Geir Magnusson Jr. wrote:
>>
>>
>> Gregory Shimansky wrote:
>>
>>> I just complained that instead of breaking things like this, the new
>>> mode could be introduced as an option to allow us to fix tests first,
>>> then switch to it.
>>
>> I'm missing something though - what broke because of this change?
>
> Well I assumed that kernel tests crashes on windows XP are because of
> the new mode when all tests are executed in the same VM. When test is
> executed separately it doesn't crash. I am going to try to find the
> problem now.
If that's the case, I'd argue that we were already broken, and we just
didn't know it.
geir
>
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>
>
> Gregory Shimansky wrote:
>> Geir Magnusson Jr. wrote:
>>>
>>>
>>> Gregory Shimansky wrote:
>>>>
>>>> I would also like a switch. I don't like it that a change which
>>>> knowingly introduces problems with acceptance tests is committed
>>>> without discussion. Should we discard kernel tests from drlvm
>>>> commits until they are fixed? I think that there was an agreement
>>>> about no regression. So now all commits to drlvm are blocked.
>>>>
>>>
>>>
>>> Can we just declare that it's time to drive to getting these tests
>>> fixed????
>>
>> I never wrote that we shouldn't fix these bugs.
>
> I know.
>
>> I just complained that instead of breaking things like this, the new
>> mode could be introduced as an option to allow us to fix tests first,
>> then switch to it.
>
> I'm missing something though - what broke because of this change?
Well I assumed that kernel tests crashes on windows XP are because of
the new mode when all tests are executed in the same VM. When test is
executed separately it doesn't crash. I am going to try to find the
problem now.
--
Gregory
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Gregory Shimansky wrote:
> Geir Magnusson Jr. wrote:
>>
>>
>> Gregory Shimansky wrote:
>>>
>>> I would also like a switch. I don't like it that a change which
>>> knowingly introduces problems with acceptance tests is committed
>>> without discussion. Should we discard kernel tests from drlvm commits
>>> until they are fixed? I think that there was an agreement about no
>>> regression. So now all commits to drlvm are blocked.
>>>
>>
>>
>> Can we just declare that it's time to drive to getting these tests
>> fixed????
>
> I never wrote that we shouldn't fix these bugs.
I know.
> I just complained that
> instead of breaking things like this, the new mode could be introduced
> as an option to allow us to fix tests first, then switch to it.
I'm missing something though - what broke because of this change?
geir
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by Gregory Shimansky <gs...@gmail.com>.
Geir Magnusson Jr. wrote:
>
>
> Gregory Shimansky wrote:
>>
>> I would also like a switch. I don't like it that a change which
>> knowingly introduces problems with acceptance tests is committed
>> without discussion. Should we discard kernel tests from drlvm commits
>> until they are fixed? I think that there was an agreement about no
>> regression. So now all commits to drlvm are blocked.
>>
>
>
> Can we just declare that it's time to drive to getting these tests
> fixed????
I never wrote that we shouldn't fix these bugs. I just complained that
instead of breaking things like this, the new mode could be introduced
as an option to allow us to fix tests first, then switch to it.
> I'll start a new thread.
--
Gregory
Re: svn commit: r477149 - in /harmony/enhanced/drlvm/trunk/build/make:
targets/kernel.test.xml test.properties
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
Gregory Shimansky wrote:
>
> I would also like a switch. I don't like it that a change which
> knowingly introduces problems with acceptance tests is committed without
> discussion. Should we discard kernel tests from drlvm commits until they
> are fixed? I think that there was an agreement about no regression. So
> now all commits to drlvm are blocked.
>
Can we just declare that it's time to drive to getting these tests fixed????
I'll start a new thread.
geir