You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@harmony.apache.org by tatyana doubtsova <ta...@gmail.com> on 2007/02/19 07:34:19 UTC
[testing][classlib] x-lists clean-up
Hi,
I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from common
x-lists passed on all 5 configurations. The tests are from auth, luni,
nio_char, rmi, sound, text, lang-manegement, and security modules. I've
prepared the patch for x-lists clean-up:
https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would you,
please, have a look and apply, if there are no objections?
Thanks,
Tanya
Re: [testing][classlib] x-lists clean-up
Posted by tatyana doubtsova <ta...@gmail.com>.
I've updated the patch
https://issues.apache.org/jira/browse/HARMONY-3194<https://issues.apache.org/jira/browse/HARMONY-3194.>
due
to the org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java is
incomplete (as Mikhail and Stepan confirmed).
Other build.xml (state before applying
https://issues.apache.org/jira/browse/HARMONY-2197) related to the list of
unexcluded tests do not containt comments to the excluded tests.
Thanks,
Tanya
On 2/19/07, tatyana doubtsova <ta...@gmail.com> wrote:
>
> The comment in auth build.xml
> <!-- excluded to avoid random failures -->
> is dated 07.09.06
> There were several clean-up commits to
> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java after this.
> All the tests pretend to be returned were run iteratively - PASSED
> All the modules with returned tests were run iteratively for 20 times in
> once mode - PASSED.
>
> In case of possible failures in iterative classlib tests runs under cc,
> I'll put the test in corresponding x-list.
>
> I also agree that JIRA number or/and comments in x-lists would be very
> helpful.
>
> Thanks,
> Tanya
>
>
> On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
> >
> > At the same time there are tests that reside in exclude lists by
> > absolutely unknown reason. I.e. initially they were added because of
> > some issue that was fixed a quarter ago. So IMHO the only way to get
> > rid of such unwanted excludes is to run *all* tests periodically and
> > check do they fail anymore. This is an exhausting work - thanks
> > Tatyana for doing this. BTW I think it will be also nice if we put
> > JIRA number or any other useful comment directly to exclude lists just
> > as a reminder why this test is here.
> >
> > Thanks,
> >
> > 2007/2/19, Mikhail Loenko <mloenko@gmail.com >:
> > > It would be good to find what were the reasons for excluding those 12
> > tests.
> > >
> > > For example, the first test in the patch:
> > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > >
> > > was originally excluded in build.xml with the following comment:
> > > <!-- The test is under development -->
> > > <!-- excluded to avoid random failures -->
> > >
> > > Since that the code of the test was not modified, so the test is still
> >
> > > incomplete and should not be returned back.
> > >
> > > Other tests may have similar problems too
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2007/2/19, tatyana doubtsova < tatyanadoubtsova@gmail.com>:
> > > > Hi,
> > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux
> > ia32
> > > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from
> > common
> > > > x-lists passed on all 5 configurations. The tests are from auth,
> > luni,
> > > > nio_char, rmi, sound, text, lang-manegement, and security modules.
> > I've
> > > > prepared the patch for x-lists clean-up:
> > > > https://issues.apache.org/jira/browse/HARMONY-3194. Commiters,
> > would you,
> > > > please, have a look and apply, if there are no objections?
> > > >
> > > > Thanks,
> > > > Tanya
> >
> >
> > --
> > Alexei Zakharov,
> > Intel ESSD
> >
>
>
Re: [testing][classlib] x-lists clean-up
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Feb 19, 2007, at 5:53 AM, tatyana doubtsova wrote:
> The comment in auth build.xml
> <!-- excluded to avoid random failures -->
> is dated 07.09.06
> There were several clean-up commits to
> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> after this.
> All the tests pretend to be returned were run iteratively - PASSED
> All the modules with returned tests were run iteratively for 20
> times in
> once mode - PASSED.
>
> In case of possible failures in iterative classlib tests runs under
> cc, I'll
> put the test in corresponding x-list.
>
> I also agree that JIRA number or/and comments in x-lists would be very
> helpful.
+1
On thing also to keep in mind is that tests could fail due to side
effects or VM issues that aren't clear at the time of exclusion. It
would be nice if there was a mechanical way to do this - for example,
maybe we can add a flag that also runs the excluded tests so we can
then examine the log and see what did pass that we didn't expect to.
This still feels like we're re-inventing the wheel. TestNG
supposedly has a lot of these features - has anyone tried looking at
TestNG recently? We talked about switching a while ago..
geir
>
> Thanks,
> Tanya
>
>
> On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
>>
>> At the same time there are tests that reside in exclude lists by
>> absolutely unknown reason. I.e. initially they were added because of
>> some issue that was fixed a quarter ago. So IMHO the only way to get
>> rid of such unwanted excludes is to run *all* tests periodically and
>> check do they fail anymore. This is an exhausting work - thanks
>> Tatyana for doing this. BTW I think it will be also nice if we put
>> JIRA number or any other useful comment directly to exclude lists
>> just
>> as a reminder why this test is here.
>>
>> Thanks,
>>
>> 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
>> > It would be good to find what were the reasons for excluding
>> those 12
>> tests.
>> >
>> > For example, the first test in the patch:
>> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>> >
>> > was originally excluded in build.xml with the following comment:
>> > <!-- The test is under development -->
>> > <!-- excluded to avoid random failures -->
>> >
>> > Since that the code of the test was not modified, so the test is
>> still
>> > incomplete and should not be returned back.
>> >
>> > Other tests may have similar problems too
>> >
>> > Thanks,
>> > Mikhail
>> >
>> > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
>> > > Hi,
>> > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
>> linux ia32
>> > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from
>> common
>> > > x-lists passed on all 5 configurations. The tests are from
>> auth, luni,
>> > > nio_char, rmi, sound, text, lang-manegement, and security
>> modules.
>> I've
>> > > prepared the patch for x-lists clean-up:
>> > > https://issues.apache.org/jira/browse/HARMONY-3194.
>> Commiters, would
>> you,
>> > > please, have a look and apply, if there are no objections?
>> > >
>> > > Thanks,
>> > > Tanya
>>
>>
>> --
>> Alexei Zakharov,
>> Intel ESSD
>>
Re: [testing][classlib] x-lists clean-up
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Feb 19, 2007, at 10:18 AM, Alexey Petrenko wrote:
> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
>> The comment in auth build.xml
>> <!-- excluded to avoid random failures -->
>> is dated 07.09.06
>> There were several clean-up commits to
>> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>> after this.
>> All the tests pretend to be returned were run iteratively - PASSED
>> All the modules with returned tests were run iteratively for 20
>> times in
>> once mode - PASSED.
>>
>> In case of possible failures in iterative classlib tests runs
>> under cc, I'll
>> put the test in corresponding x-list.
>>
>> I also agree that JIRA number or/and comments in x-lists would be
>> very
>> helpful.
> Is it possible to add comments to our exclude lists?
Yes. I put them in the drlvm ones when I first put the smoke
excludes in place.
geir
>
> SY, Alexey
>
>> On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
>> >
>> > At the same time there are tests that reside in exclude lists by
>> > absolutely unknown reason. I.e. initially they were added
>> because of
>> > some issue that was fixed a quarter ago. So IMHO the only way to
>> get
>> > rid of such unwanted excludes is to run *all* tests periodically
>> and
>> > check do they fail anymore. This is an exhausting work - thanks
>> > Tatyana for doing this. BTW I think it will be also nice if we put
>> > JIRA number or any other useful comment directly to exclude
>> lists just
>> > as a reminder why this test is here.
>> >
>> > Thanks,
>> >
>> > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
>> > > It would be good to find what were the reasons for excluding
>> those 12
>> > tests.
>> > >
>> > > For example, the first test in the patch:
>> > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>> > >
>> > > was originally excluded in build.xml with the following comment:
>> > > <!-- The test is under development -->
>> > > <!-- excluded to avoid random failures -->
>> > >
>> > > Since that the code of the test was not modified, so the test
>> is still
>> > > incomplete and should not be returned back.
>> > >
>> > > Other tests may have similar problems too
>> > >
>> > > Thanks,
>> > > Mikhail
>> > >
>> > > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
>> > > > Hi,
>> > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
>> linux ia32
>> > > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests
>> from
>> > common
>> > > > x-lists passed on all 5 configurations. The tests are from
>> auth, luni,
>> > > > nio_char, rmi, sound, text, lang-manegement, and security
>> modules.
>> > I've
>> > > > prepared the patch for x-lists clean-up:
>> > > > https://issues.apache.org/jira/browse/HARMONY-3194.
>> Commiters, would
>> > you,
>> > > > please, have a look and apply, if there are no objections?
>> > > >
>> > > > Thanks,
>> > > > Tanya
>> >
>> >
>> > --
>> > Alexei Zakharov,
>> > Intel ESSD
>> >
>>
Re: [testing][classlib] x-lists clean-up
Posted by tatyana doubtsova <ta...@gmail.com>.
Classlib x-lists tests run pass rate are available again on
http://wiki.apache.org/harmony/Classlib_exclude_lists_tracking
Alexei,
I did not publish swing test results, because of their instability.
Thanks,
Tanya
On 2/22/07, Alexei Zakharov <al...@gmail.com> wrote:
>
> Thanks Tatyana. BTW, I can't find a link to swing test results.
>
> Thanks,
>
> 2007/2/22, tatyana doubtsova <ta...@gmail.com>:
> > I have to define more accurate statistics, so it is cleaned for some
> time.
> > The lists of eror / failed tests are available
> >
> >
> > On 2/22/07, tatyana doubtsova <ta...@gmail.com> wrote:
> > >
> > > Here you can find excluded tests pass rate and detailed lists of
> failed
> > > tests:
> > >
> > > http://wiki.apache.org/harmony/Classlib_exclude_lists_tracking
> > >
> > > Thanks,
> > > Tanya
> > >
> > > On 2/20/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > > >
> > > >
> > > > On Feb 19, 2007, at 12:42 PM, Alexei Zakharov wrote:
> > > >
> > > > > Geir Magnusson Jr. wrote:
> > > > >> TestNG supposedly has a lot of these features - has anyone tried
> > > > >> looking at
> > > > >> TestNG recently? We talked about switching a while ago...
> > > > >
> > > > > The last thing I remember was the fact that *you* can't fully
> enjoy
> > > > > TestNG because it stores meta information inside the test source
> > > > > rather than in some external resource. :) However, I agree that it
> > > > > will be nice if we could get the ball moving forward here.
> > > >
> > > > Yes, that's true - I can go look at one file now to see what tests
> > > > are excluded, rather than looking at all tests to see which are :)
> > > >
> > > > >
> > > > > Alexey Petrenko wrote:
> > > > >> Is it possible to add comments to our exclude lists?
> > > > >
> > > > > Sure, you can put any stuff you want into it. If it coincides with
> the
> > > >
> > > > > name of some test then such test will be excluded. Otherwise the
> whole
> > > > > line is ignored AFAIU.
> > > > >
> > > > > Regards,
> > > > >
> > > > > 2007/2/19, Alexey Petrenko < alexey.a.petrenko@gmail.com>:
> > > > >> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > > > >> > The comment in auth build.xml
> > > > >> > <!-- excluded to avoid random failures -->
> > > > >> > is dated 07.09.06
> > > > >> > There were several clean-up commits to
> > > > >> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > > > >> after this.
> > > > >> > All the tests pretend to be returned were run iteratively -
> PASSED
> > > > >> > All the modules with returned tests were run iteratively for 20
> > > > >> times in
> > > > >> > once mode - PASSED.
> > > > >> >
> > > > >> > In case of possible failures in iterative classlib tests runs
> > > > >> under cc, I'll
> > > > >> > put the test in corresponding x-list.
> > > > >> >
> > > > >> > I also agree that JIRA number or/and comments in x-lists would
> > > > >> be very
> > > > >> > helpful.
> > > > >> Is it possible to add comments to our exclude lists?
> > > > >>
> > > > >> SY, Alexey
> > > > >>
> > > > >> > On 2/19/07, Alexei Zakharov <alexei.zakharov@gmail.com > wrote:
> > > > >> > >
> > > > >> > > At the same time there are tests that reside in exclude lists
> by
> > > > >> > > absolutely unknown reason. I.e. initially they were added
> > > > >> because of
> > > > >> > > some issue that was fixed a quarter ago. So IMHO the only way
> > > > >> to get
> > > > >> > > rid of such unwanted excludes is to run *all* tests
> > > > >> periodically and
> > > > >> > > check do they fail anymore. This is an exhausting work -
> thanks
> > > > >> > > Tatyana for doing this. BTW I think it will be also nice if
> we
> > > > >> put
> > > > >> > > JIRA number or any other useful comment directly to exclude
> > > > >> lists just
> > > > >> > > as a reminder why this test is here.
> > > > >> > >
> > > > >> > > Thanks,
> > > > >> > >
> > > > >> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > > > >> > > > It would be good to find what were the reasons for
> excluding
> > > > >> those 12
> > > > >> > > tests.
> > > > >> > > >
> > > > >> > > > For example, the first test in the patch:
> > > > >> > > >
> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > > > >> > > >
> > > > >> > > > was originally excluded in build.xml with the following
> > > > >> comment:
> > > > >> > > > <!-- The test is under development -->
> > > > >> > > > <!-- excluded to avoid random failures -->
> > > > >> > > >
> > > > >> > > > Since that the code of the test was not modified, so the
> > > > >> test is still
> > > > >> > > > incomplete and should not be returned back.
> > > > >> > > >
> > > > >> > > > Other tests may have similar problems too
> > > > >> > > >
> > > > >> > > > Thanks,
> > > > >> > > > Mikhail
> > > > >> > > >
> > > > >> > > > 2007/2/19, tatyana doubtsova < tatyanadoubtsova@gmail.com>:
> > > > >> > > > > Hi,
> > > > >> > > > > I've run excluded classlib tests on win ia32
> drlvm|ibmvme,
> > > > >> linux ia32
> > > > >> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12
> > > > >> tests from
> > > > >> > > common
> > > > >> > > > > x-lists passed on all 5 configurations. The tests are
> from
> > > > >> auth, luni,
> > > > >> > > > > nio_char, rmi, sound, text, lang-manegement, and security
> > > > >> modules.
> > > > >> > > I've
> > > > >> > > > > prepared the patch for x-lists clean-up:
> > > > >> > > > > https://issues.apache.org/jira/browse/HARMONY-3194.
> > > > >> Commiters, would
> > > > >> > > you,
> > > > >> > > > > please, have a look and apply, if there are no
> objections?
>
>
>
> --
> Alexei Zakharov,
> Intel ESSD
>
Re: [testing][classlib] x-lists clean-up
Posted by Alexei Zakharov <al...@gmail.com>.
Thanks Tatyana. BTW, I can't find a link to swing test results.
Thanks,
2007/2/22, tatyana doubtsova <ta...@gmail.com>:
> I have to define more accurate statistics, so it is cleaned for some time.
> The lists of eror / failed tests are available
>
>
> On 2/22/07, tatyana doubtsova <ta...@gmail.com> wrote:
> >
> > Here you can find excluded tests pass rate and detailed lists of failed
> > tests:
> >
> > http://wiki.apache.org/harmony/Classlib_exclude_lists_tracking
> >
> > Thanks,
> > Tanya
> >
> > On 2/20/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> > >
> > >
> > > On Feb 19, 2007, at 12:42 PM, Alexei Zakharov wrote:
> > >
> > > > Geir Magnusson Jr. wrote:
> > > >> TestNG supposedly has a lot of these features - has anyone tried
> > > >> looking at
> > > >> TestNG recently? We talked about switching a while ago...
> > > >
> > > > The last thing I remember was the fact that *you* can't fully enjoy
> > > > TestNG because it stores meta information inside the test source
> > > > rather than in some external resource. :) However, I agree that it
> > > > will be nice if we could get the ball moving forward here.
> > >
> > > Yes, that's true - I can go look at one file now to see what tests
> > > are excluded, rather than looking at all tests to see which are :)
> > >
> > > >
> > > > Alexey Petrenko wrote:
> > > >> Is it possible to add comments to our exclude lists?
> > > >
> > > > Sure, you can put any stuff you want into it. If it coincides with the
> > >
> > > > name of some test then such test will be excluded. Otherwise the whole
> > > > line is ignored AFAIU.
> > > >
> > > > Regards,
> > > >
> > > > 2007/2/19, Alexey Petrenko < alexey.a.petrenko@gmail.com>:
> > > >> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > > >> > The comment in auth build.xml
> > > >> > <!-- excluded to avoid random failures -->
> > > >> > is dated 07.09.06
> > > >> > There were several clean-up commits to
> > > >> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > > >> after this.
> > > >> > All the tests pretend to be returned were run iteratively - PASSED
> > > >> > All the modules with returned tests were run iteratively for 20
> > > >> times in
> > > >> > once mode - PASSED.
> > > >> >
> > > >> > In case of possible failures in iterative classlib tests runs
> > > >> under cc, I'll
> > > >> > put the test in corresponding x-list.
> > > >> >
> > > >> > I also agree that JIRA number or/and comments in x-lists would
> > > >> be very
> > > >> > helpful.
> > > >> Is it possible to add comments to our exclude lists?
> > > >>
> > > >> SY, Alexey
> > > >>
> > > >> > On 2/19/07, Alexei Zakharov <alexei.zakharov@gmail.com > wrote:
> > > >> > >
> > > >> > > At the same time there are tests that reside in exclude lists by
> > > >> > > absolutely unknown reason. I.e. initially they were added
> > > >> because of
> > > >> > > some issue that was fixed a quarter ago. So IMHO the only way
> > > >> to get
> > > >> > > rid of such unwanted excludes is to run *all* tests
> > > >> periodically and
> > > >> > > check do they fail anymore. This is an exhausting work - thanks
> > > >> > > Tatyana for doing this. BTW I think it will be also nice if we
> > > >> put
> > > >> > > JIRA number or any other useful comment directly to exclude
> > > >> lists just
> > > >> > > as a reminder why this test is here.
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > > >> > > > It would be good to find what were the reasons for excluding
> > > >> those 12
> > > >> > > tests.
> > > >> > > >
> > > >> > > > For example, the first test in the patch:
> > > >> > > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > > >> > > >
> > > >> > > > was originally excluded in build.xml with the following
> > > >> comment:
> > > >> > > > <!-- The test is under development -->
> > > >> > > > <!-- excluded to avoid random failures -->
> > > >> > > >
> > > >> > > > Since that the code of the test was not modified, so the
> > > >> test is still
> > > >> > > > incomplete and should not be returned back.
> > > >> > > >
> > > >> > > > Other tests may have similar problems too
> > > >> > > >
> > > >> > > > Thanks,
> > > >> > > > Mikhail
> > > >> > > >
> > > >> > > > 2007/2/19, tatyana doubtsova < tatyanadoubtsova@gmail.com>:
> > > >> > > > > Hi,
> > > >> > > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
> > > >> linux ia32
> > > >> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12
> > > >> tests from
> > > >> > > common
> > > >> > > > > x-lists passed on all 5 configurations. The tests are from
> > > >> auth, luni,
> > > >> > > > > nio_char, rmi, sound, text, lang-manegement, and security
> > > >> modules.
> > > >> > > I've
> > > >> > > > > prepared the patch for x-lists clean-up:
> > > >> > > > > https://issues.apache.org/jira/browse/HARMONY-3194.
> > > >> Commiters, would
> > > >> > > you,
> > > >> > > > > please, have a look and apply, if there are no objections?
--
Alexei Zakharov,
Intel ESSD
Re: [testing][classlib] x-lists clean-up
Posted by tatyana doubtsova <ta...@gmail.com>.
I have to define more accurate statistics, so it is cleaned for some time.
The lists of eror / failed tests are available
On 2/22/07, tatyana doubtsova <ta...@gmail.com> wrote:
>
> Here you can find excluded tests pass rate and detailed lists of failed
> tests:
>
> http://wiki.apache.org/harmony/Classlib_exclude_lists_tracking
>
> Thanks,
> Tanya
>
> On 2/20/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
> >
> >
> > On Feb 19, 2007, at 12:42 PM, Alexei Zakharov wrote:
> >
> > > Geir Magnusson Jr. wrote:
> > >> TestNG supposedly has a lot of these features - has anyone tried
> > >> looking at
> > >> TestNG recently? We talked about switching a while ago...
> > >
> > > The last thing I remember was the fact that *you* can't fully enjoy
> > > TestNG because it stores meta information inside the test source
> > > rather than in some external resource. :) However, I agree that it
> > > will be nice if we could get the ball moving forward here.
> >
> > Yes, that's true - I can go look at one file now to see what tests
> > are excluded, rather than looking at all tests to see which are :)
> >
> > >
> > > Alexey Petrenko wrote:
> > >> Is it possible to add comments to our exclude lists?
> > >
> > > Sure, you can put any stuff you want into it. If it coincides with the
> >
> > > name of some test then such test will be excluded. Otherwise the whole
> > > line is ignored AFAIU.
> > >
> > > Regards,
> > >
> > > 2007/2/19, Alexey Petrenko < alexey.a.petrenko@gmail.com>:
> > >> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > >> > The comment in auth build.xml
> > >> > <!-- excluded to avoid random failures -->
> > >> > is dated 07.09.06
> > >> > There were several clean-up commits to
> > >> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > >> after this.
> > >> > All the tests pretend to be returned were run iteratively - PASSED
> > >> > All the modules with returned tests were run iteratively for 20
> > >> times in
> > >> > once mode - PASSED.
> > >> >
> > >> > In case of possible failures in iterative classlib tests runs
> > >> under cc, I'll
> > >> > put the test in corresponding x-list.
> > >> >
> > >> > I also agree that JIRA number or/and comments in x-lists would
> > >> be very
> > >> > helpful.
> > >> Is it possible to add comments to our exclude lists?
> > >>
> > >> SY, Alexey
> > >>
> > >> > On 2/19/07, Alexei Zakharov <alexei.zakharov@gmail.com > wrote:
> > >> > >
> > >> > > At the same time there are tests that reside in exclude lists by
> > >> > > absolutely unknown reason. I.e. initially they were added
> > >> because of
> > >> > > some issue that was fixed a quarter ago. So IMHO the only way
> > >> to get
> > >> > > rid of such unwanted excludes is to run *all* tests
> > >> periodically and
> > >> > > check do they fail anymore. This is an exhausting work - thanks
> > >> > > Tatyana for doing this. BTW I think it will be also nice if we
> > >> put
> > >> > > JIRA number or any other useful comment directly to exclude
> > >> lists just
> > >> > > as a reminder why this test is here.
> > >> > >
> > >> > > Thanks,
> > >> > >
> > >> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > >> > > > It would be good to find what were the reasons for excluding
> > >> those 12
> > >> > > tests.
> > >> > > >
> > >> > > > For example, the first test in the patch:
> > >> > > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > >> > > >
> > >> > > > was originally excluded in build.xml with the following
> > >> comment:
> > >> > > > <!-- The test is under development -->
> > >> > > > <!-- excluded to avoid random failures -->
> > >> > > >
> > >> > > > Since that the code of the test was not modified, so the
> > >> test is still
> > >> > > > incomplete and should not be returned back.
> > >> > > >
> > >> > > > Other tests may have similar problems too
> > >> > > >
> > >> > > > Thanks,
> > >> > > > Mikhail
> > >> > > >
> > >> > > > 2007/2/19, tatyana doubtsova < tatyanadoubtsova@gmail.com>:
> > >> > > > > Hi,
> > >> > > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
> > >> linux ia32
> > >> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12
> > >> tests from
> > >> > > common
> > >> > > > > x-lists passed on all 5 configurations. The tests are from
> > >> auth, luni,
> > >> > > > > nio_char, rmi, sound, text, lang-manegement, and security
> > >> modules.
> > >> > > I've
> > >> > > > > prepared the patch for x-lists clean-up:
> > >> > > > > https://issues.apache.org/jira/browse/HARMONY-3194.
> > >> Commiters, would
> > >> > > you,
> > >> > > > > please, have a look and apply, if there are no objections?
> > >
> > >
> > >
> > > --
> > > Alexei Zakharov,
> > > Intel ESSD
> >
> >
>
Re: [testing][classlib] x-lists clean-up
Posted by tatyana doubtsova <ta...@gmail.com>.
Here you can find excluded tests pass rate and detailed lists of failed
tests:
http://wiki.apache.org/harmony/Classlib_exclude_lists_tracking
Thanks,
Tanya
On 2/20/07, Geir Magnusson Jr. <ge...@pobox.com> wrote:
>
>
> On Feb 19, 2007, at 12:42 PM, Alexei Zakharov wrote:
>
> > Geir Magnusson Jr. wrote:
> >> TestNG supposedly has a lot of these features - has anyone tried
> >> looking at
> >> TestNG recently? We talked about switching a while ago...
> >
> > The last thing I remember was the fact that *you* can't fully enjoy
> > TestNG because it stores meta information inside the test source
> > rather than in some external resource. :) However, I agree that it
> > will be nice if we could get the ball moving forward here.
>
> Yes, that's true - I can go look at one file now to see what tests
> are excluded, rather than looking at all tests to see which are :)
>
> >
> > Alexey Petrenko wrote:
> >> Is it possible to add comments to our exclude lists?
> >
> > Sure, you can put any stuff you want into it. If it coincides with the
> > name of some test then such test will be excluded. Otherwise the whole
> > line is ignored AFAIU.
> >
> > Regards,
> >
> > 2007/2/19, Alexey Petrenko <al...@gmail.com>:
> >> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> >> > The comment in auth build.xml
> >> > <!-- excluded to avoid random failures -->
> >> > is dated 07.09.06
> >> > There were several clean-up commits to
> >> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> >> after this.
> >> > All the tests pretend to be returned were run iteratively - PASSED
> >> > All the modules with returned tests were run iteratively for 20
> >> times in
> >> > once mode - PASSED.
> >> >
> >> > In case of possible failures in iterative classlib tests runs
> >> under cc, I'll
> >> > put the test in corresponding x-list.
> >> >
> >> > I also agree that JIRA number or/and comments in x-lists would
> >> be very
> >> > helpful.
> >> Is it possible to add comments to our exclude lists?
> >>
> >> SY, Alexey
> >>
> >> > On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
> >> > >
> >> > > At the same time there are tests that reside in exclude lists by
> >> > > absolutely unknown reason. I.e. initially they were added
> >> because of
> >> > > some issue that was fixed a quarter ago. So IMHO the only way
> >> to get
> >> > > rid of such unwanted excludes is to run *all* tests
> >> periodically and
> >> > > check do they fail anymore. This is an exhausting work - thanks
> >> > > Tatyana for doing this. BTW I think it will be also nice if we
> >> put
> >> > > JIRA number or any other useful comment directly to exclude
> >> lists just
> >> > > as a reminder why this test is here.
> >> > >
> >> > > Thanks,
> >> > >
> >> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> >> > > > It would be good to find what were the reasons for excluding
> >> those 12
> >> > > tests.
> >> > > >
> >> > > > For example, the first test in the patch:
> >> > > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> >> > > >
> >> > > > was originally excluded in build.xml with the following
> >> comment:
> >> > > > <!-- The test is under development -->
> >> > > > <!-- excluded to avoid random failures -->
> >> > > >
> >> > > > Since that the code of the test was not modified, so the
> >> test is still
> >> > > > incomplete and should not be returned back.
> >> > > >
> >> > > > Other tests may have similar problems too
> >> > > >
> >> > > > Thanks,
> >> > > > Mikhail
> >> > > >
> >> > > > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> >> > > > > Hi,
> >> > > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
> >> linux ia32
> >> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12
> >> tests from
> >> > > common
> >> > > > > x-lists passed on all 5 configurations. The tests are from
> >> auth, luni,
> >> > > > > nio_char, rmi, sound, text, lang-manegement, and security
> >> modules.
> >> > > I've
> >> > > > > prepared the patch for x-lists clean-up:
> >> > > > > https://issues.apache.org/jira/browse/HARMONY-3194.
> >> Commiters, would
> >> > > you,
> >> > > > > please, have a look and apply, if there are no objections?
> >
> >
> >
> > --
> > Alexei Zakharov,
> > Intel ESSD
>
>
Re: [testing][classlib] x-lists clean-up
Posted by "Geir Magnusson Jr." <ge...@pobox.com>.
On Feb 19, 2007, at 12:42 PM, Alexei Zakharov wrote:
> Geir Magnusson Jr. wrote:
>> TestNG supposedly has a lot of these features - has anyone tried
>> looking at
>> TestNG recently? We talked about switching a while ago...
>
> The last thing I remember was the fact that *you* can't fully enjoy
> TestNG because it stores meta information inside the test source
> rather than in some external resource. :) However, I agree that it
> will be nice if we could get the ball moving forward here.
Yes, that's true - I can go look at one file now to see what tests
are excluded, rather than looking at all tests to see which are :)
>
> Alexey Petrenko wrote:
>> Is it possible to add comments to our exclude lists?
>
> Sure, you can put any stuff you want into it. If it coincides with the
> name of some test then such test will be excluded. Otherwise the whole
> line is ignored AFAIU.
>
> Regards,
>
> 2007/2/19, Alexey Petrenko <al...@gmail.com>:
>> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
>> > The comment in auth build.xml
>> > <!-- excluded to avoid random failures -->
>> > is dated 07.09.06
>> > There were several clean-up commits to
>> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>> after this.
>> > All the tests pretend to be returned were run iteratively - PASSED
>> > All the modules with returned tests were run iteratively for 20
>> times in
>> > once mode - PASSED.
>> >
>> > In case of possible failures in iterative classlib tests runs
>> under cc, I'll
>> > put the test in corresponding x-list.
>> >
>> > I also agree that JIRA number or/and comments in x-lists would
>> be very
>> > helpful.
>> Is it possible to add comments to our exclude lists?
>>
>> SY, Alexey
>>
>> > On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
>> > >
>> > > At the same time there are tests that reside in exclude lists by
>> > > absolutely unknown reason. I.e. initially they were added
>> because of
>> > > some issue that was fixed a quarter ago. So IMHO the only way
>> to get
>> > > rid of such unwanted excludes is to run *all* tests
>> periodically and
>> > > check do they fail anymore. This is an exhausting work - thanks
>> > > Tatyana for doing this. BTW I think it will be also nice if we
>> put
>> > > JIRA number or any other useful comment directly to exclude
>> lists just
>> > > as a reminder why this test is here.
>> > >
>> > > Thanks,
>> > >
>> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
>> > > > It would be good to find what were the reasons for excluding
>> those 12
>> > > tests.
>> > > >
>> > > > For example, the first test in the patch:
>> > > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>> > > >
>> > > > was originally excluded in build.xml with the following
>> comment:
>> > > > <!-- The test is under development -->
>> > > > <!-- excluded to avoid random failures -->
>> > > >
>> > > > Since that the code of the test was not modified, so the
>> test is still
>> > > > incomplete and should not be returned back.
>> > > >
>> > > > Other tests may have similar problems too
>> > > >
>> > > > Thanks,
>> > > > Mikhail
>> > > >
>> > > > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
>> > > > > Hi,
>> > > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme,
>> linux ia32
>> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12
>> tests from
>> > > common
>> > > > > x-lists passed on all 5 configurations. The tests are from
>> auth, luni,
>> > > > > nio_char, rmi, sound, text, lang-manegement, and security
>> modules.
>> > > I've
>> > > > > prepared the patch for x-lists clean-up:
>> > > > > https://issues.apache.org/jira/browse/HARMONY-3194.
>> Commiters, would
>> > > you,
>> > > > > please, have a look and apply, if there are no objections?
>
>
>
> --
> Alexei Zakharov,
> Intel ESSD
Re: [testing][classlib] x-lists clean-up
Posted by Alexei Zakharov <al...@gmail.com>.
Geir Magnusson Jr. wrote:
> TestNG supposedly has a lot of these features - has anyone tried looking at
> TestNG recently? We talked about switching a while ago...
The last thing I remember was the fact that *you* can't fully enjoy
TestNG because it stores meta information inside the test source
rather than in some external resource. :) However, I agree that it
will be nice if we could get the ball moving forward here.
Alexey Petrenko wrote:
> Is it possible to add comments to our exclude lists?
Sure, you can put any stuff you want into it. If it coincides with the
name of some test then such test will be excluded. Otherwise the whole
line is ignored AFAIU.
Regards,
2007/2/19, Alexey Petrenko <al...@gmail.com>:
> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > The comment in auth build.xml
> > <!-- excluded to avoid random failures -->
> > is dated 07.09.06
> > There were several clean-up commits to
> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java after this.
> > All the tests pretend to be returned were run iteratively - PASSED
> > All the modules with returned tests were run iteratively for 20 times in
> > once mode - PASSED.
> >
> > In case of possible failures in iterative classlib tests runs under cc, I'll
> > put the test in corresponding x-list.
> >
> > I also agree that JIRA number or/and comments in x-lists would be very
> > helpful.
> Is it possible to add comments to our exclude lists?
>
> SY, Alexey
>
> > On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
> > >
> > > At the same time there are tests that reside in exclude lists by
> > > absolutely unknown reason. I.e. initially they were added because of
> > > some issue that was fixed a quarter ago. So IMHO the only way to get
> > > rid of such unwanted excludes is to run *all* tests periodically and
> > > check do they fail anymore. This is an exhausting work - thanks
> > > Tatyana for doing this. BTW I think it will be also nice if we put
> > > JIRA number or any other useful comment directly to exclude lists just
> > > as a reminder why this test is here.
> > >
> > > Thanks,
> > >
> > > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > > > It would be good to find what were the reasons for excluding those 12
> > > tests.
> > > >
> > > > For example, the first test in the patch:
> > > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > > >
> > > > was originally excluded in build.xml with the following comment:
> > > > <!-- The test is under development -->
> > > > <!-- excluded to avoid random failures -->
> > > >
> > > > Since that the code of the test was not modified, so the test is still
> > > > incomplete and should not be returned back.
> > > >
> > > > Other tests may have similar problems too
> > > >
> > > > Thanks,
> > > > Mikhail
> > > >
> > > > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > > > > Hi,
> > > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
> > > > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from
> > > common
> > > > > x-lists passed on all 5 configurations. The tests are from auth, luni,
> > > > > nio_char, rmi, sound, text, lang-manegement, and security modules.
> > > I've
> > > > > prepared the patch for x-lists clean-up:
> > > > > https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would
> > > you,
> > > > > please, have a look and apply, if there are no objections?
--
Alexei Zakharov,
Intel ESSD
Re: [testing][classlib] x-lists clean-up
Posted by Alexey Petrenko <al...@gmail.com>.
2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> The comment in auth build.xml
> <!-- excluded to avoid random failures -->
> is dated 07.09.06
> There were several clean-up commits to
> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java after this.
> All the tests pretend to be returned were run iteratively - PASSED
> All the modules with returned tests were run iteratively for 20 times in
> once mode - PASSED.
>
> In case of possible failures in iterative classlib tests runs under cc, I'll
> put the test in corresponding x-list.
>
> I also agree that JIRA number or/and comments in x-lists would be very
> helpful.
Is it possible to add comments to our exclude lists?
SY, Alexey
> On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
> >
> > At the same time there are tests that reside in exclude lists by
> > absolutely unknown reason. I.e. initially they were added because of
> > some issue that was fixed a quarter ago. So IMHO the only way to get
> > rid of such unwanted excludes is to run *all* tests periodically and
> > check do they fail anymore. This is an exhausting work - thanks
> > Tatyana for doing this. BTW I think it will be also nice if we put
> > JIRA number or any other useful comment directly to exclude lists just
> > as a reminder why this test is here.
> >
> > Thanks,
> >
> > 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > > It would be good to find what were the reasons for excluding those 12
> > tests.
> > >
> > > For example, the first test in the patch:
> > > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> > >
> > > was originally excluded in build.xml with the following comment:
> > > <!-- The test is under development -->
> > > <!-- excluded to avoid random failures -->
> > >
> > > Since that the code of the test was not modified, so the test is still
> > > incomplete and should not be returned back.
> > >
> > > Other tests may have similar problems too
> > >
> > > Thanks,
> > > Mikhail
> > >
> > > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > > > Hi,
> > > > I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
> > > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from
> > common
> > > > x-lists passed on all 5 configurations. The tests are from auth, luni,
> > > > nio_char, rmi, sound, text, lang-manegement, and security modules.
> > I've
> > > > prepared the patch for x-lists clean-up:
> > > > https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would
> > you,
> > > > please, have a look and apply, if there are no objections?
> > > >
> > > > Thanks,
> > > > Tanya
> >
> >
> > --
> > Alexei Zakharov,
> > Intel ESSD
> >
>
Re: [testing][classlib] x-lists clean-up
Posted by tatyana doubtsova <ta...@gmail.com>.
The comment in auth build.xml
<!-- excluded to avoid random failures -->
is dated 07.09.06
There were several clean-up commits to
org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java after this.
All the tests pretend to be returned were run iteratively - PASSED
All the modules with returned tests were run iteratively for 20 times in
once mode - PASSED.
In case of possible failures in iterative classlib tests runs under cc, I'll
put the test in corresponding x-list.
I also agree that JIRA number or/and comments in x-lists would be very
helpful.
Thanks,
Tanya
On 2/19/07, Alexei Zakharov <al...@gmail.com> wrote:
>
> At the same time there are tests that reside in exclude lists by
> absolutely unknown reason. I.e. initially they were added because of
> some issue that was fixed a quarter ago. So IMHO the only way to get
> rid of such unwanted excludes is to run *all* tests periodically and
> check do they fail anymore. This is an exhausting work - thanks
> Tatyana for doing this. BTW I think it will be also nice if we put
> JIRA number or any other useful comment directly to exclude lists just
> as a reminder why this test is here.
>
> Thanks,
>
> 2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> > It would be good to find what were the reasons for excluding those 12
> tests.
> >
> > For example, the first test in the patch:
> > org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
> >
> > was originally excluded in build.xml with the following comment:
> > <!-- The test is under development -->
> > <!-- excluded to avoid random failures -->
> >
> > Since that the code of the test was not modified, so the test is still
> > incomplete and should not be returned back.
> >
> > Other tests may have similar problems too
> >
> > Thanks,
> > Mikhail
> >
> > 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > > Hi,
> > > I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
> > > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from
> common
> > > x-lists passed on all 5 configurations. The tests are from auth, luni,
> > > nio_char, rmi, sound, text, lang-manegement, and security modules.
> I've
> > > prepared the patch for x-lists clean-up:
> > > https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would
> you,
> > > please, have a look and apply, if there are no objections?
> > >
> > > Thanks,
> > > Tanya
>
>
> --
> Alexei Zakharov,
> Intel ESSD
>
Re: [testing][classlib] x-lists clean-up
Posted by Alexei Zakharov <al...@gmail.com>.
At the same time there are tests that reside in exclude lists by
absolutely unknown reason. I.e. initially they were added because of
some issue that was fixed a quarter ago. So IMHO the only way to get
rid of such unwanted excludes is to run *all* tests periodically and
check do they fail anymore. This is an exhausting work - thanks
Tatyana for doing this. BTW I think it will be also nice if we put
JIRA number or any other useful comment directly to exclude lists just
as a reminder why this test is here.
Thanks,
2007/2/19, Mikhail Loenko <ml...@gmail.com>:
> It would be good to find what were the reasons for excluding those 12 tests.
>
> For example, the first test in the patch:
> org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
>
> was originally excluded in build.xml with the following comment:
> <!-- The test is under development -->
> <!-- excluded to avoid random failures -->
>
> Since that the code of the test was not modified, so the test is still
> incomplete and should not be returned back.
>
> Other tests may have similar problems too
>
> Thanks,
> Mikhail
>
> 2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> > Hi,
> > I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
> > drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from common
> > x-lists passed on all 5 configurations. The tests are from auth, luni,
> > nio_char, rmi, sound, text, lang-manegement, and security modules. I've
> > prepared the patch for x-lists clean-up:
> > https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would you,
> > please, have a look and apply, if there are no objections?
> >
> > Thanks,
> > Tanya
--
Alexei Zakharov,
Intel ESSD
Re: [testing][classlib] x-lists clean-up
Posted by Mikhail Loenko <ml...@gmail.com>.
It would be good to find what were the reasons for excluding those 12 tests.
For example, the first test in the patch:
org/apache/harmony/auth/tests/module/Krb5LoginModuleTest.java
was originally excluded in build.xml with the following comment:
<!-- The test is under development -->
<!-- excluded to avoid random failures -->
Since that the code of the test was not modified, so the test is still
incomplete and should not be returned back.
Other tests may have similar problems too
Thanks,
Mikhail
2007/2/19, tatyana doubtsova <ta...@gmail.com>:
> Hi,
> I've run excluded classlib tests on win ia32 drlvm|ibmvme, linux ia32
> drlvm|ibmvme, linux em64t drlvm and found out that 12 tests from common
> x-lists passed on all 5 configurations. The tests are from auth, luni,
> nio_char, rmi, sound, text, lang-manegement, and security modules. I've
> prepared the patch for x-lists clean-up:
> https://issues.apache.org/jira/browse/HARMONY-3194. Commiters, would you,
> please, have a look and apply, if there are no objections?
>
> Thanks,
> Tanya
>