You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ignite.apache.org by ALEKSEY KUZNETSOV <al...@gmail.com> on 2017/02/07 16:40:21 UTC
Test failures
We have a lot of failed tests, which is frustrating. Some of them are
flaky(floating status randomly goes from succesful to failed) which adds to
frustration. Perhaps, we should fix all the tests in first place, and then
continue doing tickets ?
--
*Best Regards,*
*Kuznetsov Aleksey*
Re: Test failures
Posted by ALEKSEY KUZNETSOV <al...@gmail.com>.
yeah, i will
чт, 9 февр. 2017 г. в 0:01, Denis Magda <dm...@apache.org>:
> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
>
> Because there is no IT project that is bug-free. In my experience, it’s a
> usual situation when a part of the system is being evolved while the other
> is being fixed - this is why both feature development & bug fixing co-exist.
>
> In any case, I do believe that most of the tests that fail used to be
> sporadic ones when a modification was applied and a contributor had been
> validating his changes. These casual failures can easily lead to the
> situation when the contributor commits the changes that introduce new bugs.
>
> We should have JIRA tickets for most of the test failures:
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20summary~%22test%22%20and%20status%20!%3D%20closed
> <
> https://issues.apache.org/jira/issues/?jql=project%20=%20Ignite%20and%20summary~%22test%22%20and%20status%20!=%20closed
> >
>
> Would you be interested picking some of them making the product more
> stable? Basically, this should help your the internals even more profoundly.
>
> —
> Denis
>
> > On Feb 7, 2017, at 11:50 PM, ALEKSEY KUZNETSOV <al...@gmail.com>
> wrote:
> >
> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
> > Moreover its time consuming to look up new failed tests when your build
> has
> > completed.
> > The last one, committers who introduced new bugs is responsible for them
> > and have to fix them, not other ones.
> > There are number of tests which i have to execute to ensure they are
> flaky
> > or permanently failed. For example this one :
> > CacheJdbcPojoStoreTest.testLoadCache()
> >
> >
> >
> > вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
> >
> >> Aleksey,
> >>
> >> Bugs fixing and features development are two processes that usually
> >> co-exist.
> >>
> >> Some of the committer/contributors fix tests/functionality while the
> >> others add new functionality. Someone does both.
> >>
> >> You’re welcomed to start fixing the failing tests. Are there any
> specific
> >> that annoys you most?
> >>
> >> —
> >> Denis
> >>
> >>> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> alkuznetsov.sb@gmail.com>
> >> wrote:
> >>>
> >>> We have a lot of failed tests, which is frustrating. Some of them are
> >>> flaky(floating status randomly goes from succesful to failed) which
> adds
> >> to
> >>> frustration. Perhaps, we should fix all the tests in first place, and
> >> then
> >>> continue doing tickets ?
> >>> --
> >>>
> >>> *Best Regards,*
> >>>
> >>> *Kuznetsov Aleksey*
> >>
> >> --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --
*Best Regards,*
*Kuznetsov Aleksey*
Re: Test failures
Posted by Denis Magda <dm...@apache.org>.
> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).
Because there is no IT project that is bug-free. In my experience, it’s a usual situation when a part of the system is being evolved while the other is being fixed - this is why both feature development & bug fixing co-exist.
In any case, I do believe that most of the tests that fail used to be sporadic ones when a modification was applied and a contributor had been validating his changes. These casual failures can easily lead to the situation when the contributor commits the changes that introduce new bugs.
We should have JIRA tickets for most of the test failures:
https://issues.apache.org/jira/issues/?jql=project%20%3D%20Ignite%20and%20summary~%22test%22%20and%20status%20!%3D%20closed <https://issues.apache.org/jira/issues/?jql=project%20=%20Ignite%20and%20summary~%22test%22%20and%20status%20!=%20closed>
Would you be interested picking some of them making the product more stable? Basically, this should help your the internals even more profoundly.
—
Denis
> On Feb 7, 2017, at 11:50 PM, ALEKSEY KUZNETSOV <al...@gmail.com> wrote:
>
> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).
> Moreover its time consuming to look up new failed tests when your build has
> completed.
> The last one, committers who introduced new bugs is responsible for them
> and have to fix them, not other ones.
> There are number of tests which i have to execute to ensure they are flaky
> or permanently failed. For example this one :
> CacheJdbcPojoStoreTest.testLoadCache()
>
>
>
> вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
>
>> Aleksey,
>>
>> Bugs fixing and features development are two processes that usually
>> co-exist.
>>
>> Some of the committer/contributors fix tests/functionality while the
>> others add new functionality. Someone does both.
>>
>> You’re welcomed to start fixing the failing tests. Are there any specific
>> that annoys you most?
>>
>> —
>> Denis
>>
>>> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <al...@gmail.com>
>> wrote:
>>>
>>> We have a lot of failed tests, which is frustrating. Some of them are
>>> flaky(floating status randomly goes from succesful to failed) which adds
>> to
>>> frustration. Perhaps, we should fix all the tests in first place, and
>> then
>>> continue doing tickets ?
>>> --
>>>
>>> *Best Regards,*
>>>
>>> *Kuznetsov Aleksey*
>>
>> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
Re: Test failures
Posted by Dmitriy Setrakyan <ds...@apache.org>.
I also agree. I think we should create tickets for all failing tests and
have the community grab them.
On Wed, Feb 8, 2017 at 5:50 AM, Александр Меньшиков <sh...@gmail.com>
wrote:
> +1 to Aleksey, Alexander and Vyacheslav.
>
> I suppose that the best option is make issue for every master-failed-test.
> And fix them all.
>
> Floating tests should be normal. I think in most case we can just add
> repeating.
>
> All new test for some not ready future should be marked like "Should be fix
> in IGNITE-9999".
>
>
> 2017-02-08 15:23 GMT+03:00 Vyacheslav Daradur <da...@gmail.com>:
>
> > I vote for the master-branche without failed-tests)
> >
> > I understand that impossible to make it quickly.
> >
> > We shall aim at this approach.
> >
> > It will be more comfortable to us to develop.
> >
> > 2017-02-08 12:17 GMT+03:00 Alexander Fedotov <
> alexander.fedotoff@gmail.com
> > >:
> >
> > > Hi,
> > >
> > > I would agree with Aleksey.
> > > From the CI perspective, failing tests should be the main concern,
> > because
> > > it prevents a durable development of new features. Also, as Aleksey has
> > > noted, developers working on different features could end up fixing the
> > > same regressions, chances are - in different ways, resulting in merge
> > > conflicts.
> > > Having failing base branch, it is not possible to determine the reason
> > of a
> > > failure from the first sight.
> > > All these points impact as the feedback from tests, so the whole
> process
> > > agility.
> > > Ideally, no new feature development should be started based on the
> > failing
> > > branch.
> > > I think we should adopt this approach, otherwise, with growing amount
> of
> > > features,
> > > we will eventually end up spending more time dealing with regressions,
> > than
> > > developing features.
> > >
> > > Regards,
> > > Alexander
> > >
> > > 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> > > alkuznetsov.sb@gmail.com> написал:
> > >
> > > How could they co-exist ? When you developing some ticket you are
> risking
> > > introduce bug which is reproduced by already failed test(s).
> > > Moreover its time consuming to look up new failed tests when your build
> > has
> > > completed.
> > > The last one, committers who introduced new bugs is responsible for
> them
> > > and have to fix them, not other ones.
> > > There are number of tests which i have to execute to ensure they are
> > flaky
> > > or permanently failed. For example this one :
> > > CacheJdbcPojoStoreTest.testLoadCache()
> > >
> > >
> > >
> > > вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
> > >
> > > > Aleksey,
> > > >
> > > > Bugs fixing and features development are two processes that usually
> > > > co-exist.
> > > >
> > > > Some of the committer/contributors fix tests/functionality while the
> > > > others add new functionality. Someone does both.
> > > >
> > > > You’re welcomed to start fixing the failing tests. Are there any
> > specific
> > > > that annoys you most?
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> > > alkuznetsov.sb@gmail.com>
> > > > wrote:
> > > > >
> > > > > We have a lot of failed tests, which is frustrating. Some of them
> are
> > > > > flaky(floating status randomly goes from succesful to failed) which
> > > adds
> > > > to
> > > > > frustration. Perhaps, we should fix all the tests in first place,
> and
> > > > then
> > > > > continue doing tickets ?
> > > > > --
> > > > >
> > > > > *Best Regards,*
> > > > >
> > > > > *Kuznetsov Aleksey*
> > > >
> > > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> > >
> >
>
Re: Test failures
Posted by Александр Меньшиков <sh...@gmail.com>.
+1 to Aleksey, Alexander and Vyacheslav.
I suppose that the best option is make issue for every master-failed-test.
And fix them all.
Floating tests should be normal. I think in most case we can just add
repeating.
All new test for some not ready future should be marked like "Should be fix
in IGNITE-9999".
2017-02-08 15:23 GMT+03:00 Vyacheslav Daradur <da...@gmail.com>:
> I vote for the master-branche without failed-tests)
>
> I understand that impossible to make it quickly.
>
> We shall aim at this approach.
>
> It will be more comfortable to us to develop.
>
> 2017-02-08 12:17 GMT+03:00 Alexander Fedotov <alexander.fedotoff@gmail.com
> >:
>
> > Hi,
> >
> > I would agree with Aleksey.
> > From the CI perspective, failing tests should be the main concern,
> because
> > it prevents a durable development of new features. Also, as Aleksey has
> > noted, developers working on different features could end up fixing the
> > same regressions, chances are - in different ways, resulting in merge
> > conflicts.
> > Having failing base branch, it is not possible to determine the reason
> of a
> > failure from the first sight.
> > All these points impact as the feedback from tests, so the whole process
> > agility.
> > Ideally, no new feature development should be started based on the
> failing
> > branch.
> > I think we should adopt this approach, otherwise, with growing amount of
> > features,
> > we will eventually end up spending more time dealing with regressions,
> than
> > developing features.
> >
> > Regards,
> > Alexander
> >
> > 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> > alkuznetsov.sb@gmail.com> написал:
> >
> > How could they co-exist ? When you developing some ticket you are risking
> > introduce bug which is reproduced by already failed test(s).
> > Moreover its time consuming to look up new failed tests when your build
> has
> > completed.
> > The last one, committers who introduced new bugs is responsible for them
> > and have to fix them, not other ones.
> > There are number of tests which i have to execute to ensure they are
> flaky
> > or permanently failed. For example this one :
> > CacheJdbcPojoStoreTest.testLoadCache()
> >
> >
> >
> > вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
> >
> > > Aleksey,
> > >
> > > Bugs fixing and features development are two processes that usually
> > > co-exist.
> > >
> > > Some of the committer/contributors fix tests/functionality while the
> > > others add new functionality. Someone does both.
> > >
> > > You’re welcomed to start fixing the failing tests. Are there any
> specific
> > > that annoys you most?
> > >
> > > —
> > > Denis
> > >
> > > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> > alkuznetsov.sb@gmail.com>
> > > wrote:
> > > >
> > > > We have a lot of failed tests, which is frustrating. Some of them are
> > > > flaky(floating status randomly goes from succesful to failed) which
> > adds
> > > to
> > > > frustration. Perhaps, we should fix all the tests in first place, and
> > > then
> > > > continue doing tickets ?
> > > > --
> > > >
> > > > *Best Regards,*
> > > >
> > > > *Kuznetsov Aleksey*
> > >
> > > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
> >
>
Re: Test failures
Posted by Vyacheslav Daradur <da...@gmail.com>.
I vote for the master-branche without failed-tests)
I understand that impossible to make it quickly.
We shall aim at this approach.
It will be more comfortable to us to develop.
2017-02-08 12:17 GMT+03:00 Alexander Fedotov <al...@gmail.com>:
> Hi,
>
> I would agree with Aleksey.
> From the CI perspective, failing tests should be the main concern, because
> it prevents a durable development of new features. Also, as Aleksey has
> noted, developers working on different features could end up fixing the
> same regressions, chances are - in different ways, resulting in merge
> conflicts.
> Having failing base branch, it is not possible to determine the reason of a
> failure from the first sight.
> All these points impact as the feedback from tests, so the whole process
> agility.
> Ideally, no new feature development should be started based on the failing
> branch.
> I think we should adopt this approach, otherwise, with growing amount of
> features,
> we will eventually end up spending more time dealing with regressions, than
> developing features.
>
> Regards,
> Alexander
>
> 8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
> alkuznetsov.sb@gmail.com> написал:
>
> How could they co-exist ? When you developing some ticket you are risking
> introduce bug which is reproduced by already failed test(s).
> Moreover its time consuming to look up new failed tests when your build has
> completed.
> The last one, committers who introduced new bugs is responsible for them
> and have to fix them, not other ones.
> There are number of tests which i have to execute to ensure they are flaky
> or permanently failed. For example this one :
> CacheJdbcPojoStoreTest.testLoadCache()
>
>
>
> вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
>
> > Aleksey,
> >
> > Bugs fixing and features development are two processes that usually
> > co-exist.
> >
> > Some of the committer/contributors fix tests/functionality while the
> > others add new functionality. Someone does both.
> >
> > You’re welcomed to start fixing the failing tests. Are there any specific
> > that annoys you most?
> >
> > —
> > Denis
> >
> > > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <
> alkuznetsov.sb@gmail.com>
> > wrote:
> > >
> > > We have a lot of failed tests, which is frustrating. Some of them are
> > > flaky(floating status randomly goes from succesful to failed) which
> adds
> > to
> > > frustration. Perhaps, we should fix all the tests in first place, and
> > then
> > > continue doing tickets ?
> > > --
> > >
> > > *Best Regards,*
> > >
> > > *Kuznetsov Aleksey*
> >
> > --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*
>
Re: Test failures
Posted by Alexander Fedotov <al...@gmail.com>.
Hi,
I would agree with Aleksey.
From the CI perspective, failing tests should be the main concern, because
it prevents a durable development of new features. Also, as Aleksey has
noted, developers working on different features could end up fixing the
same regressions, chances are - in different ways, resulting in merge
conflicts.
Having failing base branch, it is not possible to determine the reason of a
failure from the first sight.
All these points impact as the feedback from tests, so the whole process
agility.
Ideally, no new feature development should be started based on the failing
branch.
I think we should adopt this approach, otherwise, with growing amount of
features,
we will eventually end up spending more time dealing with regressions, than
developing features.
Regards,
Alexander
8 февр. 2017 г. 10:50 AM пользователь "ALEKSEY KUZNETSOV" <
alkuznetsov.sb@gmail.com> написал:
How could they co-exist ? When you developing some ticket you are risking
introduce bug which is reproduced by already failed test(s).
Moreover its time consuming to look up new failed tests when your build has
completed.
The last one, committers who introduced new bugs is responsible for them
and have to fix them, not other ones.
There are number of tests which i have to execute to ensure they are flaky
or permanently failed. For example this one :
CacheJdbcPojoStoreTest.testLoadCache()
вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
> Aleksey,
>
> Bugs fixing and features development are two processes that usually
> co-exist.
>
> Some of the committer/contributors fix tests/functionality while the
> others add new functionality. Someone does both.
>
> You’re welcomed to start fixing the failing tests. Are there any specific
> that annoys you most?
>
> —
> Denis
>
> > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <al...@gmail.com>
> wrote:
> >
> > We have a lot of failed tests, which is frustrating. Some of them are
> > flaky(floating status randomly goes from succesful to failed) which adds
> to
> > frustration. Perhaps, we should fix all the tests in first place, and
> then
> > continue doing tickets ?
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --
*Best Regards,*
*Kuznetsov Aleksey*
Re: Test failures
Posted by ALEKSEY KUZNETSOV <al...@gmail.com>.
How could they co-exist ? When you developing some ticket you are risking
introduce bug which is reproduced by already failed test(s).
Moreover its time consuming to look up new failed tests when your build has
completed.
The last one, committers who introduced new bugs is responsible for them
and have to fix them, not other ones.
There are number of tests which i have to execute to ensure they are flaky
or permanently failed. For example this one :
CacheJdbcPojoStoreTest.testLoadCache()
вт, 7 февр. 2017 г. в 22:10, Denis Magda <dm...@apache.org>:
> Aleksey,
>
> Bugs fixing and features development are two processes that usually
> co-exist.
>
> Some of the committer/contributors fix tests/functionality while the
> others add new functionality. Someone does both.
>
> You’re welcomed to start fixing the failing tests. Are there any specific
> that annoys you most?
>
> —
> Denis
>
> > On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <al...@gmail.com>
> wrote:
> >
> > We have a lot of failed tests, which is frustrating. Some of them are
> > flaky(floating status randomly goes from succesful to failed) which adds
> to
> > frustration. Perhaps, we should fix all the tests in first place, and
> then
> > continue doing tickets ?
> > --
> >
> > *Best Regards,*
> >
> > *Kuznetsov Aleksey*
>
> --
*Best Regards,*
*Kuznetsov Aleksey*
Re: Test failures
Posted by Denis Magda <dm...@apache.org>.
Aleksey,
Bugs fixing and features development are two processes that usually co-exist.
Some of the committer/contributors fix tests/functionality while the others add new functionality. Someone does both.
You’re welcomed to start fixing the failing tests. Are there any specific that annoys you most?
—
Denis
> On Feb 7, 2017, at 8:40 AM, ALEKSEY KUZNETSOV <al...@gmail.com> wrote:
>
> We have a lot of failed tests, which is frustrating. Some of them are
> flaky(floating status randomly goes from succesful to failed) which adds to
> frustration. Perhaps, we should fix all the tests in first place, and then
> continue doing tickets ?
> --
>
> *Best Regards,*
>
> *Kuznetsov Aleksey*