You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kie.apache.org by Tibor Zimányi <tz...@apache.org> on 2024/01/23 14:36:00 UTC
[PROPOSAL] Workflow for ignoring long failing tests
Hi everyone,
I want to propose a workflow for situations, when some tests are failing
for a longer time. In such cases, my proposed workflow is:
- If a test or a set of tests is failing for two days (nightlies or PR
checks), ignore those tests in the codebase, so they are not executed.
- File an issue in kie-issues repository, reporting the test failures.
- If the issue is not resolved for half a year, delete those tests from the
codebase.
What do you think please? This should make sure all failures that don't get
fixed immediately after they occur get logged in the issue tracker, so they
can be appropriately handled and don't block unrelated PR checks and
builds. It will also make sure that tests that are not fixed for a long
time (therefore they could be perceived as not important) are not a
maintenance burden for the future.
Best regards,
Tibor
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
I understood this policy refers just to flaky tests. We should not merge
any PR on red.
On Fri, Jan 26, 2024 at 8:00 AM Toni Rikkola <tr...@redhat.com> wrote:
> What happened with the SpringBoot. I made the change and actually thought
> the PR reviews would alert the right people and you could not get a PR in
> without approval from the devs responsible for that area. This is just
> since I don't know who is working on what and on what level. I don't know
> how the wording in this rule could be set so that the right person has
> actually been contacted.
>
> One other thing to take note of is that this policy applies for all the
> broken tests, not just flaky ones. So if you manage to get a bad commit in,
> it is then ok to ignore all the tests that that commit broke and now the
> responsibility of fixing the codes are on the person that wrote the test
> that picks up the bugs or changes someone else introduced.
>
> I know I like to over think these regulations, but due to the structure at
> Apache we agree on rules and some can be taken advantage of. If the thing
> mentioned above happens, there is not one great leader who can give an
> order to fix the codes. Also if a community member gets a commit in, there
> is no handle for us to get them to fix anything. It is the community's
> problem after that.
>
> I have to give this a -1, this has to target just flaky tests. If a commit
> breaks the test, the right thing would be to revert ASAP. If we really need
> to get something in that breaks tests in another location, and those tests
> need to be redone, then we should vote for that change to go in as an
> exception and ignore the broken tests in the same commit bundle.
> Acknowledging there is no guarantee the tests commented out will ever be
> fixed, even if they show a bug was introduced.
>
> So for example in the current case. If the build is broken and the tests
> are not flaky, we can vote for disabling those tests. I think that vote
> will pass.
>
> I hope we understand what this project is now. One person can make a big
> mess if the guards are not set up correctly. Five more companies can come
> in and start working on this, out ruling everyone we have now. The base has
> to be solid for many reasons. Letting this change in as a rule opens us to
> too much abuse, intentional or unintentional.
>
> Toni
>
> On Thu, Jan 25, 2024 at 2:59 PM Francisco Javier Tirado Sarti <
> ftirados@redhat.com> wrote:
>
> > +1
> > Thanks Tibor
> >
> > On Thu, Jan 25, 2024 at 9:37 AM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > Hi,
> > >
> > > as I wrote in one of the previous e-mails in this thread:
> > > "I think the notification of the involved commiter could be done as
> part
> > of
> > > creating of the issue to fix the tests. The person involved can be
> > assigned
> > > as an asignee. "
> > >
> > > To clarify, by involved commiter I meant either author of the test or
> the
> > > person who is maintaining that part of the codebase.
> > >
> > > Hopefully this clarifies this.
> > >
> > > Tibor
> > >
> > > Dňa št 25. 1. 2024, 9:22 Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > napísal(a):
> > >
> > > > I agree with Josef, I realize the SpringBoot listener example test
> was
> > > > disabled while fixing an issue related to it. If not, the test would
> > have
> > > > remained disabled.
> > > >
> > > > On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko
> > <Jozef.Marko@ibm.com.invalid
> > > >
> > > > wrote:
> > > >
> > > > > Thank you for this discussion. I think disabling flaky tests in
> given
> > > way
> > > > > is reasonable.
> > > > >
> > > > > I have one related question, once similar flaky test is disabled,
> > and a
> > > > > ticket is reported. Who will be assigned to this ticket? Will it be
> > the
> > > > > test author?
> > > > >
> > > > > My point is, that usually only test author and few other engineers
> > > > working
> > > > > on the same component understand completely the given test, but
> > > probably
> > > > > they do not check the codebase daily if the test is still enabled.
> > > > >
> > > > > So, will be these people notified? Otherwise, most of these tickets
> > > will
> > > > > be probably closed after the proposed period without any test
> > > update/fix.
> > > > >
> > > > > Jozef Marko
> > > > > Software Developer
> > > > > jozef.marko@ibm.com
> > > > >
> > > > > -----Original Message-----
> > > > > From: Toni Rikkola <tr...@redhat.com>
> > > > > Sent: Wednesday, January 24, 2024 3:58 PM
> > > > > To: dev@kie.apache.org
> > > > > Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long
> failing
> > > > tests
> > > > >
> > > > > The reason I mentioned this was that we had this same decision
> made 8
> > > > > years ago and shared it to the entire team.
> > > > > It stated that anyone can disable any flaky test and make a ticket
> > for
> > > > it.
> > > > > It was even more strict, you just had to have one flaky failure.
> > > > >
> > > > > So to me it looks like we are just stating the same thing again
> > hoping
> > > > for
> > > > > a different outcome.
> > > > >
> > > > > But yes, to make it clear for everyone we should write it down and
> we
> > > can
> > > > > see how it goes and act on it later if needed.
> > > > >
> > > > > Toni
> > > > >
> > > > > On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tzimanyi@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > I think we don't need to enforce it or gatekeep this any way. The
> > > > > > thing is, if we have the agreement that we want to have this
> > official
> > > > > > workflow, it should fix the problem you are mentioning Toni, that
> > > > > > people are afraid to do so. If it will be officially stated, that
> > > > > > failing tests can be disabled in the code, people should just do
> > it.
> > > > > > They can always point to this potential agreement. The best thing
> > > > > > would be to even document it on Confluence. I will do that if we
> > have
> > > > an
> > > > > agreement on this.
> > > > > >
> > > > > > Tibor
> > > > > >
> > > > > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com>
> > > > napísal(a):
> > > > > >
> > > > > > > Not informing the author was my mistake on that PR. I should
> have
> > > > > > > asked
> > > > > > the
> > > > > > > original author to review it.
> > > > > > >
> > > > > > > But the problem is not that we lack this procedure. The KIE
> team
> > > > > > > agreed
> > > > > > on
> > > > > > > this in 2015. It is that maybe we lack the strength to do it,
> > > > > > > especially when it is someone else's area of code. This needs
> > some
> > > > > > > sort of
> > > > > > gatekeeper
> > > > > > > or supervisor and time frames when the build quality and if
> these
> > > > > > problems
> > > > > > > were acted on are validated and measured.
> > > > > > >
> > > > > > > Meanwhile I offer myself to work on these and disable them, if
> > > > > > > anyone
> > > > > > wants
> > > > > > > to point me to some flaky test I can act on them. I could even
> > make
> > > > > > > some system for measurement. We can then, or now, decide what
> we
> > > > > > > then do with that data.
> > > > > > >
> > > > > > > Toni
> > > > > > >
> > > > > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > I think the notification of the involved commiter could be
> done
> > > as
> > > > > > > > part
> > > > > > > of
> > > > > > > > creating of the issue to fix the tests. The person involved
> can
> > > be
> > > > > > > assigned
> > > > > > > > as an asignee.
> > > > > > > >
> > > > > > > > T.
> > > > > > > >
> > > > > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > > > > > > ftirados@redhat.com>
> > > > > > > > napísal(a):
> > > > > > > >
> > > > > > > > > Ok, but I think we first need to establish a way (my
> > apologies
> > > > > > > > > is
> > > > > > that
> > > > > > > > way
> > > > > > > > > is already there) to notify committerthat a test that they
> > > were
> > > > > > > involved
> > > > > > > > > with is failing
> > > > > > > > > That way, we will avoid disable tests like this one
> > > > > > > > > INVALID URI REMOVED
> > > > > > > > >
> > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > > > > > >
> > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > > > > > >
> > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > > > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e=
> ,
> > > > > > > > which
> > > > > > > > > are quite important to ensure Springboot messaging is
> > working.
> > > > > > > > >
> > > > > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > > > > > > ricardozanini@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > > > > > > <ci...@gmail.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Martin Cimbalek
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> > > > > > tzimanyi@apache.org
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi everyone,
> > > > > > > > > > > >
> > > > > > > > > > > > I want to propose a workflow for situations, when
> some
> > > > > > > > > > > > tests
> > > > > > are
> > > > > > > > > > failing
> > > > > > > > > > > > for a longer time. In such cases, my proposed
> workflow
> > > is:
> > > > > > > > > > > > - If a test or a set of tests is failing for two days
> > > > > > (nightlies
> > > > > > > or
> > > > > > > > > PR
> > > > > > > > > > > > checks), ignore those tests in the codebase, so they
> > are
> > > > > > > > > > > > not
> > > > > > > > > executed.
> > > > > > > > > > > > - File an issue in kie-issues repository, reporting
> the
> > > > > > > > > > > > test
> > > > > > > > > failures.
> > > > > > > > > > > > - If the issue is not resolved for half a year,
> delete
> > > > > > > > > > > > those
> > > > > > > tests
> > > > > > > > > from
> > > > > > > > > > > the
> > > > > > > > > > > > codebase.
> > > > > > > > > > > >
> > > > > > > > > > > > What do you think please? This should make sure all
> > > > > > > > > > > > failures
> > > > > > that
> > > > > > > > > don't
> > > > > > > > > > > get
> > > > > > > > > > > > fixed immediately after they occur get logged in the
> > > issue
> > > > > > > tracker,
> > > > > > > > > so
> > > > > > > > > > > they
> > > > > > > > > > > > can be appropriately handled and don't block
> unrelated
> > PR
> > > > > > checks
> > > > > > > > and
> > > > > > > > > > > > builds. It will also make sure that tests that are
> not
> > > > > > > > > > > > fixed
> > > > > > for
> > > > > > > a
> > > > > > > > > long
> > > > > > > > > > > > time (therefore they could be perceived as not
> > important)
> > > > > > > > > > > > are
> > > > > > > not a
> > > > > > > > > > > > maintenance burden for the future.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Tibor
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > > >
> > > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Toni Rikkola <tr...@redhat.com>.
What happened with the SpringBoot. I made the change and actually thought
the PR reviews would alert the right people and you could not get a PR in
without approval from the devs responsible for that area. This is just
since I don't know who is working on what and on what level. I don't know
how the wording in this rule could be set so that the right person has
actually been contacted.
One other thing to take note of is that this policy applies for all the
broken tests, not just flaky ones. So if you manage to get a bad commit in,
it is then ok to ignore all the tests that that commit broke and now the
responsibility of fixing the codes are on the person that wrote the test
that picks up the bugs or changes someone else introduced.
I know I like to over think these regulations, but due to the structure at
Apache we agree on rules and some can be taken advantage of. If the thing
mentioned above happens, there is not one great leader who can give an
order to fix the codes. Also if a community member gets a commit in, there
is no handle for us to get them to fix anything. It is the community's
problem after that.
I have to give this a -1, this has to target just flaky tests. If a commit
breaks the test, the right thing would be to revert ASAP. If we really need
to get something in that breaks tests in another location, and those tests
need to be redone, then we should vote for that change to go in as an
exception and ignore the broken tests in the same commit bundle.
Acknowledging there is no guarantee the tests commented out will ever be
fixed, even if they show a bug was introduced.
So for example in the current case. If the build is broken and the tests
are not flaky, we can vote for disabling those tests. I think that vote
will pass.
I hope we understand what this project is now. One person can make a big
mess if the guards are not set up correctly. Five more companies can come
in and start working on this, out ruling everyone we have now. The base has
to be solid for many reasons. Letting this change in as a rule opens us to
too much abuse, intentional or unintentional.
Toni
On Thu, Jan 25, 2024 at 2:59 PM Francisco Javier Tirado Sarti <
ftirados@redhat.com> wrote:
> +1
> Thanks Tibor
>
> On Thu, Jan 25, 2024 at 9:37 AM Tibor Zimányi <tz...@apache.org> wrote:
>
> > Hi,
> >
> > as I wrote in one of the previous e-mails in this thread:
> > "I think the notification of the involved commiter could be done as part
> of
> > creating of the issue to fix the tests. The person involved can be
> assigned
> > as an asignee. "
> >
> > To clarify, by involved commiter I meant either author of the test or the
> > person who is maintaining that part of the codebase.
> >
> > Hopefully this clarifies this.
> >
> > Tibor
> >
> > Dňa št 25. 1. 2024, 9:22 Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > napísal(a):
> >
> > > I agree with Josef, I realize the SpringBoot listener example test was
> > > disabled while fixing an issue related to it. If not, the test would
> have
> > > remained disabled.
> > >
> > > On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko
> <Jozef.Marko@ibm.com.invalid
> > >
> > > wrote:
> > >
> > > > Thank you for this discussion. I think disabling flaky tests in given
> > way
> > > > is reasonable.
> > > >
> > > > I have one related question, once similar flaky test is disabled,
> and a
> > > > ticket is reported. Who will be assigned to this ticket? Will it be
> the
> > > > test author?
> > > >
> > > > My point is, that usually only test author and few other engineers
> > > working
> > > > on the same component understand completely the given test, but
> > probably
> > > > they do not check the codebase daily if the test is still enabled.
> > > >
> > > > So, will be these people notified? Otherwise, most of these tickets
> > will
> > > > be probably closed after the proposed period without any test
> > update/fix.
> > > >
> > > > Jozef Marko
> > > > Software Developer
> > > > jozef.marko@ibm.com
> > > >
> > > > -----Original Message-----
> > > > From: Toni Rikkola <tr...@redhat.com>
> > > > Sent: Wednesday, January 24, 2024 3:58 PM
> > > > To: dev@kie.apache.org
> > > > Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing
> > > tests
> > > >
> > > > The reason I mentioned this was that we had this same decision made 8
> > > > years ago and shared it to the entire team.
> > > > It stated that anyone can disable any flaky test and make a ticket
> for
> > > it.
> > > > It was even more strict, you just had to have one flaky failure.
> > > >
> > > > So to me it looks like we are just stating the same thing again
> hoping
> > > for
> > > > a different outcome.
> > > >
> > > > But yes, to make it clear for everyone we should write it down and we
> > can
> > > > see how it goes and act on it later if needed.
> > > >
> > > > Toni
> > > >
> > > > On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org>
> > > wrote:
> > > >
> > > > > I think we don't need to enforce it or gatekeep this any way. The
> > > > > thing is, if we have the agreement that we want to have this
> official
> > > > > workflow, it should fix the problem you are mentioning Toni, that
> > > > > people are afraid to do so. If it will be officially stated, that
> > > > > failing tests can be disabled in the code, people should just do
> it.
> > > > > They can always point to this potential agreement. The best thing
> > > > > would be to even document it on Confluence. I will do that if we
> have
> > > an
> > > > agreement on this.
> > > > >
> > > > > Tibor
> > > > >
> > > > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com>
> > > napísal(a):
> > > > >
> > > > > > Not informing the author was my mistake on that PR. I should have
> > > > > > asked
> > > > > the
> > > > > > original author to review it.
> > > > > >
> > > > > > But the problem is not that we lack this procedure. The KIE team
> > > > > > agreed
> > > > > on
> > > > > > this in 2015. It is that maybe we lack the strength to do it,
> > > > > > especially when it is someone else's area of code. This needs
> some
> > > > > > sort of
> > > > > gatekeeper
> > > > > > or supervisor and time frames when the build quality and if these
> > > > > problems
> > > > > > were acted on are validated and measured.
> > > > > >
> > > > > > Meanwhile I offer myself to work on these and disable them, if
> > > > > > anyone
> > > > > wants
> > > > > > to point me to some flaky test I can act on them. I could even
> make
> > > > > > some system for measurement. We can then, or now, decide what we
> > > > > > then do with that data.
> > > > > >
> > > > > > Toni
> > > > > >
> > > > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <
> tzimanyi@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > I think the notification of the involved commiter could be done
> > as
> > > > > > > part
> > > > > > of
> > > > > > > creating of the issue to fix the tests. The person involved can
> > be
> > > > > > assigned
> > > > > > > as an asignee.
> > > > > > >
> > > > > > > T.
> > > > > > >
> > > > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > > > > > ftirados@redhat.com>
> > > > > > > napísal(a):
> > > > > > >
> > > > > > > > Ok, but I think we first need to establish a way (my
> apologies
> > > > > > > > is
> > > > > that
> > > > > > > way
> > > > > > > > is already there) to notify committerthat a test that they
> > were
> > > > > > involved
> > > > > > > > with is failing
> > > > > > > > That way, we will avoid disable tests like this one
> > > > > > > > INVALID URI REMOVED
> > > > > > > >
> > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > > > > >
> > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > > > > >
> > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= ,
> > > > > > > which
> > > > > > > > are quite important to ensure Springboot messaging is
> working.
> > > > > > > >
> > > > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > > > > > ricardozanini@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > > > > > <ci...@gmail.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Martin Cimbalek
> > > > > > > > > >
> > > > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> > > > > tzimanyi@apache.org
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi everyone,
> > > > > > > > > > >
> > > > > > > > > > > I want to propose a workflow for situations, when some
> > > > > > > > > > > tests
> > > > > are
> > > > > > > > > failing
> > > > > > > > > > > for a longer time. In such cases, my proposed workflow
> > is:
> > > > > > > > > > > - If a test or a set of tests is failing for two days
> > > > > (nightlies
> > > > > > or
> > > > > > > > PR
> > > > > > > > > > > checks), ignore those tests in the codebase, so they
> are
> > > > > > > > > > > not
> > > > > > > > executed.
> > > > > > > > > > > - File an issue in kie-issues repository, reporting the
> > > > > > > > > > > test
> > > > > > > > failures.
> > > > > > > > > > > - If the issue is not resolved for half a year, delete
> > > > > > > > > > > those
> > > > > > tests
> > > > > > > > from
> > > > > > > > > > the
> > > > > > > > > > > codebase.
> > > > > > > > > > >
> > > > > > > > > > > What do you think please? This should make sure all
> > > > > > > > > > > failures
> > > > > that
> > > > > > > > don't
> > > > > > > > > > get
> > > > > > > > > > > fixed immediately after they occur get logged in the
> > issue
> > > > > > tracker,
> > > > > > > > so
> > > > > > > > > > they
> > > > > > > > > > > can be appropriately handled and don't block unrelated
> PR
> > > > > checks
> > > > > > > and
> > > > > > > > > > > builds. It will also make sure that tests that are not
> > > > > > > > > > > fixed
> > > > > for
> > > > > > a
> > > > > > > > long
> > > > > > > > > > > time (therefore they could be perceived as not
> important)
> > > > > > > > > > > are
> > > > > > not a
> > > > > > > > > > > maintenance burden for the future.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Tibor
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > > For additional commands, e-mail: dev-help@kie.apache.org
> > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
+1
Thanks Tibor
On Thu, Jan 25, 2024 at 9:37 AM Tibor Zimányi <tz...@apache.org> wrote:
> Hi,
>
> as I wrote in one of the previous e-mails in this thread:
> "I think the notification of the involved commiter could be done as part of
> creating of the issue to fix the tests. The person involved can be assigned
> as an asignee. "
>
> To clarify, by involved commiter I meant either author of the test or the
> person who is maintaining that part of the codebase.
>
> Hopefully this clarifies this.
>
> Tibor
>
> Dňa št 25. 1. 2024, 9:22 Francisco Javier Tirado Sarti <
> ftirados@redhat.com>
> napísal(a):
>
> > I agree with Josef, I realize the SpringBoot listener example test was
> > disabled while fixing an issue related to it. If not, the test would have
> > remained disabled.
> >
> > On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko <Jozef.Marko@ibm.com.invalid
> >
> > wrote:
> >
> > > Thank you for this discussion. I think disabling flaky tests in given
> way
> > > is reasonable.
> > >
> > > I have one related question, once similar flaky test is disabled, and a
> > > ticket is reported. Who will be assigned to this ticket? Will it be the
> > > test author?
> > >
> > > My point is, that usually only test author and few other engineers
> > working
> > > on the same component understand completely the given test, but
> probably
> > > they do not check the codebase daily if the test is still enabled.
> > >
> > > So, will be these people notified? Otherwise, most of these tickets
> will
> > > be probably closed after the proposed period without any test
> update/fix.
> > >
> > > Jozef Marko
> > > Software Developer
> > > jozef.marko@ibm.com
> > >
> > > -----Original Message-----
> > > From: Toni Rikkola <tr...@redhat.com>
> > > Sent: Wednesday, January 24, 2024 3:58 PM
> > > To: dev@kie.apache.org
> > > Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing
> > tests
> > >
> > > The reason I mentioned this was that we had this same decision made 8
> > > years ago and shared it to the entire team.
> > > It stated that anyone can disable any flaky test and make a ticket for
> > it.
> > > It was even more strict, you just had to have one flaky failure.
> > >
> > > So to me it looks like we are just stating the same thing again hoping
> > for
> > > a different outcome.
> > >
> > > But yes, to make it clear for everyone we should write it down and we
> can
> > > see how it goes and act on it later if needed.
> > >
> > > Toni
> > >
> > > On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > I think we don't need to enforce it or gatekeep this any way. The
> > > > thing is, if we have the agreement that we want to have this official
> > > > workflow, it should fix the problem you are mentioning Toni, that
> > > > people are afraid to do so. If it will be officially stated, that
> > > > failing tests can be disabled in the code, people should just do it.
> > > > They can always point to this potential agreement. The best thing
> > > > would be to even document it on Confluence. I will do that if we have
> > an
> > > agreement on this.
> > > >
> > > > Tibor
> > > >
> > > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com>
> > napísal(a):
> > > >
> > > > > Not informing the author was my mistake on that PR. I should have
> > > > > asked
> > > > the
> > > > > original author to review it.
> > > > >
> > > > > But the problem is not that we lack this procedure. The KIE team
> > > > > agreed
> > > > on
> > > > > this in 2015. It is that maybe we lack the strength to do it,
> > > > > especially when it is someone else's area of code. This needs some
> > > > > sort of
> > > > gatekeeper
> > > > > or supervisor and time frames when the build quality and if these
> > > > problems
> > > > > were acted on are validated and measured.
> > > > >
> > > > > Meanwhile I offer myself to work on these and disable them, if
> > > > > anyone
> > > > wants
> > > > > to point me to some flaky test I can act on them. I could even make
> > > > > some system for measurement. We can then, or now, decide what we
> > > > > then do with that data.
> > > > >
> > > > > Toni
> > > > >
> > > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tzimanyi@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > I think the notification of the involved commiter could be done
> as
> > > > > > part
> > > > > of
> > > > > > creating of the issue to fix the tests. The person involved can
> be
> > > > > assigned
> > > > > > as an asignee.
> > > > > >
> > > > > > T.
> > > > > >
> > > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > > > > ftirados@redhat.com>
> > > > > > napísal(a):
> > > > > >
> > > > > > > Ok, but I think we first need to establish a way (my apologies
> > > > > > > is
> > > > that
> > > > > > way
> > > > > > > is already there) to notify committerthat a test that they
> were
> > > > > involved
> > > > > > > with is failing
> > > > > > > That way, we will avoid disable tests like this one
> > > > > > > INVALID URI REMOVED
> > > > > > >
> apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > > > >
> c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > > > >
> jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= ,
> > > > > > which
> > > > > > > are quite important to ensure Springboot messaging is working.
> > > > > > >
> > > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > > > > ricardozanini@gmail.com> wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > > > > <ci...@gmail.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Martin Cimbalek
> > > > > > > > >
> > > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> > > > tzimanyi@apache.org
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi everyone,
> > > > > > > > > >
> > > > > > > > > > I want to propose a workflow for situations, when some
> > > > > > > > > > tests
> > > > are
> > > > > > > > failing
> > > > > > > > > > for a longer time. In such cases, my proposed workflow
> is:
> > > > > > > > > > - If a test or a set of tests is failing for two days
> > > > (nightlies
> > > > > or
> > > > > > > PR
> > > > > > > > > > checks), ignore those tests in the codebase, so they are
> > > > > > > > > > not
> > > > > > > executed.
> > > > > > > > > > - File an issue in kie-issues repository, reporting the
> > > > > > > > > > test
> > > > > > > failures.
> > > > > > > > > > - If the issue is not resolved for half a year, delete
> > > > > > > > > > those
> > > > > tests
> > > > > > > from
> > > > > > > > > the
> > > > > > > > > > codebase.
> > > > > > > > > >
> > > > > > > > > > What do you think please? This should make sure all
> > > > > > > > > > failures
> > > > that
> > > > > > > don't
> > > > > > > > > get
> > > > > > > > > > fixed immediately after they occur get logged in the
> issue
> > > > > tracker,
> > > > > > > so
> > > > > > > > > they
> > > > > > > > > > can be appropriately handled and don't block unrelated PR
> > > > checks
> > > > > > and
> > > > > > > > > > builds. It will also make sure that tests that are not
> > > > > > > > > > fixed
> > > > for
> > > > > a
> > > > > > > long
> > > > > > > > > > time (therefore they could be perceived as not important)
> > > > > > > > > > are
> > > > > not a
> > > > > > > > > > maintenance burden for the future.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Tibor
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > > For additional commands, e-mail: dev-help@kie.apache.org
> > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Tibor Zimányi <tz...@apache.org>.
Hi,
as I wrote in one of the previous e-mails in this thread:
"I think the notification of the involved commiter could be done as part of
creating of the issue to fix the tests. The person involved can be assigned
as an asignee. "
To clarify, by involved commiter I meant either author of the test or the
person who is maintaining that part of the codebase.
Hopefully this clarifies this.
Tibor
Dňa št 25. 1. 2024, 9:22 Francisco Javier Tirado Sarti <ft...@redhat.com>
napísal(a):
> I agree with Josef, I realize the SpringBoot listener example test was
> disabled while fixing an issue related to it. If not, the test would have
> remained disabled.
>
> On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko <Jo...@ibm.com.invalid>
> wrote:
>
> > Thank you for this discussion. I think disabling flaky tests in given way
> > is reasonable.
> >
> > I have one related question, once similar flaky test is disabled, and a
> > ticket is reported. Who will be assigned to this ticket? Will it be the
> > test author?
> >
> > My point is, that usually only test author and few other engineers
> working
> > on the same component understand completely the given test, but probably
> > they do not check the codebase daily if the test is still enabled.
> >
> > So, will be these people notified? Otherwise, most of these tickets will
> > be probably closed after the proposed period without any test update/fix.
> >
> > Jozef Marko
> > Software Developer
> > jozef.marko@ibm.com
> >
> > -----Original Message-----
> > From: Toni Rikkola <tr...@redhat.com>
> > Sent: Wednesday, January 24, 2024 3:58 PM
> > To: dev@kie.apache.org
> > Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing
> tests
> >
> > The reason I mentioned this was that we had this same decision made 8
> > years ago and shared it to the entire team.
> > It stated that anyone can disable any flaky test and make a ticket for
> it.
> > It was even more strict, you just had to have one flaky failure.
> >
> > So to me it looks like we are just stating the same thing again hoping
> for
> > a different outcome.
> >
> > But yes, to make it clear for everyone we should write it down and we can
> > see how it goes and act on it later if needed.
> >
> > Toni
> >
> > On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > I think we don't need to enforce it or gatekeep this any way. The
> > > thing is, if we have the agreement that we want to have this official
> > > workflow, it should fix the problem you are mentioning Toni, that
> > > people are afraid to do so. If it will be officially stated, that
> > > failing tests can be disabled in the code, people should just do it.
> > > They can always point to this potential agreement. The best thing
> > > would be to even document it on Confluence. I will do that if we have
> an
> > agreement on this.
> > >
> > > Tibor
> > >
> > > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com>
> napísal(a):
> > >
> > > > Not informing the author was my mistake on that PR. I should have
> > > > asked
> > > the
> > > > original author to review it.
> > > >
> > > > But the problem is not that we lack this procedure. The KIE team
> > > > agreed
> > > on
> > > > this in 2015. It is that maybe we lack the strength to do it,
> > > > especially when it is someone else's area of code. This needs some
> > > > sort of
> > > gatekeeper
> > > > or supervisor and time frames when the build quality and if these
> > > problems
> > > > were acted on are validated and measured.
> > > >
> > > > Meanwhile I offer myself to work on these and disable them, if
> > > > anyone
> > > wants
> > > > to point me to some flaky test I can act on them. I could even make
> > > > some system for measurement. We can then, or now, decide what we
> > > > then do with that data.
> > > >
> > > > Toni
> > > >
> > > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org>
> > > wrote:
> > > >
> > > > > I think the notification of the involved commiter could be done as
> > > > > part
> > > > of
> > > > > creating of the issue to fix the tests. The person involved can be
> > > > assigned
> > > > > as an asignee.
> > > > >
> > > > > T.
> > > > >
> > > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > > > ftirados@redhat.com>
> > > > > napísal(a):
> > > > >
> > > > > > Ok, but I think we first need to establish a way (my apologies
> > > > > > is
> > > that
> > > > > way
> > > > > > is already there) to notify committerthat a test that they were
> > > > involved
> > > > > > with is failing
> > > > > > That way, we will avoid disable tests like this one
> > > > > > INVALID URI REMOVED
> > > > > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= ,
> > > > > which
> > > > > > are quite important to ensure Springboot messaging is working.
> > > > > >
> > > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > > > ricardozanini@gmail.com> wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > > > <ci...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Martin Cimbalek
> > > > > > > >
> > > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> > > tzimanyi@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi everyone,
> > > > > > > > >
> > > > > > > > > I want to propose a workflow for situations, when some
> > > > > > > > > tests
> > > are
> > > > > > > failing
> > > > > > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > > > > > - If a test or a set of tests is failing for two days
> > > (nightlies
> > > > or
> > > > > > PR
> > > > > > > > > checks), ignore those tests in the codebase, so they are
> > > > > > > > > not
> > > > > > executed.
> > > > > > > > > - File an issue in kie-issues repository, reporting the
> > > > > > > > > test
> > > > > > failures.
> > > > > > > > > - If the issue is not resolved for half a year, delete
> > > > > > > > > those
> > > > tests
> > > > > > from
> > > > > > > > the
> > > > > > > > > codebase.
> > > > > > > > >
> > > > > > > > > What do you think please? This should make sure all
> > > > > > > > > failures
> > > that
> > > > > > don't
> > > > > > > > get
> > > > > > > > > fixed immediately after they occur get logged in the issue
> > > > tracker,
> > > > > > so
> > > > > > > > they
> > > > > > > > > can be appropriately handled and don't block unrelated PR
> > > checks
> > > > > and
> > > > > > > > > builds. It will also make sure that tests that are not
> > > > > > > > > fixed
> > > for
> > > > a
> > > > > > long
> > > > > > > > > time (therefore they could be perceived as not important)
> > > > > > > > > are
> > > > not a
> > > > > > > > > maintenance burden for the future.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Tibor
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> > For additional commands, e-mail: dev-help@kie.apache.org
> >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
I agree with Josef, I realize the SpringBoot listener example test was
disabled while fixing an issue related to it. If not, the test would have
remained disabled.
On Thu, Jan 25, 2024 at 5:51 AM Jozef Marko <Jo...@ibm.com.invalid>
wrote:
> Thank you for this discussion. I think disabling flaky tests in given way
> is reasonable.
>
> I have one related question, once similar flaky test is disabled, and a
> ticket is reported. Who will be assigned to this ticket? Will it be the
> test author?
>
> My point is, that usually only test author and few other engineers working
> on the same component understand completely the given test, but probably
> they do not check the codebase daily if the test is still enabled.
>
> So, will be these people notified? Otherwise, most of these tickets will
> be probably closed after the proposed period without any test update/fix.
>
> Jozef Marko
> Software Developer
> jozef.marko@ibm.com
>
> -----Original Message-----
> From: Toni Rikkola <tr...@redhat.com>
> Sent: Wednesday, January 24, 2024 3:58 PM
> To: dev@kie.apache.org
> Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing tests
>
> The reason I mentioned this was that we had this same decision made 8
> years ago and shared it to the entire team.
> It stated that anyone can disable any flaky test and make a ticket for it.
> It was even more strict, you just had to have one flaky failure.
>
> So to me it looks like we are just stating the same thing again hoping for
> a different outcome.
>
> But yes, to make it clear for everyone we should write it down and we can
> see how it goes and act on it later if needed.
>
> Toni
>
> On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > I think we don't need to enforce it or gatekeep this any way. The
> > thing is, if we have the agreement that we want to have this official
> > workflow, it should fix the problem you are mentioning Toni, that
> > people are afraid to do so. If it will be officially stated, that
> > failing tests can be disabled in the code, people should just do it.
> > They can always point to this potential agreement. The best thing
> > would be to even document it on Confluence. I will do that if we have an
> agreement on this.
> >
> > Tibor
> >
> > Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com> napísal(a):
> >
> > > Not informing the author was my mistake on that PR. I should have
> > > asked
> > the
> > > original author to review it.
> > >
> > > But the problem is not that we lack this procedure. The KIE team
> > > agreed
> > on
> > > this in 2015. It is that maybe we lack the strength to do it,
> > > especially when it is someone else's area of code. This needs some
> > > sort of
> > gatekeeper
> > > or supervisor and time frames when the build quality and if these
> > problems
> > > were acted on are validated and measured.
> > >
> > > Meanwhile I offer myself to work on these and disable them, if
> > > anyone
> > wants
> > > to point me to some flaky test I can act on them. I could even make
> > > some system for measurement. We can then, or now, decide what we
> > > then do with that data.
> > >
> > > Toni
> > >
> > > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > I think the notification of the involved commiter could be done as
> > > > part
> > > of
> > > > creating of the issue to fix the tests. The person involved can be
> > > assigned
> > > > as an asignee.
> > > >
> > > > T.
> > > >
> > > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > > ftirados@redhat.com>
> > > > napísal(a):
> > > >
> > > > > Ok, but I think we first need to establish a way (my apologies
> > > > > is
> > that
> > > > way
> > > > > is already there) to notify committerthat a test that they were
> > > involved
> > > > > with is failing
> > > > > That way, we will avoid disable tests like this one
> > > > > INVALID URI REMOVED
> > > > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= ,
> > > > which
> > > > > are quite important to ensure Springboot messaging is working.
> > > > >
> > > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > > ricardozanini@gmail.com> wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > > <ci...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > Regards,
> > > > > > > Martin Cimbalek
> > > > > > >
> > > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> > tzimanyi@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > I want to propose a workflow for situations, when some
> > > > > > > > tests
> > are
> > > > > > failing
> > > > > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > > > > - If a test or a set of tests is failing for two days
> > (nightlies
> > > or
> > > > > PR
> > > > > > > > checks), ignore those tests in the codebase, so they are
> > > > > > > > not
> > > > > executed.
> > > > > > > > - File an issue in kie-issues repository, reporting the
> > > > > > > > test
> > > > > failures.
> > > > > > > > - If the issue is not resolved for half a year, delete
> > > > > > > > those
> > > tests
> > > > > from
> > > > > > > the
> > > > > > > > codebase.
> > > > > > > >
> > > > > > > > What do you think please? This should make sure all
> > > > > > > > failures
> > that
> > > > > don't
> > > > > > > get
> > > > > > > > fixed immediately after they occur get logged in the issue
> > > tracker,
> > > > > so
> > > > > > > they
> > > > > > > > can be appropriately handled and don't block unrelated PR
> > checks
> > > > and
> > > > > > > > builds. It will also make sure that tests that are not
> > > > > > > > fixed
> > for
> > > a
> > > > > long
> > > > > > > > time (therefore they could be perceived as not important)
> > > > > > > > are
> > > not a
> > > > > > > > maintenance burden for the future.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Tibor
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@kie.apache.org
> For additional commands, e-mail: dev-help@kie.apache.org
>
>
RE: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Jozef Marko <Jo...@ibm.com.INVALID>.
Thank you for this discussion. I think disabling flaky tests in given way is reasonable.
I have one related question, once similar flaky test is disabled, and a ticket is reported. Who will be assigned to this ticket? Will it be the test author?
My point is, that usually only test author and few other engineers working on the same component understand completely the given test, but probably they do not check the codebase daily if the test is still enabled.
So, will be these people notified? Otherwise, most of these tickets will be probably closed after the proposed period without any test update/fix.
Jozef Marko
Software Developer
jozef.marko@ibm.com
-----Original Message-----
From: Toni Rikkola <tr...@redhat.com>
Sent: Wednesday, January 24, 2024 3:58 PM
To: dev@kie.apache.org
Subject: [EXTERNAL] Re: [PROPOSAL] Workflow for ignoring long failing tests
The reason I mentioned this was that we had this same decision made 8 years ago and shared it to the entire team.
It stated that anyone can disable any flaky test and make a ticket for it.
It was even more strict, you just had to have one flaky failure.
So to me it looks like we are just stating the same thing again hoping for a different outcome.
But yes, to make it clear for everyone we should write it down and we can see how it goes and act on it later if needed.
Toni
On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org> wrote:
> I think we don't need to enforce it or gatekeep this any way. The
> thing is, if we have the agreement that we want to have this official
> workflow, it should fix the problem you are mentioning Toni, that
> people are afraid to do so. If it will be officially stated, that
> failing tests can be disabled in the code, people should just do it.
> They can always point to this potential agreement. The best thing
> would be to even document it on Confluence. I will do that if we have an agreement on this.
>
> Tibor
>
> Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com> napísal(a):
>
> > Not informing the author was my mistake on that PR. I should have
> > asked
> the
> > original author to review it.
> >
> > But the problem is not that we lack this procedure. The KIE team
> > agreed
> on
> > this in 2015. It is that maybe we lack the strength to do it,
> > especially when it is someone else's area of code. This needs some
> > sort of
> gatekeeper
> > or supervisor and time frames when the build quality and if these
> problems
> > were acted on are validated and measured.
> >
> > Meanwhile I offer myself to work on these and disable them, if
> > anyone
> wants
> > to point me to some flaky test I can act on them. I could even make
> > some system for measurement. We can then, or now, decide what we
> > then do with that data.
> >
> > Toni
> >
> > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > I think the notification of the involved commiter could be done as
> > > part
> > of
> > > creating of the issue to fix the tests. The person involved can be
> > assigned
> > > as an asignee.
> > >
> > > T.
> > >
> > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > napísal(a):
> > >
> > > > Ok, but I think we first need to establish a way (my apologies
> > > > is
> that
> > > way
> > > > is already there) to notify committerthat a test that they were
> > involved
> > > > with is failing
> > > > That way, we will avoid disable tests like this one
> > > > INVALID URI REMOVED
> > > > apache_incubator-2Dkie-2Dkogito-2Dexamples_issues_1831&d=DwIFaQ&
> > > > c=jf_iaSHvJObTbx-siA1ZOg&r=Tvtr6trQuIEn20iSHtFwcpynBt8ViCybz4kYp
> > > > jmDDUQ&m=C6gAvciwgXz7uJX__01AVZ3YHxSotIhuktwbnBnNuTvnLRU-NHMRqOd
> > > > NENKhsqKe&s=8p3WYlpcXfcLl1n4BnrSpBWNIdPIUhslTPsb41eE4IM&e= ,
> > > which
> > > > are quite important to ensure Springboot messaging is working.
> > > >
> > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > ricardozanini@gmail.com> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek
> > > > > <ci...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Regards,
> > > > > > Martin Cimbalek
> > > > > >
> > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> tzimanyi@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I want to propose a workflow for situations, when some
> > > > > > > tests
> are
> > > > > failing
> > > > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > > > - If a test or a set of tests is failing for two days
> (nightlies
> > or
> > > > PR
> > > > > > > checks), ignore those tests in the codebase, so they are
> > > > > > > not
> > > > executed.
> > > > > > > - File an issue in kie-issues repository, reporting the
> > > > > > > test
> > > > failures.
> > > > > > > - If the issue is not resolved for half a year, delete
> > > > > > > those
> > tests
> > > > from
> > > > > > the
> > > > > > > codebase.
> > > > > > >
> > > > > > > What do you think please? This should make sure all
> > > > > > > failures
> that
> > > > don't
> > > > > > get
> > > > > > > fixed immediately after they occur get logged in the issue
> > tracker,
> > > > so
> > > > > > they
> > > > > > > can be appropriately handled and don't block unrelated PR
> checks
> > > and
> > > > > > > builds. It will also make sure that tests that are not
> > > > > > > fixed
> for
> > a
> > > > long
> > > > > > > time (therefore they could be perceived as not important)
> > > > > > > are
> > not a
> > > > > > > maintenance burden for the future.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Tibor
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Toni Rikkola <tr...@redhat.com>.
The reason I mentioned this was that we had this same decision made 8 years
ago and shared it to the entire team.
It stated that anyone can disable any flaky test and make a ticket for it.
It was even more strict, you just had to have one flaky failure.
So to me it looks like we are just stating the same thing again hoping for
a different outcome.
But yes, to make it clear for everyone we should write it down and we can
see how it goes and act on it later if needed.
Toni
On Wed, Jan 24, 2024 at 2:09 PM Tibor Zimányi <tz...@apache.org> wrote:
> I think we don't need to enforce it or gatekeep this any way. The thing is,
> if we have the agreement that we want to have this official workflow, it
> should fix the problem you are mentioning Toni, that people are afraid to
> do so. If it will be officially stated, that failing tests can be disabled
> in the code, people should just do it. They can always point to this
> potential agreement. The best thing would be to even document it on
> Confluence. I will do that if we have an agreement on this.
>
> Tibor
>
> Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com> napísal(a):
>
> > Not informing the author was my mistake on that PR. I should have asked
> the
> > original author to review it.
> >
> > But the problem is not that we lack this procedure. The KIE team agreed
> on
> > this in 2015. It is that maybe we lack the strength to do it, especially
> > when it is someone else's area of code. This needs some sort of
> gatekeeper
> > or supervisor and time frames when the build quality and if these
> problems
> > were acted on are validated and measured.
> >
> > Meanwhile I offer myself to work on these and disable them, if anyone
> wants
> > to point me to some flaky test I can act on them. I could even make some
> > system for measurement. We can then, or now, decide what we then do with
> > that data.
> >
> > Toni
> >
> > On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > I think the notification of the involved commiter could be done as part
> > of
> > > creating of the issue to fix the tests. The person involved can be
> > assigned
> > > as an asignee.
> > >
> > > T.
> > >
> > > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > > ftirados@redhat.com>
> > > napísal(a):
> > >
> > > > Ok, but I think we first need to establish a way (my apologies is
> that
> > > way
> > > > is already there) to notify committerthat a test that they were
> > involved
> > > > with is failing
> > > > That way, we will avoid disable tests like this one
> > > > https://github.com/apache/incubator-kie-kogito-examples/issues/1831,
> > > which
> > > > are quite important to ensure Springboot messaging is working.
> > > >
> > > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > > ricardozanini@gmail.com> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com>
> > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Regards,
> > > > > > Martin Cimbalek
> > > > > >
> > > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <
> tzimanyi@apache.org
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I want to propose a workflow for situations, when some tests
> are
> > > > > failing
> > > > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > > > - If a test or a set of tests is failing for two days
> (nightlies
> > or
> > > > PR
> > > > > > > checks), ignore those tests in the codebase, so they are not
> > > > executed.
> > > > > > > - File an issue in kie-issues repository, reporting the test
> > > > failures.
> > > > > > > - If the issue is not resolved for half a year, delete those
> > tests
> > > > from
> > > > > > the
> > > > > > > codebase.
> > > > > > >
> > > > > > > What do you think please? This should make sure all failures
> that
> > > > don't
> > > > > > get
> > > > > > > fixed immediately after they occur get logged in the issue
> > tracker,
> > > > so
> > > > > > they
> > > > > > > can be appropriately handled and don't block unrelated PR
> checks
> > > and
> > > > > > > builds. It will also make sure that tests that are not fixed
> for
> > a
> > > > long
> > > > > > > time (therefore they could be perceived as not important) are
> > not a
> > > > > > > maintenance burden for the future.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Tibor
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Tibor Zimányi <tz...@apache.org>.
I think we don't need to enforce it or gatekeep this any way. The thing is,
if we have the agreement that we want to have this official workflow, it
should fix the problem you are mentioning Toni, that people are afraid to
do so. If it will be officially stated, that failing tests can be disabled
in the code, people should just do it. They can always point to this
potential agreement. The best thing would be to even document it on
Confluence. I will do that if we have an agreement on this.
Tibor
Dňa st 24. 1. 2024, 12:40 Toni Rikkola <tr...@redhat.com> napísal(a):
> Not informing the author was my mistake on that PR. I should have asked the
> original author to review it.
>
> But the problem is not that we lack this procedure. The KIE team agreed on
> this in 2015. It is that maybe we lack the strength to do it, especially
> when it is someone else's area of code. This needs some sort of gatekeeper
> or supervisor and time frames when the build quality and if these problems
> were acted on are validated and measured.
>
> Meanwhile I offer myself to work on these and disable them, if anyone wants
> to point me to some flaky test I can act on them. I could even make some
> system for measurement. We can then, or now, decide what we then do with
> that data.
>
> Toni
>
> On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > I think the notification of the involved commiter could be done as part
> of
> > creating of the issue to fix the tests. The person involved can be
> assigned
> > as an asignee.
> >
> > T.
> >
> > Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> > ftirados@redhat.com>
> > napísal(a):
> >
> > > Ok, but I think we first need to establish a way (my apologies is that
> > way
> > > is already there) to notify committerthat a test that they were
> involved
> > > with is failing
> > > That way, we will avoid disable tests like this one
> > > https://github.com/apache/incubator-kie-kogito-examples/issues/1831,
> > which
> > > are quite important to ensure Springboot messaging is working.
> > >
> > > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > > ricardozanini@gmail.com> wrote:
> > >
> > > > +1
> > > >
> > > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com>
> > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Regards,
> > > > > Martin Cimbalek
> > > > >
> > > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tzimanyi@apache.org
> >
> > > > wrote:
> > > > >
> > > > > > Hi everyone,
> > > > > >
> > > > > > I want to propose a workflow for situations, when some tests are
> > > > failing
> > > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > > - If a test or a set of tests is failing for two days (nightlies
> or
> > > PR
> > > > > > checks), ignore those tests in the codebase, so they are not
> > > executed.
> > > > > > - File an issue in kie-issues repository, reporting the test
> > > failures.
> > > > > > - If the issue is not resolved for half a year, delete those
> tests
> > > from
> > > > > the
> > > > > > codebase.
> > > > > >
> > > > > > What do you think please? This should make sure all failures that
> > > don't
> > > > > get
> > > > > > fixed immediately after they occur get logged in the issue
> tracker,
> > > so
> > > > > they
> > > > > > can be appropriately handled and don't block unrelated PR checks
> > and
> > > > > > builds. It will also make sure that tests that are not fixed for
> a
> > > long
> > > > > > time (therefore they could be perceived as not important) are
> not a
> > > > > > maintenance burden for the future.
> > > > > >
> > > > > > Best regards,
> > > > > > Tibor
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Toni Rikkola <tr...@redhat.com>.
Not informing the author was my mistake on that PR. I should have asked the
original author to review it.
But the problem is not that we lack this procedure. The KIE team agreed on
this in 2015. It is that maybe we lack the strength to do it, especially
when it is someone else's area of code. This needs some sort of gatekeeper
or supervisor and time frames when the build quality and if these problems
were acted on are validated and measured.
Meanwhile I offer myself to work on these and disable them, if anyone wants
to point me to some flaky test I can act on them. I could even make some
system for measurement. We can then, or now, decide what we then do with
that data.
Toni
On Tue, Jan 23, 2024 at 8:04 PM Tibor Zimányi <tz...@apache.org> wrote:
> I think the notification of the involved commiter could be done as part of
> creating of the issue to fix the tests. The person involved can be assigned
> as an asignee.
>
> T.
>
> Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <
> ftirados@redhat.com>
> napísal(a):
>
> > Ok, but I think we first need to establish a way (my apologies is that
> way
> > is already there) to notify committerthat a test that they were involved
> > with is failing
> > That way, we will avoid disable tests like this one
> > https://github.com/apache/incubator-kie-kogito-examples/issues/1831,
> which
> > are quite important to ensure Springboot messaging is working.
> >
> > On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> > ricardozanini@gmail.com> wrote:
> >
> > > +1
> > >
> > > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com>
> > wrote:
> > >
> > > > +1
> > > >
> > > > Regards,
> > > > Martin Cimbalek
> > > >
> > > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org>
> > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > I want to propose a workflow for situations, when some tests are
> > > failing
> > > > > for a longer time. In such cases, my proposed workflow is:
> > > > > - If a test or a set of tests is failing for two days (nightlies or
> > PR
> > > > > checks), ignore those tests in the codebase, so they are not
> > executed.
> > > > > - File an issue in kie-issues repository, reporting the test
> > failures.
> > > > > - If the issue is not resolved for half a year, delete those tests
> > from
> > > > the
> > > > > codebase.
> > > > >
> > > > > What do you think please? This should make sure all failures that
> > don't
> > > > get
> > > > > fixed immediately after they occur get logged in the issue tracker,
> > so
> > > > they
> > > > > can be appropriately handled and don't block unrelated PR checks
> and
> > > > > builds. It will also make sure that tests that are not fixed for a
> > long
> > > > > time (therefore they could be perceived as not important) are not a
> > > > > maintenance burden for the future.
> > > > >
> > > > > Best regards,
> > > > > Tibor
> > > > >
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Tibor Zimányi <tz...@apache.org>.
I think the notification of the involved commiter could be done as part of
creating of the issue to fix the tests. The person involved can be assigned
as an asignee.
T.
Dňa ut 23. 1. 2024, 18:50 Francisco Javier Tirado Sarti <ft...@redhat.com>
napísal(a):
> Ok, but I think we first need to establish a way (my apologies is that way
> is already there) to notify committerthat a test that they were involved
> with is failing
> That way, we will avoid disable tests like this one
> https://github.com/apache/incubator-kie-kogito-examples/issues/1831, which
> are quite important to ensure Springboot messaging is working.
>
> On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
> ricardozanini@gmail.com> wrote:
>
> > +1
> >
> > On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com>
> wrote:
> >
> > > +1
> > >
> > > Regards,
> > > Martin Cimbalek
> > >
> > > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org>
> > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > I want to propose a workflow for situations, when some tests are
> > failing
> > > > for a longer time. In such cases, my proposed workflow is:
> > > > - If a test or a set of tests is failing for two days (nightlies or
> PR
> > > > checks), ignore those tests in the codebase, so they are not
> executed.
> > > > - File an issue in kie-issues repository, reporting the test
> failures.
> > > > - If the issue is not resolved for half a year, delete those tests
> from
> > > the
> > > > codebase.
> > > >
> > > > What do you think please? This should make sure all failures that
> don't
> > > get
> > > > fixed immediately after they occur get logged in the issue tracker,
> so
> > > they
> > > > can be appropriately handled and don't block unrelated PR checks and
> > > > builds. It will also make sure that tests that are not fixed for a
> long
> > > > time (therefore they could be perceived as not important) are not a
> > > > maintenance burden for the future.
> > > >
> > > > Best regards,
> > > > Tibor
> > > >
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Francisco Javier Tirado Sarti <ft...@redhat.com>.
Ok, but I think we first need to establish a way (my apologies is that way
is already there) to notify committerthat a test that they were involved
with is failing
That way, we will avoid disable tests like this one
https://github.com/apache/incubator-kie-kogito-examples/issues/1831, which
are quite important to ensure Springboot messaging is working.
On Tue, Jan 23, 2024 at 6:38 PM ricardo zanini fernandes <
ricardozanini@gmail.com> wrote:
> +1
>
> On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com> wrote:
>
> > +1
> >
> > Regards,
> > Martin Cimbalek
> >
> > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I want to propose a workflow for situations, when some tests are
> failing
> > > for a longer time. In such cases, my proposed workflow is:
> > > - If a test or a set of tests is failing for two days (nightlies or PR
> > > checks), ignore those tests in the codebase, so they are not executed.
> > > - File an issue in kie-issues repository, reporting the test failures.
> > > - If the issue is not resolved for half a year, delete those tests from
> > the
> > > codebase.
> > >
> > > What do you think please? This should make sure all failures that don't
> > get
> > > fixed immediately after they occur get logged in the issue tracker, so
> > they
> > > can be appropriately handled and don't block unrelated PR checks and
> > > builds. It will also make sure that tests that are not fixed for a long
> > > time (therefore they could be perceived as not important) are not a
> > > maintenance burden for the future.
> > >
> > > Best regards,
> > > Tibor
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Enrique Gonzalez Martinez <eg...@apache.org>.
+1
El mar, 23 ene 2024, 18:38, ricardo zanini fernandes <
ricardozanini@gmail.com> escribió:
> +1
>
> On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com> wrote:
>
> > +1
> >
> > Regards,
> > Martin Cimbalek
> >
> > On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org>
> wrote:
> >
> > > Hi everyone,
> > >
> > > I want to propose a workflow for situations, when some tests are
> failing
> > > for a longer time. In such cases, my proposed workflow is:
> > > - If a test or a set of tests is failing for two days (nightlies or PR
> > > checks), ignore those tests in the codebase, so they are not executed.
> > > - File an issue in kie-issues repository, reporting the test failures.
> > > - If the issue is not resolved for half a year, delete those tests from
> > the
> > > codebase.
> > >
> > > What do you think please? This should make sure all failures that don't
> > get
> > > fixed immediately after they occur get logged in the issue tracker, so
> > they
> > > can be appropriately handled and don't block unrelated PR checks and
> > > builds. It will also make sure that tests that are not fixed for a long
> > > time (therefore they could be perceived as not important) are not a
> > > maintenance burden for the future.
> > >
> > > Best regards,
> > > Tibor
> > >
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by ricardo zanini fernandes <ri...@gmail.com>.
+1
On Tue, Jan 23, 2024 at 1:03 PM Martin Cimbálek <ci...@gmail.com> wrote:
> +1
>
> Regards,
> Martin Cimbalek
>
> On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org> wrote:
>
> > Hi everyone,
> >
> > I want to propose a workflow for situations, when some tests are failing
> > for a longer time. In such cases, my proposed workflow is:
> > - If a test or a set of tests is failing for two days (nightlies or PR
> > checks), ignore those tests in the codebase, so they are not executed.
> > - File an issue in kie-issues repository, reporting the test failures.
> > - If the issue is not resolved for half a year, delete those tests from
> the
> > codebase.
> >
> > What do you think please? This should make sure all failures that don't
> get
> > fixed immediately after they occur get logged in the issue tracker, so
> they
> > can be appropriately handled and don't block unrelated PR checks and
> > builds. It will also make sure that tests that are not fixed for a long
> > time (therefore they could be perceived as not important) are not a
> > maintenance burden for the future.
> >
> > Best regards,
> > Tibor
> >
>
Re: [PROPOSAL] Workflow for ignoring long failing tests
Posted by Martin Cimbálek <ci...@gmail.com>.
+1
Regards,
Martin Cimbalek
On Tue, Jan 23, 2024 at 3:36 PM Tibor Zimányi <tz...@apache.org> wrote:
> Hi everyone,
>
> I want to propose a workflow for situations, when some tests are failing
> for a longer time. In such cases, my proposed workflow is:
> - If a test or a set of tests is failing for two days (nightlies or PR
> checks), ignore those tests in the codebase, so they are not executed.
> - File an issue in kie-issues repository, reporting the test failures.
> - If the issue is not resolved for half a year, delete those tests from the
> codebase.
>
> What do you think please? This should make sure all failures that don't get
> fixed immediately after they occur get logged in the issue tracker, so they
> can be appropriately handled and don't block unrelated PR checks and
> builds. It will also make sure that tests that are not fixed for a long
> time (therefore they could be perceived as not important) are not a
> maintenance burden for the future.
>
> Best regards,
> Tibor
>