Unstable tests ruin our days!

[Zoltan Haindrich <ki...@rxd.hu>]

Hey all!

Recently the number of unstable tests have rised above the "usual" level...
These are usually hard to fix - and in most cases need a deep dive in the area where the test operates.
Because of that I tend to just reattach the patch to the jira to get another run in a day or so...

The downside of the above approach is that reattaching on unrelated failures has a positive hivqqa queuesize coefficient.
There is another downside which might not be obvious first: it reduces the trust in the system and as a result there were cases when I did reattach the patch; but it was a 
genuine failure...it seemed unrelated; but actually it was.

Instead of continuing to reattach patches every day; I would like to propose a way to handle them:

* check that the falling test has nothing to do with the actual patch
   * it's important to be able to run test on our machines - but the most important is to maintain that HiveQA is able to run them successfully; for this reason I think 
having 2 HiveQA runs for the same changeset where in one of them the unstable test fails is the best
   * you can search the jira for the testcase and look if other patches have also bumped into it
   * ?
* add a comment about that you are about the disable the test in HIVE-22621 and commit it
   * I think it would be ok to skip the regular code change process
* create a new subtask under HIVE-22619 with the details you know about the falling testcase
* (resubmit your patch)

What do you think?

cheers,
Zoltan

Re: Unstable tests ruin our days!

[Peter Vary <pv...@cloudera.com.INVALID>]

+1 for disabling flakies

Or a flaky hunt? I would pay a beer for every fixed flaky test  for the
contributor, and another 2 if it was actually a bug... :)
(3 years ago when we were actively fixing flakies 1 in every 3 were real
bugs :( )

Zoltan Haindrich <ki...@rxd.hu> ezt írta (időpont: 2019. dec. 11., Sze
10:17):

> Hey all!
>
> Recently the number of unstable tests have rised above the "usual" level...
> These are usually hard to fix - and in most cases need a deep dive in the
> area where the test operates.
> Because of that I tend to just reattach the patch to the jira to get
> another run in a day or so...
>
> The downside of the above approach is that reattaching on unrelated
> failures has a positive hivqqa queuesize coefficient.
> There is another downside which might not be obvious first: it reduces the
> trust in the system and as a result there were cases when I did reattach
> the patch; but it was a
> genuine failure...it seemed unrelated; but actually it was.
>
> Instead of continuing to reattach patches every day; I would like to
> propose a way to handle them:
>
> * check that the falling test has nothing to do with the actual patch
>    * it's important to be able to run test on our machines - but the most
> important is to maintain that HiveQA is able to run them successfully; for
> this reason I think
> having 2 HiveQA runs for the same changeset where in one of them the
> unstable test fails is the best
>    * you can search the jira for the testcase and look if other patches
> have also bumped into it
>    * ?
> * add a comment about that you are about the disable the test in
> HIVE-22621 and commit it
>    * I think it would be ok to skip the regular code change process
> * create a new subtask under HIVE-22619 with the details you know about
> the falling testcase
> * (resubmit your patch)
>
> What do you think?
>
> cheers,
> Zoltan
>