Green unit test results

[Peter Vary <pv...@cloudera.com>]

Hi,

Congratulations for everyone who have helped taking care of the unit test failures!
I have got my first green run! :)

If any of you interested in:
https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/ <https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/> :)

Great day, and again thanks everyone!

Peter

Re: Green unit test results

[Vihang Karajgaonkar <vi...@cloudera.com>]

One thing which could be done by each author of a patch is to do a search
on the HiveQA test failures. If there are flaky tests which fail on the
HiveQA run may be should create a JIRA for each of them and link it to the
JIRA. The JIRA could have useful information like stacktraces or attaching
the logs pertaining to the test run. If there is already an existing JIRA
for that flaky test we should still link it. This way we can prioritize the
flaky tests which has been linked a lot of times.

On Mon, Mar 13, 2017 at 11:40 AM, Siddharth Seth <ss...@apache.org> wrote:

> There's far too many flaky tests to enforce a clean test run prior to
> commit.
> However, there also needs to be a very conscious effort to fix flaky tests
> - otherwise, we're not going to reach the state where this can be enforced.
> I'm also not sure whether this is just flaky tests, or whether there are
> real problems. Maybe a few volunteers could drive the test fixes, and as an
> artifact of this, come up with guidelines on writing tests (avoid stats
> comparisons in qfiles, skip lines which may be environment specific,
> introduce a new compare mechanism for lines which may be in a different
> order based on internal structures, etc).
>
> On Thu, Mar 9, 2017 at 2:39 PM, Ashutosh Chauhan <ha...@apache.org>
> wrote:
>
> > I went through all the builds of last 24 hours and though there are
> couple
> > of green runs, we still have quite a bit of flakiness in our tests. Most
> of
> > that is captured via jiras on :
> > https://issues.apache.org/jira/browse/HIVE-15058 Till we have that
> > flakiness I am not sure if we can enforce no commit on test failures
> > policy. Once flakiness is straightened out then this discussion will
> become
> > moot anyways.
> > Towards that I think this flaky detector job is a step in right
> direction.
> >
> >
> > On Thu, Mar 9, 2017 at 11:38 AM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> > > - Probably avoiding committing a patch if a flaky test is shown on the
> > test
> > > results?
> > > - Should we add a jenkins job that checks for flaky tests like the
> hbase
> > > project did?
> > > https://builds.apache.org/view/H-L/view/HBase/job/HBase-
> > Find-Flaky-Tests/
> > >
> > > On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan <
> hashutosh@apache.org>
> > > wrote:
> > >
> > > > Great news! Thanks to everyone who contributed in getting our tests
> and
> > > > test infra sorted out.
> > > > We would definitely want to keep the status either green or blue
> > > definitely
> > > > not red :) All our previous efforts in keeping builds green didn't
> bear
> > > > fruit.
> > > > So, I think we need to make some changes here.
> > > >
> > > > Any ideas what we can do to ensure green builds going forward?
> > > >
> > > > Thanks,
> > > > Ashutosh
> > > >
> > > > On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <
> sergio.pena@cloudera.com>
> > > > wrote:
> > > >
> > > > > It's actually blue Peter :).
> > > > >
> > > > > But good job, I see that the console output is:
> > > > >
> > > > > {color:red}ERROR:{color} -1 due to no test(s) being added or
> > modified.
> > > > >
> > > > > {color:green}SUCCESS:{color} +1 due to 10336 tests passed
> > > > >
> > > > >
> > > > > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com>
> > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Congratulations for everyone who have helped taking care of the
> > unit
> > > > test
> > > > > > failures!
> > > > > > I have got my first green run! :)
> > > > > >
> > > > > > If any of you interested in:
> > > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> > testReport/
> > > <
> > > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> > testReport/>
> > > > :)
> > > > > >
> > > > > > Great day, and again thanks everyone!
> > > > > >
> > > > > > Peter
> > > > >
> > > >
> > >
> >
>

Re: Green unit test results

[Siddharth Seth <ss...@apache.org>]

There's far too many flaky tests to enforce a clean test run prior to
commit.
However, there also needs to be a very conscious effort to fix flaky tests
- otherwise, we're not going to reach the state where this can be enforced.
I'm also not sure whether this is just flaky tests, or whether there are
real problems. Maybe a few volunteers could drive the test fixes, and as an
artifact of this, come up with guidelines on writing tests (avoid stats
comparisons in qfiles, skip lines which may be environment specific,
introduce a new compare mechanism for lines which may be in a different
order based on internal structures, etc).

On Thu, Mar 9, 2017 at 2:39 PM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> I went through all the builds of last 24 hours and though there are couple
> of green runs, we still have quite a bit of flakiness in our tests. Most of
> that is captured via jiras on :
> https://issues.apache.org/jira/browse/HIVE-15058 Till we have that
> flakiness I am not sure if we can enforce no commit on test failures
> policy. Once flakiness is straightened out then this discussion will become
> moot anyways.
> Towards that I think this flaky detector job is a step in right direction.
>
>
> On Thu, Mar 9, 2017 at 11:38 AM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > - Probably avoiding committing a patch if a flaky test is shown on the
> test
> > results?
> > - Should we add a jenkins job that checks for flaky tests like the hbase
> > project did?
> > https://builds.apache.org/view/H-L/view/HBase/job/HBase-
> Find-Flaky-Tests/
> >
> > On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan <ha...@apache.org>
> > wrote:
> >
> > > Great news! Thanks to everyone who contributed in getting our tests and
> > > test infra sorted out.
> > > We would definitely want to keep the status either green or blue
> > definitely
> > > not red :) All our previous efforts in keeping builds green didn't bear
> > > fruit.
> > > So, I think we need to make some changes here.
> > >
> > > Any ideas what we can do to ensure green builds going forward?
> > >
> > > Thanks,
> > > Ashutosh
> > >
> > > On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <se...@cloudera.com>
> > > wrote:
> > >
> > > > It's actually blue Peter :).
> > > >
> > > > But good job, I see that the console output is:
> > > >
> > > > {color:red}ERROR:{color} -1 due to no test(s) being added or
> modified.
> > > >
> > > > {color:green}SUCCESS:{color} +1 due to 10336 tests passed
> > > >
> > > >
> > > > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com>
> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Congratulations for everyone who have helped taking care of the
> unit
> > > test
> > > > > failures!
> > > > > I have got my first green run! :)
> > > > >
> > > > > If any of you interested in:
> > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> testReport/
> > <
> > > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/
> testReport/>
> > > :)
> > > > >
> > > > > Great day, and again thanks everyone!
> > > > >
> > > > > Peter
> > > >
> > >
> >
>

Re: Green unit test results

[Ashutosh Chauhan <ha...@apache.org>]

I went through all the builds of last 24 hours and though there are couple
of green runs, we still have quite a bit of flakiness in our tests. Most of
that is captured via jiras on :
https://issues.apache.org/jira/browse/HIVE-15058 Till we have that
flakiness I am not sure if we can enforce no commit on test failures
policy. Once flakiness is straightened out then this discussion will become
moot anyways.
Towards that I think this flaky detector job is a step in right direction.


On Thu, Mar 9, 2017 at 11:38 AM, Sergio Pena <se...@cloudera.com>
wrote:

> - Probably avoiding committing a patch if a flaky test is shown on the test
> results?
> - Should we add a jenkins job that checks for flaky tests like the hbase
> project did?
> https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/
>
> On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan <ha...@apache.org>
> wrote:
>
> > Great news! Thanks to everyone who contributed in getting our tests and
> > test infra sorted out.
> > We would definitely want to keep the status either green or blue
> definitely
> > not red :) All our previous efforts in keeping builds green didn't bear
> > fruit.
> > So, I think we need to make some changes here.
> >
> > Any ideas what we can do to ensure green builds going forward?
> >
> > Thanks,
> > Ashutosh
> >
> > On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <se...@cloudera.com>
> > wrote:
> >
> > > It's actually blue Peter :).
> > >
> > > But good job, I see that the console output is:
> > >
> > > {color:red}ERROR:{color} -1 due to no test(s) being added or modified.
> > >
> > > {color:green}SUCCESS:{color} +1 due to 10336 tests passed
> > >
> > >
> > > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com> wrote:
> > >
> > > > Hi,
> > > >
> > > > Congratulations for everyone who have helped taking care of the unit
> > test
> > > > failures!
> > > > I have got my first green run! :)
> > > >
> > > > If any of you interested in:
> > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/
> <
> > > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/>
> > :)
> > > >
> > > > Great day, and again thanks everyone!
> > > >
> > > > Peter
> > >
> >
>

Re: Green unit test results

[Sergio Pena <se...@cloudera.com>]

- Probably avoiding committing a patch if a flaky test is shown on the test
results?
- Should we add a jenkins job that checks for flaky tests like the hbase
project did?
https://builds.apache.org/view/H-L/view/HBase/job/HBase-Find-Flaky-Tests/

On Thu, Mar 9, 2017 at 10:21 AM, Ashutosh Chauhan <ha...@apache.org>
wrote:

> Great news! Thanks to everyone who contributed in getting our tests and
> test infra sorted out.
> We would definitely want to keep the status either green or blue definitely
> not red :) All our previous efforts in keeping builds green didn't bear
> fruit.
> So, I think we need to make some changes here.
>
> Any ideas what we can do to ensure green builds going forward?
>
> Thanks,
> Ashutosh
>
> On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <se...@cloudera.com>
> wrote:
>
> > It's actually blue Peter :).
> >
> > But good job, I see that the console output is:
> >
> > {color:red}ERROR:{color} -1 due to no test(s) being added or modified.
> >
> > {color:green}SUCCESS:{color} +1 due to 10336 tests passed
> >
> >
> > On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com> wrote:
> >
> > > Hi,
> > >
> > > Congratulations for everyone who have helped taking care of the unit
> test
> > > failures!
> > > I have got my first green run! :)
> > >
> > > If any of you interested in:
> > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/ <
> > > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/>
> :)
> > >
> > > Great day, and again thanks everyone!
> > >
> > > Peter
> >
>

Re: Green unit test results

[Ashutosh Chauhan <ha...@apache.org>]

Great news! Thanks to everyone who contributed in getting our tests and
test infra sorted out.
We would definitely want to keep the status either green or blue definitely
not red :) All our previous efforts in keeping builds green didn't bear
fruit.
So, I think we need to make some changes here.

Any ideas what we can do to ensure green builds going forward?

Thanks,
Ashutosh

On Thu, Mar 9, 2017 at 8:07 AM, Sergio Pena <se...@cloudera.com>
wrote:

> It's actually blue Peter :).
>
> But good job, I see that the console output is:
>
> {color:red}ERROR:{color} -1 due to no test(s) being added or modified.
>
> {color:green}SUCCESS:{color} +1 due to 10336 tests passed
>
>
> On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com> wrote:
>
> > Hi,
> >
> > Congratulations for everyone who have helped taking care of the unit test
> > failures!
> > I have got my first green run! :)
> >
> > If any of you interested in:
> > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/ <
> > https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/> :)
> >
> > Great day, and again thanks everyone!
> >
> > Peter
>

Re: Green unit test results

[Sergio Pena <se...@cloudera.com>]

It's actually blue Peter :).

But good job, I see that the console output is:

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:green}SUCCESS:{color} +1 due to 10336 tests passed


On Thu, Mar 9, 2017 at 8:12 AM, Peter Vary <pv...@cloudera.com> wrote:

> Hi,
>
> Congratulations for everyone who have helped taking care of the unit test
> failures!
> I have got my first green run! :)
>
> If any of you interested in:
> https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/ <
> https://builds.apache.org/job/PreCommit-HIVE-Build/4049/testReport/> :)
>
> Great day, and again thanks everyone!
>
> Peter