CI problems

[Chris Olivier <cj...@gmail.com>]

How are so many broken unit tests getting into master?  Is stuff being
merged without passing CI/unit testing?  I have been trying to get three
PR's to build for over a week now.  Each time it's some broken test or
another that has nothing to do with my code changes.  It's extremely
frustrating -- I waste whole days on this, trying to figure out why my code
is breaking strange things only to realize later it's broken in all
branches.

Re: CI problems

[Joern Kottmann <ko...@gmail.com>]

GitHub shows red warnings for PRs that didn't pass all the tests. You
should never merge PRs which are red, or not current anymore (this
could also be a red status indicator).
If the failing tests can't be resolved quickly it might be worth
splitting the tests between stable / unstable and have different
status indicators for these tests.
Then people can see easily if the PR is breaking the stable tests.

Jörn

On Wed, Sep 27, 2017 at 8:12 PM, Gautam <ga...@gmail.com> wrote:
> Hi Chris,
>
>     There could be possibility that someone might have merged the changes
> without having all checks running, or overlooked the result. Currently we
> don't have any mechanism where git farm can reject such merge where CI/Unit
> test fails. I will try to explore any such possibilities. Meanwhile I would
> rather disable those fail tests with a git hub issue so that build can be
> clean for now. We can revisit those issue later.
>
>
>
>
> On Wed, Sep 27, 2017 at 11:02 AM, kellen sunderland <
> kellen.sunderland@gmail.com> wrote:
>
>> I share in your frustration Chris. I've also spent a fair amount of time in
>> the past few days digging through console logs to try and see if there was
>> anything actionable.  I haven't noticed any tests that were failing
>> consistently, maybe you can post an issue with some specific tests?  For me
>> the larger issue is Sanity Check failures and segfaults at the end of test
>> runs (after all tests pass).  I'm assuming that people are working on the
>> Sanity Check issues.  If there's anything external contributors can do
>> please let us know.
>>
>> On Wed, Sep 27, 2017 at 5:48 PM, Chris Olivier <cj...@gmail.com>
>> wrote:
>>
>> > By the way, I am not referring to a few tests that are known to fail
>> 1%-10%
>> > or so of the time (ie test_batchnorm_training) and are being actively
>> > worked on. I am referring to tests that fail 100% of the time and are
>> still
>> > merged into master, and thus propagate to all branches when sync'd from
>> > master.
>> >
>> > On Wed, Sep 27, 2017 at 8:43 AM, Chris Olivier <cj...@gmail.com>
>> > wrote:
>> >
>> > > How are so many broken unit tests getting into master?  Is stuff being
>> > > merged without passing CI/unit testing?  I have been trying to get
>> three
>> > > PR's to build for over a week now.  Each time it's some broken test or
>> > > another that has nothing to do with my code changes.  It's extremely
>> > > frustrating -- I waste whole days on this, trying to figure out why my
>> > code
>> > > is breaking strange things only to realize later it's broken in all
>> > > branches.
>> > >
>> >
>>
>
>
>
> --
> Best Regards,
> Gautam Kumar

Re: CI problems

[Gautam <ga...@gmail.com>]

Hi Chris,

    There could be possibility that someone might have merged the changes
without having all checks running, or overlooked the result. Currently we
don't have any mechanism where git farm can reject such merge where CI/Unit
test fails. I will try to explore any such possibilities. Meanwhile I would
rather disable those fail tests with a git hub issue so that build can be
clean for now. We can revisit those issue later.




On Wed, Sep 27, 2017 at 11:02 AM, kellen sunderland <
kellen.sunderland@gmail.com> wrote:

> I share in your frustration Chris. I've also spent a fair amount of time in
> the past few days digging through console logs to try and see if there was
> anything actionable.  I haven't noticed any tests that were failing
> consistently, maybe you can post an issue with some specific tests?  For me
> the larger issue is Sanity Check failures and segfaults at the end of test
> runs (after all tests pass).  I'm assuming that people are working on the
> Sanity Check issues.  If there's anything external contributors can do
> please let us know.
>
> On Wed, Sep 27, 2017 at 5:48 PM, Chris Olivier <cj...@gmail.com>
> wrote:
>
> > By the way, I am not referring to a few tests that are known to fail
> 1%-10%
> > or so of the time (ie test_batchnorm_training) and are being actively
> > worked on. I am referring to tests that fail 100% of the time and are
> still
> > merged into master, and thus propagate to all branches when sync'd from
> > master.
> >
> > On Wed, Sep 27, 2017 at 8:43 AM, Chris Olivier <cj...@gmail.com>
> > wrote:
> >
> > > How are so many broken unit tests getting into master?  Is stuff being
> > > merged without passing CI/unit testing?  I have been trying to get
> three
> > > PR's to build for over a week now.  Each time it's some broken test or
> > > another that has nothing to do with my code changes.  It's extremely
> > > frustrating -- I waste whole days on this, trying to figure out why my
> > code
> > > is breaking strange things only to realize later it's broken in all
> > > branches.
> > >
> >
>



-- 
Best Regards,
Gautam Kumar

Re: CI problems

[kellen sunderland <ke...@gmail.com>]

I share in your frustration Chris. I've also spent a fair amount of time in
the past few days digging through console logs to try and see if there was
anything actionable.  I haven't noticed any tests that were failing
consistently, maybe you can post an issue with some specific tests?  For me
the larger issue is Sanity Check failures and segfaults at the end of test
runs (after all tests pass).  I'm assuming that people are working on the
Sanity Check issues.  If there's anything external contributors can do
please let us know.

On Wed, Sep 27, 2017 at 5:48 PM, Chris Olivier <cj...@gmail.com>
wrote:

> By the way, I am not referring to a few tests that are known to fail 1%-10%
> or so of the time (ie test_batchnorm_training) and are being actively
> worked on. I am referring to tests that fail 100% of the time and are still
> merged into master, and thus propagate to all branches when sync'd from
> master.
>
> On Wed, Sep 27, 2017 at 8:43 AM, Chris Olivier <cj...@gmail.com>
> wrote:
>
> > How are so many broken unit tests getting into master?  Is stuff being
> > merged without passing CI/unit testing?  I have been trying to get three
> > PR's to build for over a week now.  Each time it's some broken test or
> > another that has nothing to do with my code changes.  It's extremely
> > frustrating -- I waste whole days on this, trying to figure out why my
> code
> > is breaking strange things only to realize later it's broken in all
> > branches.
> >
>

Re: CI problems

[Chris Olivier <cj...@gmail.com>]

By the way, I am not referring to a few tests that are known to fail 1%-10%
or so of the time (ie test_batchnorm_training) and are being actively
worked on. I am referring to tests that fail 100% of the time and are still
merged into master, and thus propagate to all branches when sync'd from
master.

On Wed, Sep 27, 2017 at 8:43 AM, Chris Olivier <cj...@gmail.com>
wrote:

> How are so many broken unit tests getting into master?  Is stuff being
> merged without passing CI/unit testing?  I have been trying to get three
> PR's to build for over a week now.  Each time it's some broken test or
> another that has nothing to do with my code changes.  It's extremely
> frustrating -- I waste whole days on this, trying to figure out why my code
> is breaking strange things only to realize later it's broken in all
> branches.
>