You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@couchdb.apache.org by Joan Touzet <wo...@apache.org> on 2019/12/12 21:39:02 UTC

[PROPOSAL] Remove `make elixir` from `make check`

Hi again,

As I've been looking more closely at the CI suite for the Jenkins 
transition, I've noticed that our Elixir test cases are actually the 
most likely to fail. In 6 consecutive CI runs, 5 runs failed due to 
failures in the Elixir suite. (The 6th failed due to a JS test failure.)

We started the Elixir effort to retire the JS suite. We reached a 
decision some months ago to put it into `make check` so that people 
would pay attention to its output, and work to fix those tests, 
accelerating our chances to get rid of the JS suite.

Unfortunately, that's not materialised. Our Elixir test porters seem to 
have stopped their work for a while now, and no one is systematically 
addressing the failures in that suite. I've also heard other developers 
mention (via IRC) that some of the test cases hold invalid assumptions 
about how CouchDB works, especially with the Erlang-based clustering 
code. It sounds to me like the effort needs a full code review.

With 3.0 around the corner, I want people to be able to trust the output 
of `make check` when downloading the tarball. If there is no objection, 
when I merge the Erlang version / CI changes on Monday, I will also 
comment out the call to `make elixir` as part of make check.

When the Elixir porting team is more confident in the reliability and 
completeness of their work, and we can successfully retire the JS suite, 
we can reconsider.

-Joan "really wanting to see green, but only seeing red" Touzet

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Jan Lehnardt <ja...@apache.org>.

> On 14. Dec 2019, at 18:29, Garren Smith <ga...@apache.org> wrote:
> 
> I’m a bit nervous to remove the full elixir test suite. The only partition
> tests are in the elixir suite. So we would have no test coverage for any of
> that if we remove this. Given that we want to make this the best CouchDB
> release ever it feels counter-intuitive to them be removing tests.
> 
> I also think there is a few JS tests we have disabled because they are
> covered in Elixir so we would need to re-enable those in the same PR that we remove the elixir tests.

That hasn’t landed yet afaict[1]: but I suggest we get this going to create some progress on the deprecate-js-tests train.

Best
Jan
—
[1]: https://github.com/apache/couchdb/commit/20e7b2e0cf496c925e99c80befa8b50fd69b5017



> 
> Cheers
> Garren
> 
> On Sat, Dec 14, 2019 at 2:51 AM Jay Doane <ja...@apache.org> wrote:
> 
>> On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <wo...@apache.org> wrote:
>> 
>>> Jay, consider this a +1 to remove those if you can get to them before I
>> do.
>>> 
>> 
>> Joan, I don't think I was clear, but when I said
>>> I'm +1 to removing these tests temporarily.
>> by "these tests" I actually meant all the elixir tests, not just the two I
>> mentioned. I don't think we'd get very far by just disabling those two, as
>> there are many others that fail intermittently as well. I'm +1 to remove
>> them, and here's a PR in case someone wants it to happen sooner rather than
>> later:
>> 
>> https://github.com/apache/couchdb/pull/2362
>> 
>> Thanks,
>> Jay
>> 
>>> 
>>> If this makes the Elixir tests useful again, then it's a good compromise.
>>> 
>>> -Joan
>>> 
>>> On 2019-12-13 2:18 a.m., Jay Doane wrote:
>>>> I've been trying to merge into master for the last 24+ hours, and have
>>>> restarted the build about 10 times so far, but it has yet to pass.
>>>> 
>>>> I kept track of failures, and these came up numerous times:
>>>> 
>>>>  1) test GET /dbname/_design_docs (PartitionDDocTest)
>>>>      test/elixir/test/partition_ddoc_test.exs:160
>>>>      ** (RuntimeError) timed out after 10294 ms
>>>>      code: retry_until(
>>>>      stacktrace:
>>>>        (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
>>>> Couch.DBTest.retry_until/4
>>>>        test/elixir/test/partition_ddoc_test.exs:163: (test)
>>>> 
>>>>  1) test All Docs tests (AllDocsTest)
>>>>      test/elixir/test/all_docs_test.exs:15
>>>>      ** (RuntimeError) timed out after 30097 ms
>>>>      code: retry_until(fn ->
>>>>      stacktrace:
>>>>        (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
>>>> Couch.DBTest.retry_until/4
>>>>        test/elixir/test/all_docs_test.exs:75: (test)
>>>> 
>>>> which both seem to be timing out.
>>>> 
>>>> I'm +1 to removing these tests temporarily.
>>>> 
>>>> Jay
>>>> 
>>>> On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:
>>>> 
>>>>> Hi again,
>>>>> 
>>>>> As I've been looking more closely at the CI suite for the Jenkins
>>>>> transition, I've noticed that our Elixir test cases are actually the
>>>>> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
>>>>> failures in the Elixir suite. (The 6th failed due to a JS test
>> failure.)
>>>>> 
>>>>> We started the Elixir effort to retire the JS suite. We reached a
>>>>> decision some months ago to put it into `make check` so that people
>>>>> would pay attention to its output, and work to fix those tests,
>>>>> accelerating our chances to get rid of the JS suite.
>>>>> 
>>>>> Unfortunately, that's not materialised. Our Elixir test porters seem
>> to
>>>>> have stopped their work for a while now, and no one is systematically
>>>>> addressing the failures in that suite. I've also heard other
>> developers
>>>>> mention (via IRC) that some of the test cases hold invalid assumptions
>>>>> about how CouchDB works, especially with the Erlang-based clustering
>>>>> code. It sounds to me like the effort needs a full code review.
>>>>> 
>>>>> With 3.0 around the corner, I want people to be able to trust the
>> output
>>>>> of `make check` when downloading the tarball. If there is no
>> objection,
>>>>> when I merge the Erlang version / CI changes on Monday, I will also
>>>>> comment out the call to `make elixir` as part of make check.
>>>>> 
>>>>> When the Elixir porting team is more confident in the reliability and
>>>>> completeness of their work, and we can successfully retire the JS
>> suite,
>>>>> we can reconsider.
>>>>> 
>>>>> -Joan "really wanting to see green, but only seeing red" Touzet
>>>>> 
>>>> 
>>> 
>> 

-- 
Professional Support for Apache CouchDB:
https://neighbourhood.ie/couchdb-support/


Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Joan Touzet <wo...@apache.org>.
Closing the loop, a change was checked into master that reduces the 
Elixir suite to run against an n=1 CouchDB cluster, permanently. This 
should help significantly for now.

Any remaining Elixir test failures should have issues opened on them.

On 2019-12-14 12:29 p.m., Garren Smith wrote:
> I’m a bit nervous to remove the full elixir test suite. The only partition
> tests are in the elixir suite. So we would have no test coverage for any of
> that if we remove this. Given that we want to make this the best CouchDB
> release ever it feels counter-intuitive to them be removing tests.
> 
> I also think there is a few JS tests we have disabled because they are
> covered in Elixir so we would need to re-enable those in the same PR that
> we remove the elixir tests.
> 
> Cheers
> Garren
> 
> On Sat, Dec 14, 2019 at 2:51 AM Jay Doane <ja...@apache.org> wrote:
> 
>> On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <wo...@apache.org> wrote:
>>
>>> Jay, consider this a +1 to remove those if you can get to them before I
>> do.
>>>
>>
>> Joan, I don't think I was clear, but when I said
>>> I'm +1 to removing these tests temporarily.
>> by "these tests" I actually meant all the elixir tests, not just the two I
>> mentioned. I don't think we'd get very far by just disabling those two, as
>> there are many others that fail intermittently as well. I'm +1 to remove
>> them, and here's a PR in case someone wants it to happen sooner rather than
>> later:
>>
>> https://github.com/apache/couchdb/pull/2362
>>
>> Thanks,
>> Jay
>>
>>>
>>> If this makes the Elixir tests useful again, then it's a good compromise.
>>>
>>> -Joan
>>>
>>> On 2019-12-13 2:18 a.m., Jay Doane wrote:
>>>> I've been trying to merge into master for the last 24+ hours, and have
>>>> restarted the build about 10 times so far, but it has yet to pass.
>>>>
>>>> I kept track of failures, and these came up numerous times:
>>>>
>>>>    1) test GET /dbname/_design_docs (PartitionDDocTest)
>>>>        test/elixir/test/partition_ddoc_test.exs:160
>>>>        ** (RuntimeError) timed out after 10294 ms
>>>>        code: retry_until(
>>>>        stacktrace:
>>>>          (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
>>>> Couch.DBTest.retry_until/4
>>>>          test/elixir/test/partition_ddoc_test.exs:163: (test)
>>>>
>>>>    1) test All Docs tests (AllDocsTest)
>>>>        test/elixir/test/all_docs_test.exs:15
>>>>        ** (RuntimeError) timed out after 30097 ms
>>>>        code: retry_until(fn ->
>>>>        stacktrace:
>>>>          (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
>>>> Couch.DBTest.retry_until/4
>>>>          test/elixir/test/all_docs_test.exs:75: (test)
>>>>
>>>> which both seem to be timing out.
>>>>
>>>> I'm +1 to removing these tests temporarily.
>>>>
>>>> Jay
>>>>
>>>> On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:
>>>>
>>>>> Hi again,
>>>>>
>>>>> As I've been looking more closely at the CI suite for the Jenkins
>>>>> transition, I've noticed that our Elixir test cases are actually the
>>>>> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
>>>>> failures in the Elixir suite. (The 6th failed due to a JS test
>> failure.)
>>>>>
>>>>> We started the Elixir effort to retire the JS suite. We reached a
>>>>> decision some months ago to put it into `make check` so that people
>>>>> would pay attention to its output, and work to fix those tests,
>>>>> accelerating our chances to get rid of the JS suite.
>>>>>
>>>>> Unfortunately, that's not materialised. Our Elixir test porters seem
>> to
>>>>> have stopped their work for a while now, and no one is systematically
>>>>> addressing the failures in that suite. I've also heard other
>> developers
>>>>> mention (via IRC) that some of the test cases hold invalid assumptions
>>>>> about how CouchDB works, especially with the Erlang-based clustering
>>>>> code. It sounds to me like the effort needs a full code review.
>>>>>
>>>>> With 3.0 around the corner, I want people to be able to trust the
>> output
>>>>> of `make check` when downloading the tarball. If there is no
>> objection,
>>>>> when I merge the Erlang version / CI changes on Monday, I will also
>>>>> comment out the call to `make elixir` as part of make check.
>>>>>
>>>>> When the Elixir porting team is more confident in the reliability and
>>>>> completeness of their work, and we can successfully retire the JS
>> suite,
>>>>> we can reconsider.
>>>>>
>>>>> -Joan "really wanting to see green, but only seeing red" Touzet
>>>>>
>>>>
>>>
>>
> 

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Garren Smith <ga...@apache.org>.
I’m a bit nervous to remove the full elixir test suite. The only partition
tests are in the elixir suite. So we would have no test coverage for any of
that if we remove this. Given that we want to make this the best CouchDB
release ever it feels counter-intuitive to them be removing tests.

I also think there is a few JS tests we have disabled because they are
covered in Elixir so we would need to re-enable those in the same PR that
we remove the elixir tests.

Cheers
Garren

On Sat, Dec 14, 2019 at 2:51 AM Jay Doane <ja...@apache.org> wrote:

> On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <wo...@apache.org> wrote:
>
> > Jay, consider this a +1 to remove those if you can get to them before I
> do.
> >
>
> Joan, I don't think I was clear, but when I said
> > I'm +1 to removing these tests temporarily.
> by "these tests" I actually meant all the elixir tests, not just the two I
> mentioned. I don't think we'd get very far by just disabling those two, as
> there are many others that fail intermittently as well. I'm +1 to remove
> them, and here's a PR in case someone wants it to happen sooner rather than
> later:
>
> https://github.com/apache/couchdb/pull/2362
>
> Thanks,
> Jay
>
> >
> > If this makes the Elixir tests useful again, then it's a good compromise.
> >
> > -Joan
> >
> > On 2019-12-13 2:18 a.m., Jay Doane wrote:
> > > I've been trying to merge into master for the last 24+ hours, and have
> > > restarted the build about 10 times so far, but it has yet to pass.
> > >
> > > I kept track of failures, and these came up numerous times:
> > >
> > >   1) test GET /dbname/_design_docs (PartitionDDocTest)
> > >       test/elixir/test/partition_ddoc_test.exs:160
> > >       ** (RuntimeError) timed out after 10294 ms
> > >       code: retry_until(
> > >       stacktrace:
> > >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > > Couch.DBTest.retry_until/4
> > >         test/elixir/test/partition_ddoc_test.exs:163: (test)
> > >
> > >   1) test All Docs tests (AllDocsTest)
> > >       test/elixir/test/all_docs_test.exs:15
> > >       ** (RuntimeError) timed out after 30097 ms
> > >       code: retry_until(fn ->
> > >       stacktrace:
> > >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > > Couch.DBTest.retry_until/4
> > >         test/elixir/test/all_docs_test.exs:75: (test)
> > >
> > > which both seem to be timing out.
> > >
> > > I'm +1 to removing these tests temporarily.
> > >
> > > Jay
> > >
> > > On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:
> > >
> > >> Hi again,
> > >>
> > >> As I've been looking more closely at the CI suite for the Jenkins
> > >> transition, I've noticed that our Elixir test cases are actually the
> > >> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
> > >> failures in the Elixir suite. (The 6th failed due to a JS test
> failure.)
> > >>
> > >> We started the Elixir effort to retire the JS suite. We reached a
> > >> decision some months ago to put it into `make check` so that people
> > >> would pay attention to its output, and work to fix those tests,
> > >> accelerating our chances to get rid of the JS suite.
> > >>
> > >> Unfortunately, that's not materialised. Our Elixir test porters seem
> to
> > >> have stopped their work for a while now, and no one is systematically
> > >> addressing the failures in that suite. I've also heard other
> developers
> > >> mention (via IRC) that some of the test cases hold invalid assumptions
> > >> about how CouchDB works, especially with the Erlang-based clustering
> > >> code. It sounds to me like the effort needs a full code review.
> > >>
> > >> With 3.0 around the corner, I want people to be able to trust the
> output
> > >> of `make check` when downloading the tarball. If there is no
> objection,
> > >> when I merge the Erlang version / CI changes on Monday, I will also
> > >> comment out the call to `make elixir` as part of make check.
> > >>
> > >> When the Elixir porting team is more confident in the reliability and
> > >> completeness of their work, and we can successfully retire the JS
> suite,
> > >> we can reconsider.
> > >>
> > >> -Joan "really wanting to see green, but only seeing red" Touzet
> > >>
> > >
> >
>

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Jay Doane <ja...@apache.org>.
On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <wo...@apache.org> wrote:

> Jay, consider this a +1 to remove those if you can get to them before I do.
>

Joan, I don't think I was clear, but when I said
> I'm +1 to removing these tests temporarily.
by "these tests" I actually meant all the elixir tests, not just the two I
mentioned. I don't think we'd get very far by just disabling those two, as
there are many others that fail intermittently as well. I'm +1 to remove
them, and here's a PR in case someone wants it to happen sooner rather than
later:

https://github.com/apache/couchdb/pull/2362

Thanks,
Jay

>
> If this makes the Elixir tests useful again, then it's a good compromise.
>
> -Joan
>
> On 2019-12-13 2:18 a.m., Jay Doane wrote:
> > I've been trying to merge into master for the last 24+ hours, and have
> > restarted the build about 10 times so far, but it has yet to pass.
> >
> > I kept track of failures, and these came up numerous times:
> >
> >   1) test GET /dbname/_design_docs (PartitionDDocTest)
> >       test/elixir/test/partition_ddoc_test.exs:160
> >       ** (RuntimeError) timed out after 10294 ms
> >       code: retry_until(
> >       stacktrace:
> >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > Couch.DBTest.retry_until/4
> >         test/elixir/test/partition_ddoc_test.exs:163: (test)
> >
> >   1) test All Docs tests (AllDocsTest)
> >       test/elixir/test/all_docs_test.exs:15
> >       ** (RuntimeError) timed out after 30097 ms
> >       code: retry_until(fn ->
> >       stacktrace:
> >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > Couch.DBTest.retry_until/4
> >         test/elixir/test/all_docs_test.exs:75: (test)
> >
> > which both seem to be timing out.
> >
> > I'm +1 to removing these tests temporarily.
> >
> > Jay
> >
> > On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:
> >
> >> Hi again,
> >>
> >> As I've been looking more closely at the CI suite for the Jenkins
> >> transition, I've noticed that our Elixir test cases are actually the
> >> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
> >> failures in the Elixir suite. (The 6th failed due to a JS test failure.)
> >>
> >> We started the Elixir effort to retire the JS suite. We reached a
> >> decision some months ago to put it into `make check` so that people
> >> would pay attention to its output, and work to fix those tests,
> >> accelerating our chances to get rid of the JS suite.
> >>
> >> Unfortunately, that's not materialised. Our Elixir test porters seem to
> >> have stopped their work for a while now, and no one is systematically
> >> addressing the failures in that suite. I've also heard other developers
> >> mention (via IRC) that some of the test cases hold invalid assumptions
> >> about how CouchDB works, especially with the Erlang-based clustering
> >> code. It sounds to me like the effort needs a full code review.
> >>
> >> With 3.0 around the corner, I want people to be able to trust the output
> >> of `make check` when downloading the tarball. If there is no objection,
> >> when I merge the Erlang version / CI changes on Monday, I will also
> >> comment out the call to `make elixir` as part of make check.
> >>
> >> When the Elixir porting team is more confident in the reliability and
> >> completeness of their work, and we can successfully retire the JS suite,
> >> we can reconsider.
> >>
> >> -Joan "really wanting to see green, but only seeing red" Touzet
> >>
> >
>

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Joan Touzet <wo...@apache.org>.
Jay, consider this a +1 to remove those if you can get to them before I do.

If this makes the Elixir tests useful again, then it's a good compromise.

-Joan

On 2019-12-13 2:18 a.m., Jay Doane wrote:
> I've been trying to merge into master for the last 24+ hours, and have
> restarted the build about 10 times so far, but it has yet to pass.
> 
> I kept track of failures, and these came up numerous times:
> 
>   1) test GET /dbname/_design_docs (PartitionDDocTest)
>       test/elixir/test/partition_ddoc_test.exs:160
>       ** (RuntimeError) timed out after 10294 ms
>       code: retry_until(
>       stacktrace:
>         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> Couch.DBTest.retry_until/4
>         test/elixir/test/partition_ddoc_test.exs:163: (test)
> 
>   1) test All Docs tests (AllDocsTest)
>       test/elixir/test/all_docs_test.exs:15
>       ** (RuntimeError) timed out after 30097 ms
>       code: retry_until(fn ->
>       stacktrace:
>         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> Couch.DBTest.retry_until/4
>         test/elixir/test/all_docs_test.exs:75: (test)
> 
> which both seem to be timing out.
> 
> I'm +1 to removing these tests temporarily.
> 
> Jay
> 
> On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:
> 
>> Hi again,
>>
>> As I've been looking more closely at the CI suite for the Jenkins
>> transition, I've noticed that our Elixir test cases are actually the
>> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
>> failures in the Elixir suite. (The 6th failed due to a JS test failure.)
>>
>> We started the Elixir effort to retire the JS suite. We reached a
>> decision some months ago to put it into `make check` so that people
>> would pay attention to its output, and work to fix those tests,
>> accelerating our chances to get rid of the JS suite.
>>
>> Unfortunately, that's not materialised. Our Elixir test porters seem to
>> have stopped their work for a while now, and no one is systematically
>> addressing the failures in that suite. I've also heard other developers
>> mention (via IRC) that some of the test cases hold invalid assumptions
>> about how CouchDB works, especially with the Erlang-based clustering
>> code. It sounds to me like the effort needs a full code review.
>>
>> With 3.0 around the corner, I want people to be able to trust the output
>> of `make check` when downloading the tarball. If there is no objection,
>> when I merge the Erlang version / CI changes on Monday, I will also
>> comment out the call to `make elixir` as part of make check.
>>
>> When the Elixir porting team is more confident in the reliability and
>> completeness of their work, and we can successfully retire the JS suite,
>> we can reconsider.
>>
>> -Joan "really wanting to see green, but only seeing red" Touzet
>>
> 

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Jay Doane <ja...@apache.org>.
I've been trying to merge into master for the last 24+ hours, and have
restarted the build about 10 times so far, but it has yet to pass.

I kept track of failures, and these came up numerous times:

 1) test GET /dbname/_design_docs (PartitionDDocTest)
     test/elixir/test/partition_ddoc_test.exs:160
     ** (RuntimeError) timed out after 10294 ms
     code: retry_until(
     stacktrace:
       (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
Couch.DBTest.retry_until/4
       test/elixir/test/partition_ddoc_test.exs:163: (test)

 1) test All Docs tests (AllDocsTest)
     test/elixir/test/all_docs_test.exs:15
     ** (RuntimeError) timed out after 30097 ms
     code: retry_until(fn ->
     stacktrace:
       (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
Couch.DBTest.retry_until/4
       test/elixir/test/all_docs_test.exs:75: (test)

which both seem to be timing out.

I'm +1 to removing these tests temporarily.

Jay

On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wo...@apache.org> wrote:

> Hi again,
>
> As I've been looking more closely at the CI suite for the Jenkins
> transition, I've noticed that our Elixir test cases are actually the
> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
> failures in the Elixir suite. (The 6th failed due to a JS test failure.)
>
> We started the Elixir effort to retire the JS suite. We reached a
> decision some months ago to put it into `make check` so that people
> would pay attention to its output, and work to fix those tests,
> accelerating our chances to get rid of the JS suite.
>
> Unfortunately, that's not materialised. Our Elixir test porters seem to
> have stopped their work for a while now, and no one is systematically
> addressing the failures in that suite. I've also heard other developers
> mention (via IRC) that some of the test cases hold invalid assumptions
> about how CouchDB works, especially with the Erlang-based clustering
> code. It sounds to me like the effort needs a full code review.
>
> With 3.0 around the corner, I want people to be able to trust the output
> of `make check` when downloading the tarball. If there is no objection,
> when I merge the Erlang version / CI changes on Monday, I will also
> comment out the call to `make elixir` as part of make check.
>
> When the Elixir porting team is more confident in the reliability and
> completeness of their work, and we can successfully retire the JS suite,
> we can reconsider.
>
> -Joan "really wanting to see green, but only seeing red" Touzet
>

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Joan Touzet <wo...@apache.org>.
Hi Adam,

As long as someone is working on fixing the Elixir tests, fine. They 
*are* failing significantly more often on Jenkins than on Travis, for 
what it's worth.

All our work to build a much better setup for Jenkins may be lost if 
people don't fix these tests promptly.

Would a suitable workaround be having the CI suites run those tests in a 
"doesn't block the build" way? Or do you want to put that hurdle in the 
way of people merging to master/release branches?

Here's the tests that have failed on me during my CI work just today:

Elixir.DesignDocsTest: 4) test POST with empty body (DesignDocsTest)
test/elixir/test/design_docs_test.exs:55
Assertion with == failed

Elixir.PartitionMangoTest: 16) test explain works with non partitioned 
db (PartitionMangoTest)
test/elixir/test/partition_mango_test.exs:438
Assertion with == failed

(2x) Elixir.AllDocsTest: 5) test POST with empty body (AllDocsTest)
test/elixir/test/all_docs_test.exs:223
Assertion with == failed
code: assert length(Map.get(resp, :body)["rows"]) == 3

JS design_docs.js: doc insertion should have succeeded
402: test/javascript/tests/design_docs.js

Elixir harness setup failure:
Failed to start all the nodes. Check the dev/logs/*.log for errors.

-Joan

On 2019-12-12 5:31 p.m., Adam Kocoloski wrote:
> Hi Joan,
> 
> I’ve seen the Elixir suite implicated more frequently as well. I haven’t done the analysis to see if the failures are concentrated in one or two flakes or if they’re more evenly distributed. If it’s a small number of  flaky tests I think we have time to fix/disable them rather than knocking out the whole suite.
> 
> I agree that we need `make check` to be trustworthy in 3.0 release candidates. I would like to keep running the elixir tests in the CI regardless of whether they’re in the `check` suite. Cheers,
> 
> Adam
> 
>> On Dec 12, 2019, at 4:39 PM, Joan Touzet <wo...@apache.org> wrote:
>>
>> Hi again,
>>
>> As I've been looking more closely at the CI suite for the Jenkins transition, I've noticed that our Elixir test cases are actually the most likely to fail. In 6 consecutive CI runs, 5 runs failed due to failures in the Elixir suite. (The 6th failed due to a JS test failure.)
>>
>> We started the Elixir effort to retire the JS suite. We reached a decision some months ago to put it into `make check` so that people would pay attention to its output, and work to fix those tests, accelerating our chances to get rid of the JS suite.
>>
>> Unfortunately, that's not materialised. Our Elixir test porters seem to have stopped their work for a while now, and no one is systematically addressing the failures in that suite. I've also heard other developers mention (via IRC) that some of the test cases hold invalid assumptions about how CouchDB works, especially with the Erlang-based clustering code. It sounds to me like the effort needs a full code review.
>>
>> With 3.0 around the corner, I want people to be able to trust the output of `make check` when downloading the tarball. If there is no objection, when I merge the Erlang version / CI changes on Monday, I will also comment out the call to `make elixir` as part of make check.
>>
>> When the Elixir porting team is more confident in the reliability and completeness of their work, and we can successfully retire the JS suite, we can reconsider.
>>
>> -Joan "really wanting to see green, but only seeing red" Touzet
> 

Re: [PROPOSAL] Remove `make elixir` from `make check`

Posted by Adam Kocoloski <ko...@apache.org>.
Hi Joan,

I’ve seen the Elixir suite implicated more frequently as well. I haven’t done the analysis to see if the failures are concentrated in one or two flakes or if they’re more evenly distributed. If it’s a small number of  flaky tests I think we have time to fix/disable them rather than knocking out the whole suite.

I agree that we need `make check` to be trustworthy in 3.0 release candidates. I would like to keep running the elixir tests in the CI regardless of whether they’re in the `check` suite. Cheers,

Adam

> On Dec 12, 2019, at 4:39 PM, Joan Touzet <wo...@apache.org> wrote:
> 
> Hi again,
> 
> As I've been looking more closely at the CI suite for the Jenkins transition, I've noticed that our Elixir test cases are actually the most likely to fail. In 6 consecutive CI runs, 5 runs failed due to failures in the Elixir suite. (The 6th failed due to a JS test failure.)
> 
> We started the Elixir effort to retire the JS suite. We reached a decision some months ago to put it into `make check` so that people would pay attention to its output, and work to fix those tests, accelerating our chances to get rid of the JS suite.
> 
> Unfortunately, that's not materialised. Our Elixir test porters seem to have stopped their work for a while now, and no one is systematically addressing the failures in that suite. I've also heard other developers mention (via IRC) that some of the test cases hold invalid assumptions about how CouchDB works, especially with the Erlang-based clustering code. It sounds to me like the effort needs a full code review.
> 
> With 3.0 around the corner, I want people to be able to trust the output of `make check` when downloading the tarball. If there is no objection, when I merge the Erlang version / CI changes on Monday, I will also comment out the call to `make elixir` as part of make check.
> 
> When the Elixir porting team is more confident in the reliability and completeness of their work, and we can successfully retire the JS suite, we can reconsider.
> 
> -Joan "really wanting to see green, but only seeing red" Touzet