[PROPOSAL] Improving the quality of CouchDB

[Robert Kowalski <ro...@kowalski.gd>]

Hi list,

I had my first real programming job in a Web Agency. We were building 
mobile websites for our external customers. The development process 
followed a strict waterfall model, after you finished your project it was 
thrown over the fence to the QA folks, which tested the website manually. 
At some point, when all found bugs were fixed, the product got released. 
Sometimes a bugfix took several iterations of going back and forth between 
QA and dev. Sometimes the QA folks forgot to test if the bug was really 
fixed after the third iteration as humans make errors, especially when it 
is a boring task.

After a lifetime of 2 years, the customer usually wanted a redesign, and 
the next developer throwed everything of the old website away, as not much 
of it was reusable. When something broke during the life of such an 
application/website, the company and developers noticed that by the 
customer calling them.

These days I think often about my time in my first job and the customers 
calling the company regarding bugs. Right now the Erlang core of CouchDB 
reminds me quite often how our customers called us when something broke. 
For CouchDB 2 it is not the users that call us right now, but PouchDB and 
Fauxton when their testsuites gets red or incredible slow.

When I was starting to work on Fauxton we were in a somehow similar 
situation, a merge usually had 2-3 fixes on the same or next day, because 
something somehow broke. A proper review for a change took incredible much 
time and every pull request needed a lot of iterations. The most evil 
changes were the ones where you fixed a bug in one area, and a completely 
different area broke. These bugs were so evil because they were surprising 
and unexpected.

After a real big issue we tried out that every change would need unit 
and/or integration tests for several weeks.

Writing unit tests was very hard these days, most of the code was not 
designed to get tested afterwards. So most of the tests written in that 
time were Selenium based integration tests. This was better than nothing 
but they were slow. Nevertheless, they caught a lot of bugs, even before 
code was sent to the review. When we made a switch to React and a different 
architecture for our components and their communication, better and easier 
unit/functional testing was one of the many design goals.

Writing good Selenium tests is incredible hard, as you basically face the 
same problems you have in the distributed systems world in the world of 
modern UIs. State and data changes asynchronously and you never know when 
the result of your request, which changes the UI is returned. It even gets 
harder by putting this on top of an eventually consistent system :)

Sometimes we also pair-programmed in a video call in front of a test where 
coming up with a good test was the problem, as we also had to share and 
gain knowledge in the team how to write tests. One mistake we made was of 
not fixing flaky tests immediately and adding more and more flaky tests and 
just restarting the CI for a few weeks. At some point our CI system just 
collapsed and it took the team a long time to fix those, partly also 
because tests were a new topic and it is hard to understand why Selenium 
tests fail in an async JS application with a cluster beneath it which is 
eventually consistent. Additionally it decreased the confidence in a red 
tests signaling a bug.

After fixing the testsuite in this early crisis we've got pretty stable 
tests. Additionally reviews take much less time. Reviews got even faster 
when we added a pre-check for our coding guidelines to the testsuite. If 
you take a look at a day in the time when we began writing tests, we 
sometimes lost an hour to write those tests, because it was hard. Looking 
just at this short timeframe it felt like we were losing time. But if we 
take a look at a longer timeframe we did not decrease in our speed to 
deliver features for several reasons: less bugfixes needed after code got 
on master, less tickets and communication because of less bugs, faster 
reviews because many bugs are caught before the reviewer takes a look and 
so on. Some people in the team still hate writing tests but also everyone 
on the team realized and also mentions that we have no other options.

Right now I am seeing the same pattern we observed some time ago in Fauxton 
for CouchDB's Erlang core :

 - not noticing something broke (if you break it, will you even notice?)
 - eventually a user in the broader sense notices something broke
 - a lot of patching to fix patches where stuff broke in another place
 - regressions
 - it takes time to review code properly because of a large api surface and 
a lot of functionality

I would like to propose that every PR needs unit, functional AND/OR 
integration tests for CouchDB's Erlang core, too - given it is not covered 
by an existing test that run. Maybe we could try the new approach for 3 
months and then decide if we keep it.

Most of CouchDB's bugs are getting noticed by interacting with the HTTP 
API. A simple integration test that CouchDB compiles and the bug got fixed 
/ the feature works would be enough as a minimum.

Let's take a look what we have for testing now:

- Erlang unit tests
- Erlang functional tests
- integration tests written in Erlang, which use ibrowse [1]

- integration tests in JavaScript which test the backdoor ports

- Fauxton's Integration tests simulating a user and their browsers
- PouchDB's integration tests

So there are a lot of different ways to write tests. One problem for 
testing is our multi-repository approach. The multiple PRs over different 
repos must somehow get united again. Here is how we could solve it as a 
first step [2]. The testsuite is kicked off once all PRs are opened by 
putting the branchnames for the repositories that should not be master into 
the config fields. 

When the testsuite is green another button could appear, to automate the 
merge of all repos involved int he change, but that would we an additional 
feature [3]. 

The reviewer and the submitter of the PR would be both responsible that a 
merge is covered by tests and the testsuite is green.

Important is that it is a first step and depending how creative we get it 
can indefinitely improve, like with Fauxton where a lot changed in the very 
short timeframe since we are requiring tests. The important step is that we 
somehow start at some point, and everybody is welcome and invited (and 
should feel empowered) to make the processes how we write software and 
tests better.

Flaky tests are a problem and should get fixed immediately, as an 
unreliable testsuite does not make much sense (remember the crying wolf kid 
story). We learned that quite painfully in Fauxton.

The change will need some discipline and maybe also stepping temporarily 
out of our comfort zone, but I am certain we will gain speed, improve our 
daily workflows and improve CouchDB's Quality over the long term. I think a 
3 month timeframe to try the new approach would be a good time to find out 
if we are improving or things get worse.


Best,
Robert :)


[1] 
https://github.com/apache/couchdb-chttpd/blob/master/test/chttpd_db_test.erl#L67
[2] https://cloudup.com/cOgxRPbt9aP
[3] https://cloudup.com/c0VJDIoYqmI

Re: [PROPOSAL] Improving the quality of CouchDB

[Alexander Shorin <kx...@gmail.com>]

Hi Robert,

TL;DR what's the step-by-step plan?


P.S.

On Mon, Sep 7, 2015 at 5:49 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
> [2] https://cloudup.com/cOgxRPbt9aP
> [3] https://cloudup.com/c0VJDIoYqmI

Flash... /:

--
,,,^..^,,,

Re: [PROPOSAL] Improving the quality of CouchDB

[Sebastian Rothbucher <se...@googlemail.com>]

Hi Robert,

also many thanks from my side for writing all this down! There are in fact
some integration tests coming with CouchDB using the HTTP API (can be run
via `make javascript`) which need to be transferred to run against the
cluster port (instead of local). I started working on this on my fork, but
there's quite some work left.

Maybe that's also a part of the puzzle besides having more unit tests or
leveraging PouchDB et all...

Best
    Sebastian

On Tue, Sep 8, 2015 at 11:38 AM, Jason Smith <ja...@gmail.com>
wrote:

> Hi, Robert. Well, that was very long and I read every word! Thank you for
> your thoughtful assessment and feedback!
>
> You make many points across several different areas of concern. It sounds
> to me like the primary thrust of your argument is about continuous
> integration--somehow making it more easy and more transparent for
> developers (seasoned and novice) to see the effects of their change. Is
> that right?
>
> So, what is the immediate next step that you propose to fix or improve?
> What is the most squeaky wheel, or the most modest gain that we can make?
>
> Thanks again!
>
> On Mon, Sep 7, 2015 at 9:49 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
>
> > Hi list,
> >
> > I had my first real programming job in a Web Agency. We were building
> > mobile websites for our external customers. The development process
> > followed a strict waterfall model, after you finished your project it was
> > thrown over the fence to the QA folks, which tested the website manually.
> > At some point, when all found bugs were fixed, the product got released.
> > Sometimes a bugfix took several iterations of going back and forth
> between
> > QA and dev. Sometimes the QA folks forgot to test if the bug was really
> > fixed after the third iteration as humans make errors, especially when it
> > is a boring task.
> >
> > After a lifetime of 2 years, the customer usually wanted a redesign, and
> > the next developer throwed everything of the old website away, as not
> much
> > of it was reusable. When something broke during the life of such an
> > application/website, the company and developers noticed that by the
> > customer calling them.
> >
> > These days I think often about my time in my first job and the customers
> > calling the company regarding bugs. Right now the Erlang core of CouchDB
> > reminds me quite often how our customers called us when something broke.
> > For CouchDB 2 it is not the users that call us right now, but PouchDB and
> > Fauxton when their testsuites gets red or incredible slow.
> >
> > When I was starting to work on Fauxton we were in a somehow similar
> > situation, a merge usually had 2-3 fixes on the same or next day, because
> > something somehow broke. A proper review for a change took incredible
> much
> > time and every pull request needed a lot of iterations. The most evil
> > changes were the ones where you fixed a bug in one area, and a completely
> > different area broke. These bugs were so evil because they were
> surprising
> > and unexpected.
> >
> > After a real big issue we tried out that every change would need unit
> > and/or integration tests for several weeks.
> >
> > Writing unit tests was very hard these days, most of the code was not
> > designed to get tested afterwards. So most of the tests written in that
> > time were Selenium based integration tests. This was better than nothing
> > but they were slow. Nevertheless, they caught a lot of bugs, even before
> > code was sent to the review. When we made a switch to React and a
> different
> > architecture for our components and their communication, better and
> easier
> > unit/functional testing was one of the many design goals.
> >
> > Writing good Selenium tests is incredible hard, as you basically face the
> > same problems you have in the distributed systems world in the world of
> > modern UIs. State and data changes asynchronously and you never know when
> > the result of your request, which changes the UI is returned. It even
> gets
> > harder by putting this on top of an eventually consistent system :)
> >
> > Sometimes we also pair-programmed in a video call in front of a test
> where
> > coming up with a good test was the problem, as we also had to share and
> > gain knowledge in the team how to write tests. One mistake we made was of
> > not fixing flaky tests immediately and adding more and more flaky tests
> and
> > just restarting the CI for a few weeks. At some point our CI system just
> > collapsed and it took the team a long time to fix those, partly also
> > because tests were a new topic and it is hard to understand why Selenium
> > tests fail in an async JS application with a cluster beneath it which is
> > eventually consistent. Additionally it decreased the confidence in a red
> > tests signaling a bug.
> >
> > After fixing the testsuite in this early crisis we've got pretty stable
> > tests. Additionally reviews take much less time. Reviews got even faster
> > when we added a pre-check for our coding guidelines to the testsuite. If
> > you take a look at a day in the time when we began writing tests, we
> > sometimes lost an hour to write those tests, because it was hard. Looking
> > just at this short timeframe it felt like we were losing time. But if we
> > take a look at a longer timeframe we did not decrease in our speed to
> > deliver features for several reasons: less bugfixes needed after code got
> > on master, less tickets and communication because of less bugs, faster
> > reviews because many bugs are caught before the reviewer takes a look and
> > so on. Some people in the team still hate writing tests but also everyone
> > on the team realized and also mentions that we have no other options.
> >
> > Right now I am seeing the same pattern we observed some time ago in
> > Fauxton for CouchDB's Erlang core :
> >
> >  - not noticing something broke (if you break it, will you even notice?)
> >  - eventually a user in the broader sense notices something broke
> >  - a lot of patching to fix patches where stuff broke in another place
> >  - regressions
> >  - it takes time to review code properly because of a large api surface
> > and a lot of functionality
> >
> > I would like to propose that every PR needs unit, functional AND/OR
> > integration tests for CouchDB's Erlang core, too - given it is not
> covered
> > by an existing test that run. Maybe we could try the new approach for 3
> > months and then decide if we keep it.
> >
> > Most of CouchDB's bugs are getting noticed by interacting with the HTTP
> > API. A simple integration test that CouchDB compiles and the bug got
> fixed
> > / the feature works would be enough as a minimum.
> >
> > Let's take a look what we have for testing now:
> >
> > - Erlang unit tests
> > - Erlang functional tests
> > - integration tests written in Erlang, which use ibrowse [1]
> >
> > - integration tests in JavaScript which test the backdoor ports
> >
> > - Fauxton's Integration tests simulating a user and their browsers
> > - PouchDB's integration tests
> >
> > So there are a lot of different ways to write tests. One problem for
> > testing is our multi-repository approach. The multiple PRs over different
> > repos must somehow get united again. Here is how we could solve it as a
> > first step [2]. The testsuite is kicked off once all PRs are opened by
> > putting the branchnames for the repositories that should not be master
> into
> > the config fields.
> >
> > When the testsuite is green another button could appear, to automate the
> > merge of all repos involved int he change, but that would we an
> additional
> > feature [3].
> >
> > The reviewer and the submitter of the PR would be both responsible that a
> > merge is covered by tests and the testsuite is green.
> >
> > Important is that it is a first step and depending how creative we get it
> > can indefinitely improve, like with Fauxton where a lot changed in the
> very
> > short timeframe since we are requiring tests. The important step is that
> we
> > somehow start at some point, and everybody is welcome and invited (and
> > should feel empowered) to make the processes how we write software and
> > tests better.
> >
> > Flaky tests are a problem and should get fixed immediately, as an
> > unreliable testsuite does not make much sense (remember the crying wolf
> kid
> > story). We learned that quite painfully in Fauxton.
> >
> > The change will need some discipline and maybe also stepping temporarily
> > out of our comfort zone, but I am certain we will gain speed, improve our
> > daily workflows and improve CouchDB's Quality over the long term. I
> think a
> > 3 month timeframe to try the new approach would be a good time to find
> out
> > if we are improving or things get worse.
> >
> >
> > Best,
> > Robert :)
> >
> >
> > [1]
> >
> https://github.com/apache/couchdb-chttpd/blob/master/test/chttpd_db_test.erl#L67
> > [2] https://cloudup.com/cOgxRPbt9aP
> > [3] https://cloudup.com/c0VJDIoYqmI
> >
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Jason Smith <ja...@gmail.com>]

On Thu, Sep 10, 2015 at 8:58 PM, Ilya Khlopotov <ii...@ca.ibm.com> wrote:

> >  Is there a way to publish an Erlang module for downstream users
> independently of git? If yes, we could move back to a single-git-tree
> model for
> CouchDB, but we can still release parts as independent modules. I know
> there have
> been a few attempts at Erlang package managers, but has anything taken off?
>
> There is https://hex.pm/. It is for erlang and elixir based projects and
> has 917 packages currently.
> However for whatever reason erlang community wants to implement their own
> and failed so far.
>

Surely there is a system which people know, which works, and which is quite
easy: Release tarballs of source code. This worked for most of the history
of the free software community.

Release tarballs satisfy both of Jan's requirements:

1. Separate code into different modules.
2. Make a module available to downstream users.

The only thing I see missing (perhaps implicit in Jan's idea) is automatic
dependency chasing. With tarballs, it is only a matter of reading the
release notes and following up to download more packages. In the case of
CouchDB, I do not think our dependency tree is very deep; so that might not
be so bad.

(Tarballs strike me as more or less equivalent to Git tags, it's just a
different file transfer protocol, really.)

Re: [PROPOSAL] Improving the quality of CouchDB

[Ilya Khlopotov <ii...@ca.ibm.com>]

>  Is there a way to publish an Erlang module for downstream users
independently of git? If yes, we could move back to a single-git-tree model
for
CouchDB, but we can still release parts as independent modules. I know
there have
been a few attempts at Erlang package managers, but has anything taken off?

There is https://hex.pm/. It is for erlang and elixir based projects and
has 917 packages currently.
However for whatever reason erlang community wants to implement their own
and failed so far.

BR,
ILYA



From:	Jan Lehnardt <ja...@apache.org>
To:	dev@couchdb.apache.org
Date:	2015/09/10 05:11 AM
Subject:	Re: [PROPOSAL] Improving the quality of CouchDB




> On 10 Sep 2015, at 12:18, Alexander Shorin <kx...@gmail.com> wrote:
>
> On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com>
wrote:
>> I dont think CI is a dream, It should take an afternoon at most to get
>> CouchDB setup on travis on a single platform to ensure no major
regressions
>> come though. If anyone wants help doing that feel free to ping me on
>> #pouchdb / #couchdb, we already test couchdb master in travis.
>
> It is a dream for now.
>
> We have travis enabled, but it has practical flaw: it cannot handle
> changes that affected multiple repositories at the same time. Also,
> for each repository we have to build full CouchDB to test what is a
> bit overkill.
>
>> I do think multirepos is an issue and that solution is not so simple, we
>> went through the same issues with pouchdb as we attempted to split out
our
>> repository.
>
> Well, there was a good theory that each of our repository will contain
> an app which could be reused outside of CouchDB or being easily
> replaced by alternative implementation. Suddenly, for now most of our
> apps are heavy coupled

There are two distinct functions that git does for us here:
1. Separate code into different modules.
2. Make a module available to downstream users.

In a Node/npm, Python/pip, Ruby/Gems world, 1. is solved by git, 2. is
solved by npm/pip/Gems. Is there a way to publish an Erlang module for
downstream users
independently of git? If yes, we could move back to a single-git-tree model
for
CouchDB, but we can still release parts as independent modules. I know
there have
been a few attempts at Erlang package managers, but has anything taken off?

* * *

For Hoodie, we came up with this, based on our experience with using the
full top level test suite on each module (it is indeed overkill):

- separate modules
- one module that is the top level module (like our couchdb.git) that only
includes module dependencies and a top-level integration test suite.
- two release trains: stable and next.
- a service that attempts to create a new version of the top level module
with every commit to a dependent module. E.g. we automatically bump the
dependency in a checkout of the top level module, run the integration
tests, if they fail, nothing happens and we report that back on the commit
to the dependent module. If it succeeds, we automatically determine the
next version number on the next release train (SemVer makes that easy) and
release the new top level module with the updated dependency.
- once we are comfortable that the next release train is stable enough for
a release, we change switch out the release trains: next becomes stable,
and we have a new empty next train (npm uses release tags for this, but the
mechanics are irrelevant, although something like that needs to be
supported in the package manager).

This requires:
- Strict SemVer on all modules (we use
https://github.com/semantic-release/semantic-release) to make sure humans
can’t screw this up.
- Breaking change detection, so we don’t accidentally release a BC break in
a non-major version (available as semantic-release plugins).
- the service that does the auto-update dance described above.

This is a lot of infrastructure (and some of it we are still developing),
but once in place, it is really simple to not screw this up and it takes a
huge cognitive load off of everyday development.

Semantic-Release is language agnostic in theory, but afaik there are only
Node and Python versions, so we’d have to find someone to make an Erlang
variant. Also, the process uses npm heavily, so some sort of package
management system would be nice.

Eventually, I’d love to see this for CouchDB.

PS: The person who invented semantic-release works for my company, and I’d
be more than happy to have them have a crack at this. If anyone wants to
sponsor an erlang-semantic-release, please get in touch.

* * *

Either way, this is a bit of a long shot. For now:

We should do what Dale proposed in 2.: pin top level dependencies. I
believe that was the plan all along, but with a fast moving 2.0 we just
didn’t do it yet. Now is the time to pin versions and manually upgrade them
after thorough testing (can still be in PRs and tested on Travis and
everything).

That means all of these modules (and their sub-deps) should get releases
and release tags and those tags need to be filled into our top-level
rebar.config.script:

https://github.com/apache/couchdb/blob/master/rebar.config.script#L19-L53

Let’s get going :)

Thanks
Jan
--




>
>> 1. Dont split out into lots of repositories, if you put those components
>> inside the CouchDB repo, then they will get the CouchDB tests run
against
>> them when changes are made and you wont break the CouchDB repo.
>
> It's good in retrospective, but useless now. We have to decide how our
> CI will work: with multirepo layout or first fold all core repos back
> to the one. I think we are ready to revise this too and probably this
> should be done before CI work.
>
>> 2. Anything that does live outside the CouchDB repo, pin their version
>> inside the CouchDB repo, dont have commits to subproject X automatically
be
>> applied to CouchDB, that means you can commit whatever you want to X but
>> CouchDB will still be working, when you come to update the version of X
you
>> pinned, you can see that it breaks and not update until it is fixed.
>
> It's good strategy for milestones or when you don't have to manage 40+
repos (:
>
>>  2.5 Another strategy that can help with this is to have the full
CouchDB
>> test suite run inside X, so changes to X will run the full CouchDB suite
>> with an updated X
>
> This works, but until changes are belongs to a single repository. See
> above about travis issue.
>
>
> P.S. Jason, you are right again (:
>
> --
> ,,,^..^,,,

--
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/

Re: [PROPOSAL] Improving the quality of CouchDB

[Robert Kowalski <ro...@kowalski.gd>]

Cool! I like the idea! :)


On Thu, Sep 10, 2015 at 6:38 PM, Jan Lehnardt <ja...@apache.org> wrote:

>
> > On 10 Sep 2015, at 18:31, Robert Kowalski <ro...@kowalski.gd> wrote:
> >
> > Hi Jan,
> >
> > how would testing of a PR/change work in practice?
> >
> > Example: I am opening 3 Pull-Requests that depend on each other, one in
> > chttpd, one in fabric and one in rexi.
> >
> > Or is the idea that the testsuite turns red when the deps are updated in
> > the top level module?
>
> Yeah, if the top module only works with all three PRs in concert, you get
> two broken builds and then a working build.
>
> (some of the details of what I explained earlier might make it sound that
> this isn’t possible, rest assured, we’ve considered this case :)
>
> Best
> Jan
> --
>
> >
> > On Thu, Sep 10, 2015 at 2:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> >
> >>
> >>> On 10 Sep 2015, at 12:18, Alexander Shorin <kx...@gmail.com> wrote:
> >>>
> >>> On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com>
> >> wrote:
> >>>> I dont think CI is a dream, It should take an afternoon at most to get
> >>>> CouchDB setup on travis on a single platform to ensure no major
> >> regressions
> >>>> come though. If anyone wants help doing that feel free to ping me on
> >>>> #pouchdb / #couchdb, we already test couchdb master in travis.
> >>>
> >>> It is a dream for now.
> >>>
> >>> We have travis enabled, but it has practical flaw: it cannot handle
> >>> changes that affected multiple repositories at the same time. Also,
> >>> for each repository we have to build full CouchDB to test what is a
> >>> bit overkill.
> >>>
> >>>> I do think multirepos is an issue and that solution is not so simple,
> we
> >>>> went through the same issues with pouchdb as we attempted to split out
> >> our
> >>>> repository.
> >>>
> >>> Well, there was a good theory that each of our repository will contain
> >>> an app which could be reused outside of CouchDB or being easily
> >>> replaced by alternative implementation. Suddenly, for now most of our
> >>> apps are heavy coupled
> >>
> >> There are two distinct functions that git does for us here:
> >> 1. Separate code into different modules.
> >> 2. Make a module available to downstream users.
> >>
> >> In a Node/npm, Python/pip, Ruby/Gems world, 1. is solved by git, 2. is
> >> solved by npm/pip/Gems. Is there a way to publish an Erlang module for
> >> downstream users
> >> independently of git? If yes, we could move back to a single-git-tree
> >> model for
> >> CouchDB, but we can still release parts as independent modules. I know
> >> there have
> >> been a few attempts at Erlang package managers, but has anything taken
> off?
> >>
> >> * * *
> >>
> >> For Hoodie, we came up with this, based on our experience with using the
> >> full top level test suite on each module (it is indeed overkill):
> >>
> >> - separate modules
> >> - one module that is the top level module (like our couchdb.git) that
> only
> >> includes module dependencies and a top-level integration test suite.
> >> - two release trains: stable and next.
> >> - a service that attempts to create a new version of the top level
> module
> >> with every commit to a dependent module. E.g. we automatically bump the
> >> dependency in a checkout of the top level module, run the integration
> >> tests, if they fail, nothing happens and we report that back on the
> commit
> >> to the dependent module. If it succeeds, we automatically determine the
> >> next version number on the next release train (SemVer makes that easy)
> and
> >> release the new top level module with the updated dependency.
> >> - once we are comfortable that the next release train is stable enough
> for
> >> a release, we change switch out the release trains: next becomes stable,
> >> and we have a new empty next train (npm uses release tags for this, but
> the
> >> mechanics are irrelevant, although something like that needs to be
> >> supported in the package manager).
> >>
> >> This requires:
> >> - Strict SemVer on all modules (we use
> >> https://github.com/semantic-release/semantic-release) to make sure
> humans
> >> can’t screw this up.
> >> - Breaking change detection, so we don’t accidentally release a BC break
> >> in a non-major version (available as semantic-release plugins).
> >> - the service that does the auto-update dance described above.
> >>
> >> This is a lot of infrastructure (and some of it we are still
> developing),
> >> but once in place, it is really simple to not screw this up and it
> takes a
> >> huge cognitive load off of everyday development.
> >>
> >> Semantic-Release is language agnostic in theory, but afaik there are
> only
> >> Node and Python versions, so we’d have to find someone to make an Erlang
> >> variant. Also, the process uses npm heavily, so some sort of package
> >> management system would be nice.
> >>
> >> Eventually, I’d love to see this for CouchDB.
> >>
> >> PS: The person who invented semantic-release works for my company, and
> I’d
> >> be more than happy to have them have a crack at this. If anyone wants to
> >> sponsor an erlang-semantic-release, please get in touch.
> >>
> >> * * *
> >>
> >> Either way, this is a bit of a long shot. For now:
> >>
> >> We should do what Dale proposed in 2.: pin top level dependencies. I
> >> believe that was the plan all along, but with a fast moving 2.0 we just
> >> didn’t do it yet. Now is the time to pin versions and manually upgrade
> them
> >> after thorough testing (can still be in PRs and tested on Travis and
> >> everything).
> >>
> >> That means all of these modules (and their sub-deps) should get releases
> >> and release tags and those tags need to be filled into our top-level
> >> rebar.config.script:
> >>
> >>
> https://github.com/apache/couchdb/blob/master/rebar.config.script#L19-L53
> >>
> >> Let’s get going :)
> >>
> >> Thanks
> >> Jan
> >> --
> >>
> >>
> >>
> >>
> >>>
> >>>> 1. Dont split out into lots of repositories, if you put those
> components
> >>>> inside the CouchDB repo, then they will get the CouchDB tests run
> >> against
> >>>> them when changes are made and you wont break the CouchDB repo.
> >>>
> >>> It's good in retrospective, but useless now. We have to decide how our
> >>> CI will work: with multirepo layout or first fold all core repos back
> >>> to the one. I think we are ready to revise this too and probably this
> >>> should be done before CI work.
> >>>
> >>>> 2. Anything that does live outside the CouchDB repo, pin their version
> >>>> inside the CouchDB repo, dont have commits to subproject X
> >> automatically be
> >>>> applied to CouchDB, that means you can commit whatever you want to X
> but
> >>>> CouchDB will still be working, when you come to update the version of
> X
> >> you
> >>>> pinned, you can see that it breaks and not update until it is fixed.
> >>>
> >>> It's good strategy for milestones or when you don't have to manage 40+
> >> repos (:
> >>>
> >>>> 2.5 Another strategy that can help with this is to have the full
> >> CouchDB
> >>>> test suite run inside X, so changes to X will run the full CouchDB
> suite
> >>>> with an updated X
> >>>
> >>> This works, but until changes are belongs to a single repository. See
> >>> above about travis issue.
> >>>
> >>>
> >>> P.S. Jason, you are right again (:
> >>>
> >>> --
> >>> ,,,^..^,,,
> >>
> >> --
> >> Professional Support for Apache CouchDB:
> >> http://www.neighbourhood.ie/couchdb-support/
> >>
> >>
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Jan Lehnardt <ja...@apache.org>]

> On 10 Sep 2015, at 18:31, Robert Kowalski <ro...@kowalski.gd> wrote:
> 
> Hi Jan,
> 
> how would testing of a PR/change work in practice?
> 
> Example: I am opening 3 Pull-Requests that depend on each other, one in
> chttpd, one in fabric and one in rexi.
> 
> Or is the idea that the testsuite turns red when the deps are updated in
> the top level module?

Yeah, if the top module only works with all three PRs in concert, you get two broken builds and then a working build.

(some of the details of what I explained earlier might make it sound that this isn’t possible, rest assured, we’ve considered this case :)

Best
Jan
--

> 
> On Thu, Sep 10, 2015 at 2:09 PM, Jan Lehnardt <ja...@apache.org> wrote:
> 
>> 
>>> On 10 Sep 2015, at 12:18, Alexander Shorin <kx...@gmail.com> wrote:
>>> 
>>> On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com>
>> wrote:
>>>> I dont think CI is a dream, It should take an afternoon at most to get
>>>> CouchDB setup on travis on a single platform to ensure no major
>> regressions
>>>> come though. If anyone wants help doing that feel free to ping me on
>>>> #pouchdb / #couchdb, we already test couchdb master in travis.
>>> 
>>> It is a dream for now.
>>> 
>>> We have travis enabled, but it has practical flaw: it cannot handle
>>> changes that affected multiple repositories at the same time. Also,
>>> for each repository we have to build full CouchDB to test what is a
>>> bit overkill.
>>> 
>>>> I do think multirepos is an issue and that solution is not so simple, we
>>>> went through the same issues with pouchdb as we attempted to split out
>> our
>>>> repository.
>>> 
>>> Well, there was a good theory that each of our repository will contain
>>> an app which could be reused outside of CouchDB or being easily
>>> replaced by alternative implementation. Suddenly, for now most of our
>>> apps are heavy coupled
>> 
>> There are two distinct functions that git does for us here:
>> 1. Separate code into different modules.
>> 2. Make a module available to downstream users.
>> 
>> In a Node/npm, Python/pip, Ruby/Gems world, 1. is solved by git, 2. is
>> solved by npm/pip/Gems. Is there a way to publish an Erlang module for
>> downstream users
>> independently of git? If yes, we could move back to a single-git-tree
>> model for
>> CouchDB, but we can still release parts as independent modules. I know
>> there have
>> been a few attempts at Erlang package managers, but has anything taken off?
>> 
>> * * *
>> 
>> For Hoodie, we came up with this, based on our experience with using the
>> full top level test suite on each module (it is indeed overkill):
>> 
>> - separate modules
>> - one module that is the top level module (like our couchdb.git) that only
>> includes module dependencies and a top-level integration test suite.
>> - two release trains: stable and next.
>> - a service that attempts to create a new version of the top level module
>> with every commit to a dependent module. E.g. we automatically bump the
>> dependency in a checkout of the top level module, run the integration
>> tests, if they fail, nothing happens and we report that back on the commit
>> to the dependent module. If it succeeds, we automatically determine the
>> next version number on the next release train (SemVer makes that easy) and
>> release the new top level module with the updated dependency.
>> - once we are comfortable that the next release train is stable enough for
>> a release, we change switch out the release trains: next becomes stable,
>> and we have a new empty next train (npm uses release tags for this, but the
>> mechanics are irrelevant, although something like that needs to be
>> supported in the package manager).
>> 
>> This requires:
>> - Strict SemVer on all modules (we use
>> https://github.com/semantic-release/semantic-release) to make sure humans
>> can’t screw this up.
>> - Breaking change detection, so we don’t accidentally release a BC break
>> in a non-major version (available as semantic-release plugins).
>> - the service that does the auto-update dance described above.
>> 
>> This is a lot of infrastructure (and some of it we are still developing),
>> but once in place, it is really simple to not screw this up and it takes a
>> huge cognitive load off of everyday development.
>> 
>> Semantic-Release is language agnostic in theory, but afaik there are only
>> Node and Python versions, so we’d have to find someone to make an Erlang
>> variant. Also, the process uses npm heavily, so some sort of package
>> management system would be nice.
>> 
>> Eventually, I’d love to see this for CouchDB.
>> 
>> PS: The person who invented semantic-release works for my company, and I’d
>> be more than happy to have them have a crack at this. If anyone wants to
>> sponsor an erlang-semantic-release, please get in touch.
>> 
>> * * *
>> 
>> Either way, this is a bit of a long shot. For now:
>> 
>> We should do what Dale proposed in 2.: pin top level dependencies. I
>> believe that was the plan all along, but with a fast moving 2.0 we just
>> didn’t do it yet. Now is the time to pin versions and manually upgrade them
>> after thorough testing (can still be in PRs and tested on Travis and
>> everything).
>> 
>> That means all of these modules (and their sub-deps) should get releases
>> and release tags and those tags need to be filled into our top-level
>> rebar.config.script:
>> 
>> https://github.com/apache/couchdb/blob/master/rebar.config.script#L19-L53
>> 
>> Let’s get going :)
>> 
>> Thanks
>> Jan
>> --
>> 
>> 
>> 
>> 
>>> 
>>>> 1. Dont split out into lots of repositories, if you put those components
>>>> inside the CouchDB repo, then they will get the CouchDB tests run
>> against
>>>> them when changes are made and you wont break the CouchDB repo.
>>> 
>>> It's good in retrospective, but useless now. We have to decide how our
>>> CI will work: with multirepo layout or first fold all core repos back
>>> to the one. I think we are ready to revise this too and probably this
>>> should be done before CI work.
>>> 
>>>> 2. Anything that does live outside the CouchDB repo, pin their version
>>>> inside the CouchDB repo, dont have commits to subproject X
>> automatically be
>>>> applied to CouchDB, that means you can commit whatever you want to X but
>>>> CouchDB will still be working, when you come to update the version of X
>> you
>>>> pinned, you can see that it breaks and not update until it is fixed.
>>> 
>>> It's good strategy for milestones or when you don't have to manage 40+
>> repos (:
>>> 
>>>> 2.5 Another strategy that can help with this is to have the full
>> CouchDB
>>>> test suite run inside X, so changes to X will run the full CouchDB suite
>>>> with an updated X
>>> 
>>> This works, but until changes are belongs to a single repository. See
>>> above about travis issue.
>>> 
>>> 
>>> P.S. Jason, you are right again (:
>>> 
>>> --
>>> ,,,^..^,,,
>> 
>> --
>> Professional Support for Apache CouchDB:
>> http://www.neighbourhood.ie/couchdb-support/
>> 
>> 

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/

Re: [PROPOSAL] Improving the quality of CouchDB

[Robert Kowalski <ro...@kowalski.gd>]

Hi Jan,

how would testing of a PR/change work in practice?

Example: I am opening 3 Pull-Requests that depend on each other, one in
chttpd, one in fabric and one in rexi.

Or is the idea that the testsuite turns red when the deps are updated in
the top level module?

On Thu, Sep 10, 2015 at 2:09 PM, Jan Lehnardt <ja...@apache.org> wrote:

>
> > On 10 Sep 2015, at 12:18, Alexander Shorin <kx...@gmail.com> wrote:
> >
> > On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com>
> wrote:
> >> I dont think CI is a dream, It should take an afternoon at most to get
> >> CouchDB setup on travis on a single platform to ensure no major
> regressions
> >> come though. If anyone wants help doing that feel free to ping me on
> >> #pouchdb / #couchdb, we already test couchdb master in travis.
> >
> > It is a dream for now.
> >
> > We have travis enabled, but it has practical flaw: it cannot handle
> > changes that affected multiple repositories at the same time. Also,
> > for each repository we have to build full CouchDB to test what is a
> > bit overkill.
> >
> >> I do think multirepos is an issue and that solution is not so simple, we
> >> went through the same issues with pouchdb as we attempted to split out
> our
> >> repository.
> >
> > Well, there was a good theory that each of our repository will contain
> > an app which could be reused outside of CouchDB or being easily
> > replaced by alternative implementation. Suddenly, for now most of our
> > apps are heavy coupled
>
> There are two distinct functions that git does for us here:
> 1. Separate code into different modules.
> 2. Make a module available to downstream users.
>
> In a Node/npm, Python/pip, Ruby/Gems world, 1. is solved by git, 2. is
> solved by npm/pip/Gems. Is there a way to publish an Erlang module for
> downstream users
> independently of git? If yes, we could move back to a single-git-tree
> model for
> CouchDB, but we can still release parts as independent modules. I know
> there have
> been a few attempts at Erlang package managers, but has anything taken off?
>
> * * *
>
> For Hoodie, we came up with this, based on our experience with using the
> full top level test suite on each module (it is indeed overkill):
>
> - separate modules
> - one module that is the top level module (like our couchdb.git) that only
> includes module dependencies and a top-level integration test suite.
> - two release trains: stable and next.
> - a service that attempts to create a new version of the top level module
> with every commit to a dependent module. E.g. we automatically bump the
> dependency in a checkout of the top level module, run the integration
> tests, if they fail, nothing happens and we report that back on the commit
> to the dependent module. If it succeeds, we automatically determine the
> next version number on the next release train (SemVer makes that easy) and
> release the new top level module with the updated dependency.
> - once we are comfortable that the next release train is stable enough for
> a release, we change switch out the release trains: next becomes stable,
> and we have a new empty next train (npm uses release tags for this, but the
> mechanics are irrelevant, although something like that needs to be
> supported in the package manager).
>
> This requires:
> - Strict SemVer on all modules (we use
> https://github.com/semantic-release/semantic-release) to make sure humans
> can’t screw this up.
> - Breaking change detection, so we don’t accidentally release a BC break
> in a non-major version (available as semantic-release plugins).
> - the service that does the auto-update dance described above.
>
> This is a lot of infrastructure (and some of it we are still developing),
> but once in place, it is really simple to not screw this up and it takes a
> huge cognitive load off of everyday development.
>
> Semantic-Release is language agnostic in theory, but afaik there are only
> Node and Python versions, so we’d have to find someone to make an Erlang
> variant. Also, the process uses npm heavily, so some sort of package
> management system would be nice.
>
> Eventually, I’d love to see this for CouchDB.
>
> PS: The person who invented semantic-release works for my company, and I’d
> be more than happy to have them have a crack at this. If anyone wants to
> sponsor an erlang-semantic-release, please get in touch.
>
> * * *
>
> Either way, this is a bit of a long shot. For now:
>
> We should do what Dale proposed in 2.: pin top level dependencies. I
> believe that was the plan all along, but with a fast moving 2.0 we just
> didn’t do it yet. Now is the time to pin versions and manually upgrade them
> after thorough testing (can still be in PRs and tested on Travis and
> everything).
>
> That means all of these modules (and their sub-deps) should get releases
> and release tags and those tags need to be filled into our top-level
> rebar.config.script:
>
> https://github.com/apache/couchdb/blob/master/rebar.config.script#L19-L53
>
> Let’s get going :)
>
> Thanks
> Jan
> --
>
>
>
>
> >
> >> 1. Dont split out into lots of repositories, if you put those components
> >> inside the CouchDB repo, then they will get the CouchDB tests run
> against
> >> them when changes are made and you wont break the CouchDB repo.
> >
> > It's good in retrospective, but useless now. We have to decide how our
> > CI will work: with multirepo layout or first fold all core repos back
> > to the one. I think we are ready to revise this too and probably this
> > should be done before CI work.
> >
> >> 2. Anything that does live outside the CouchDB repo, pin their version
> >> inside the CouchDB repo, dont have commits to subproject X
> automatically be
> >> applied to CouchDB, that means you can commit whatever you want to X but
> >> CouchDB will still be working, when you come to update the version of X
> you
> >> pinned, you can see that it breaks and not update until it is fixed.
> >
> > It's good strategy for milestones or when you don't have to manage 40+
> repos (:
> >
> >>  2.5 Another strategy that can help with this is to have the full
> CouchDB
> >> test suite run inside X, so changes to X will run the full CouchDB suite
> >> with an updated X
> >
> > This works, but until changes are belongs to a single repository. See
> > above about travis issue.
> >
> >
> > P.S. Jason, you are right again (:
> >
> > --
> > ,,,^..^,,,
>
> --
> Professional Support for Apache CouchDB:
> http://www.neighbourhood.ie/couchdb-support/
>
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Jan Lehnardt <ja...@apache.org>]

> On 10 Sep 2015, at 12:18, Alexander Shorin <kx...@gmail.com> wrote:
> 
> On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com> wrote:
>> I dont think CI is a dream, It should take an afternoon at most to get
>> CouchDB setup on travis on a single platform to ensure no major regressions
>> come though. If anyone wants help doing that feel free to ping me on
>> #pouchdb / #couchdb, we already test couchdb master in travis.
> 
> It is a dream for now.
> 
> We have travis enabled, but it has practical flaw: it cannot handle
> changes that affected multiple repositories at the same time. Also,
> for each repository we have to build full CouchDB to test what is a
> bit overkill.
> 
>> I do think multirepos is an issue and that solution is not so simple, we
>> went through the same issues with pouchdb as we attempted to split out our
>> repository.
> 
> Well, there was a good theory that each of our repository will contain
> an app which could be reused outside of CouchDB or being easily
> replaced by alternative implementation. Suddenly, for now most of our
> apps are heavy coupled

There are two distinct functions that git does for us here:
1. Separate code into different modules.
2. Make a module available to downstream users.

In a Node/npm, Python/pip, Ruby/Gems world, 1. is solved by git, 2. is solved by npm/pip/Gems. Is there a way to publish an Erlang module for downstream users
independently of git? If yes, we could move back to a single-git-tree model for
CouchDB, but we can still release parts as independent modules. I know there have
been a few attempts at Erlang package managers, but has anything taken off?

* * *

For Hoodie, we came up with this, based on our experience with using the full top level test suite on each module (it is indeed overkill):

- separate modules
- one module that is the top level module (like our couchdb.git) that only includes module dependencies and a top-level integration test suite.
- two release trains: stable and next.
- a service that attempts to create a new version of the top level module with every commit to a dependent module. E.g. we automatically bump the dependency in a checkout of the top level module, run the integration tests, if they fail, nothing happens and we report that back on the commit to the dependent module. If it succeeds, we automatically determine the next version number on the next release train (SemVer makes that easy) and release the new top level module with the updated dependency.
- once we are comfortable that the next release train is stable enough for a release, we change switch out the release trains: next becomes stable, and we have a new empty next train (npm uses release tags for this, but the mechanics are irrelevant, although something like that needs to be supported in the package manager).

This requires:
- Strict SemVer on all modules (we use https://github.com/semantic-release/semantic-release) to make sure humans can’t screw this up.
- Breaking change detection, so we don’t accidentally release a BC break in a non-major version (available as semantic-release plugins).
- the service that does the auto-update dance described above.

This is a lot of infrastructure (and some of it we are still developing), but once in place, it is really simple to not screw this up and it takes a huge cognitive load off of everyday development.

Semantic-Release is language agnostic in theory, but afaik there are only Node and Python versions, so we’d have to find someone to make an Erlang variant. Also, the process uses npm heavily, so some sort of package management system would be nice.

Eventually, I’d love to see this for CouchDB.

PS: The person who invented semantic-release works for my company, and I’d be more than happy to have them have a crack at this. If anyone wants to sponsor an erlang-semantic-release, please get in touch.

* * *

Either way, this is a bit of a long shot. For now:

We should do what Dale proposed in 2.: pin top level dependencies. I believe that was the plan all along, but with a fast moving 2.0 we just didn’t do it yet. Now is the time to pin versions and manually upgrade them after thorough testing (can still be in PRs and tested on Travis and everything).

That means all of these modules (and their sub-deps) should get releases and release tags and those tags need to be filled into our top-level rebar.config.script:

https://github.com/apache/couchdb/blob/master/rebar.config.script#L19-L53

Let’s get going :)

Thanks
Jan
--




> 
>> 1. Dont split out into lots of repositories, if you put those components
>> inside the CouchDB repo, then they will get the CouchDB tests run against
>> them when changes are made and you wont break the CouchDB repo.
> 
> It's good in retrospective, but useless now. We have to decide how our
> CI will work: with multirepo layout or first fold all core repos back
> to the one. I think we are ready to revise this too and probably this
> should be done before CI work.
> 
>> 2. Anything that does live outside the CouchDB repo, pin their version
>> inside the CouchDB repo, dont have commits to subproject X automatically be
>> applied to CouchDB, that means you can commit whatever you want to X but
>> CouchDB will still be working, when you come to update the version of X you
>> pinned, you can see that it breaks and not update until it is fixed.
> 
> It's good strategy for milestones or when you don't have to manage 40+ repos (:
> 
>>  2.5 Another strategy that can help with this is to have the full CouchDB
>> test suite run inside X, so changes to X will run the full CouchDB suite
>> with an updated X
> 
> This works, but until changes are belongs to a single repository. See
> above about travis issue.
> 
> 
> P.S. Jason, you are right again (:
> 
> --
> ,,,^..^,,,

-- 
Professional Support for Apache CouchDB:
http://www.neighbourhood.ie/couchdb-support/

Re: [PROPOSAL] Improving the quality of CouchDB

[Alexander Shorin <kx...@gmail.com>]

On Thu, Sep 10, 2015 at 12:40 PM, Dale Harvey <da...@arandomurl.com> wrote:
> I dont think CI is a dream, It should take an afternoon at most to get
> CouchDB setup on travis on a single platform to ensure no major regressions
> come though. If anyone wants help doing that feel free to ping me on
> #pouchdb / #couchdb, we already test couchdb master in travis.

It is a dream for now.

We have travis enabled, but it has practical flaw: it cannot handle
changes that affected multiple repositories at the same time. Also,
for each repository we have to build full CouchDB to test what is a
bit overkill.

> I do think multirepos is an issue and that solution is not so simple, we
> went through the same issues with pouchdb as we attempted to split out our
> repository.

Well, there was a good theory that each of our repository will contain
an app which could be reused outside of CouchDB or being easily
replaced by alternative implementation. Suddenly, for now most of our
apps are heavy coupled

> 1. Dont split out into lots of repositories, if you put those components
> inside the CouchDB repo, then they will get the CouchDB tests run against
> them when changes are made and you wont break the CouchDB repo.

It's good in retrospective, but useless now. We have to decide how our
CI will work: with multirepo layout or first fold all core repos back
to the one. I think we are ready to revise this too and probably this
should be done before CI work.

> 2. Anything that does live outside the CouchDB repo, pin their version
> inside the CouchDB repo, dont have commits to subproject X automatically be
> applied to CouchDB, that means you can commit whatever you want to X but
> CouchDB will still be working, when you come to update the version of X you
> pinned, you can see that it breaks and not update until it is fixed.

It's good strategy for milestones or when you don't have to manage 40+ repos (:

>   2.5 Another strategy that can help with this is to have the full CouchDB
> test suite run inside X, so changes to X will run the full CouchDB suite
> with an updated X

This works, but until changes are belongs to a single repository. See
above about travis issue.


P.S. Jason, you are right again (:

--
,,,^..^,,,

Re: [PROPOSAL] Improving the quality of CouchDB

[Jason Smith <ja...@gmail.com>]

On Thu, Sep 10, 2015 at 4:40 PM, Dale Harvey <da...@arandomurl.com> wrote:

> I do think multirepos is an issue and that solution is not so simple, we
> went through the same issues with pouchdb as we attempted to split out our
> repository.
>

If I understand correctly, this is what we do now...


>
> 1. Dont split out into lots of repositories, if you put those components
> inside the CouchDB repo, then they will get the CouchDB tests run against
> them when changes are made and you wont break the CouchDB repo.
>

...and that is what we used to do.


> 2. Anything that does live outside the CouchDB repo, pin their version
> inside the CouchDB repo, dont have commits to subproject X automatically be
> applied to CouchDB, that means you can commit whatever you want to X but
> CouchDB will still be working, when you come to update the version of X you
> pinned, you can see that it breaks and not update until it is fixed.
>

As I read this, I immediately thought of package systems. Perhaps we have a
problem where we have splintered the project into different parts, but
those parts are insufficiently managed, insufficiently stitched together.

Maybe it is something as simple as tying the software together through Git
tags, and those tags are semver releases.

In any case, I feel like we have used separate repos for long enough to get
a feel for it. It sounds like it is time to revisit the pros and cons of
this and to decide how to proceed.

Re: [PROPOSAL] Improving the quality of CouchDB

[Ilya Khlopotov <ii...@ca.ibm.com>]

One more option to consider (based on Russell idea expressed personally).
We could follow following process when we create PR.
1. create PR against dependency (or multiple deps)
2. create PR (this is not for merge) for couchdb repo which updates
rebar.config.script to point to PR created in #1
3. travis would be pointed to couchdb repo only and will build and test
whole project.

This approach would simplify the review job as well since he/she don't need
to manually checkout right versions of updated dependencies.

We can go further with this approach by converting couchdb repository into
meta repository which would only have meta information for release.
Here is what I'm talking about https://github.com/iilyak/couchdb-manifest

BR,
ILYA



From:	Dale Harvey <da...@arandomurl.com>
To:	dev@couchdb.apache.org
Date:	2015/09/10 02:40 AM
Subject:	Re: [PROPOSAL] Improving the quality of CouchDB



I dont think CI is a dream, It should take an afternoon at most to get
CouchDB setup on travis on a single platform to ensure no major regressions
come though. If anyone wants help doing that feel free to ping me on
#pouchdb / #couchdb, we already test couchdb master in travis.

I do think multirepos is an issue and that solution is not so simple, we
went through the same issues with pouchdb as we attempted to split out our
repository.

The basic problem is, X is a downstream component of CouchDB in its own
repo, X may have its own unit tests etc, however I can still make a change
to X, have it all passing but it can still complete break the CouchDB
suite. This also entirely break git bisect so not only are breakages more
likely but they are significantly harder to debug.

The 2 main solutions to that work to some degree for us

1. Dont split out into lots of repositories, if you put those components
inside the CouchDB repo, then they will get the CouchDB tests run against
them when changes are made and you wont break the CouchDB repo.

2. Anything that does live outside the CouchDB repo, pin their version
inside the CouchDB repo, dont have commits to subproject X automatically be
applied to CouchDB, that means you can commit whatever you want to X but
CouchDB will still be working, when you come to update the version of X you
pinned, you can see that it breaks and not update until it is fixed.

  2.5 Another strategy that can help with this is to have the full CouchDB
test suite run inside X, so changes to X will run the full CouchDB suite
with an updated X

Cheers
Dale


On 9 September 2015 at 19:39, Alexander Shorin <kx...@gmail.com> wrote:

> On Wed, Sep 9, 2015 at 8:13 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
> >> So, what is the immediate next step that you propose to fix or
improve?
> >> What is the most squeaky wheel, or the most modest gain that we can
> make?
> >
> > I think one of the biggest problems is that we don't have a working CI
> and
> > that PRs are getting merged without running it. The other problem is
that
> > we have a lot of untested changes going into master.
> >
> > For a step by step plan I suggest:
> >
> >  - Setup a simple CI that makes it easy to test before a merge, e.g.
like
> > this [1]. The CI can be really simple, just 1 Erlang version and one OS
-
> > but let's get started somewhere. Do you have other ideas how to solve
CI,
> > PRs and the multi-repo-approach?
>
> Who will setup and host this CI machine? I guess, we don't speak about
> CI within ASF infra here.
>
> Solution of multirepo PRs is simple: use GitHub API to subscript to PR
> comments and changes. With special comment author sets references to
> other PR's. Subscription to changes is required to catch PR updates.
> Special comment is required to prevent false start when CI will try to
> build PR without related ones. Also it should be able to catch new
> references over time e.g. because author add missed tests to another
> repository after CI green build.
>
> >  - With a working CI system for the multi-repo setup merging should
just
> > happen with a green CI. Flaky tests should get fixed or deleted.
>
> Need to define policy what to do if CI is red by a reason not related to
> PR.
>
> >  - We would require that every change needs tests if not covered
> elsewhere
> > already. This might include that untestable code has to be refactored
> > first, a problem many legacy applications face (btw a good book on this
> is
> > "Working Effectively with Legacy Code". This would automatically make
our
> > code more testable and increase test coverage over time in a
> > very efficient way. This would also allow us to add features while
slowly
> > refactoring the code.
>
> What tools will be used to make this requirement mandatory?
>
> And the final note:  what is due to date (approximately) for all of this?
>
>
> While CI is a dream, what we can do to tolerate this problem *now*?
> For instance we can:
> - Reviewer who said +1 must first run full tests locally and ensure they
> are ok;
> - Reviewer must ensure that tests to the related changes are included;
> and so on. Anything else?
>
> --
> ,,,^..^,,,
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Dale Harvey <da...@arandomurl.com>]

And if anyone wants to make a start on travis

https://github.com/pouchdb/pouchdb/blob/master/.travis.yml and
https://github.com/pouchdb/pouchdb/blob/master/bin/run-couch-master-on-travis.sh

Should get you most of the way there

On 10 September 2015 at 11:40, Dale Harvey <da...@arandomurl.com> wrote:

> I dont think CI is a dream, It should take an afternoon at most to get
> CouchDB setup on travis on a single platform to ensure no major regressions
> come though. If anyone wants help doing that feel free to ping me on
> #pouchdb / #couchdb, we already test couchdb master in travis.
>
> I do think multirepos is an issue and that solution is not so simple, we
> went through the same issues with pouchdb as we attempted to split out our
> repository.
>
> The basic problem is, X is a downstream component of CouchDB in its own
> repo, X may have its own unit tests etc, however I can still make a change
> to X, have it all passing but it can still complete break the CouchDB
> suite. This also entirely break git bisect so not only are breakages more
> likely but they are significantly harder to debug.
>
> The 2 main solutions to that work to some degree for us
>
> 1. Dont split out into lots of repositories, if you put those components
> inside the CouchDB repo, then they will get the CouchDB tests run against
> them when changes are made and you wont break the CouchDB repo.
>
> 2. Anything that does live outside the CouchDB repo, pin their version
> inside the CouchDB repo, dont have commits to subproject X automatically be
> applied to CouchDB, that means you can commit whatever you want to X but
> CouchDB will still be working, when you come to update the version of X you
> pinned, you can see that it breaks and not update until it is fixed.
>
>   2.5 Another strategy that can help with this is to have the full CouchDB
> test suite run inside X, so changes to X will run the full CouchDB suite
> with an updated X
>
> Cheers
> Dale
>
>
> On 9 September 2015 at 19:39, Alexander Shorin <kx...@gmail.com> wrote:
>
>> On Wed, Sep 9, 2015 at 8:13 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
>> >> So, what is the immediate next step that you propose to fix or improve?
>> >> What is the most squeaky wheel, or the most modest gain that we can
>> make?
>> >
>> > I think one of the biggest problems is that we don't have a working CI
>> and
>> > that PRs are getting merged without running it. The other problem is
>> that
>> > we have a lot of untested changes going into master.
>> >
>> > For a step by step plan I suggest:
>> >
>> >  - Setup a simple CI that makes it easy to test before a merge, e.g.
>> like
>> > this [1]. The CI can be really simple, just 1 Erlang version and one OS
>> -
>> > but let's get started somewhere. Do you have other ideas how to solve
>> CI,
>> > PRs and the multi-repo-approach?
>>
>> Who will setup and host this CI machine? I guess, we don't speak about
>> CI within ASF infra here.
>>
>> Solution of multirepo PRs is simple: use GitHub API to subscript to PR
>> comments and changes. With special comment author sets references to
>> other PR's. Subscription to changes is required to catch PR updates.
>> Special comment is required to prevent false start when CI will try to
>> build PR without related ones. Also it should be able to catch new
>> references over time e.g. because author add missed tests to another
>> repository after CI green build.
>>
>> >  - With a working CI system for the multi-repo setup merging should just
>> > happen with a green CI. Flaky tests should get fixed or deleted.
>>
>> Need to define policy what to do if CI is red by a reason not related to
>> PR.
>>
>> >  - We would require that every change needs tests if not covered
>> elsewhere
>> > already. This might include that untestable code has to be refactored
>> > first, a problem many legacy applications face (btw a good book on this
>> is
>> > "Working Effectively with Legacy Code". This would automatically make
>> our
>> > code more testable and increase test coverage over time in a
>> > very efficient way. This would also allow us to add features while
>> slowly
>> > refactoring the code.
>>
>> What tools will be used to make this requirement mandatory?
>>
>> And the final note:  what is due to date (approximately) for all of this?
>>
>>
>> While CI is a dream, what we can do to tolerate this problem *now*?
>> For instance we can:
>> - Reviewer who said +1 must first run full tests locally and ensure they
>> are ok;
>> - Reviewer must ensure that tests to the related changes are included;
>> and so on. Anything else?
>>
>> --
>> ,,,^..^,,,
>>
>
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Dale Harvey <da...@arandomurl.com>]

I dont think CI is a dream, It should take an afternoon at most to get
CouchDB setup on travis on a single platform to ensure no major regressions
come though. If anyone wants help doing that feel free to ping me on
#pouchdb / #couchdb, we already test couchdb master in travis.

I do think multirepos is an issue and that solution is not so simple, we
went through the same issues with pouchdb as we attempted to split out our
repository.

The basic problem is, X is a downstream component of CouchDB in its own
repo, X may have its own unit tests etc, however I can still make a change
to X, have it all passing but it can still complete break the CouchDB
suite. This also entirely break git bisect so not only are breakages more
likely but they are significantly harder to debug.

The 2 main solutions to that work to some degree for us

1. Dont split out into lots of repositories, if you put those components
inside the CouchDB repo, then they will get the CouchDB tests run against
them when changes are made and you wont break the CouchDB repo.

2. Anything that does live outside the CouchDB repo, pin their version
inside the CouchDB repo, dont have commits to subproject X automatically be
applied to CouchDB, that means you can commit whatever you want to X but
CouchDB will still be working, when you come to update the version of X you
pinned, you can see that it breaks and not update until it is fixed.

  2.5 Another strategy that can help with this is to have the full CouchDB
test suite run inside X, so changes to X will run the full CouchDB suite
with an updated X

Cheers
Dale


On 9 September 2015 at 19:39, Alexander Shorin <kx...@gmail.com> wrote:

> On Wed, Sep 9, 2015 at 8:13 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
> >> So, what is the immediate next step that you propose to fix or improve?
> >> What is the most squeaky wheel, or the most modest gain that we can
> make?
> >
> > I think one of the biggest problems is that we don't have a working CI
> and
> > that PRs are getting merged without running it. The other problem is that
> > we have a lot of untested changes going into master.
> >
> > For a step by step plan I suggest:
> >
> >  - Setup a simple CI that makes it easy to test before a merge, e.g. like
> > this [1]. The CI can be really simple, just 1 Erlang version and one OS -
> > but let's get started somewhere. Do you have other ideas how to solve CI,
> > PRs and the multi-repo-approach?
>
> Who will setup and host this CI machine? I guess, we don't speak about
> CI within ASF infra here.
>
> Solution of multirepo PRs is simple: use GitHub API to subscript to PR
> comments and changes. With special comment author sets references to
> other PR's. Subscription to changes is required to catch PR updates.
> Special comment is required to prevent false start when CI will try to
> build PR without related ones. Also it should be able to catch new
> references over time e.g. because author add missed tests to another
> repository after CI green build.
>
> >  - With a working CI system for the multi-repo setup merging should just
> > happen with a green CI. Flaky tests should get fixed or deleted.
>
> Need to define policy what to do if CI is red by a reason not related to
> PR.
>
> >  - We would require that every change needs tests if not covered
> elsewhere
> > already. This might include that untestable code has to be refactored
> > first, a problem many legacy applications face (btw a good book on this
> is
> > "Working Effectively with Legacy Code". This would automatically make our
> > code more testable and increase test coverage over time in a
> > very efficient way. This would also allow us to add features while slowly
> > refactoring the code.
>
> What tools will be used to make this requirement mandatory?
>
> And the final note:  what is due to date (approximately) for all of this?
>
>
> While CI is a dream, what we can do to tolerate this problem *now*?
> For instance we can:
> - Reviewer who said +1 must first run full tests locally and ensure they
> are ok;
> - Reviewer must ensure that tests to the related changes are included;
> and so on. Anything else?
>
> --
> ,,,^..^,,,
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Alexander Shorin <kx...@gmail.com>]

On Wed, Sep 9, 2015 at 8:13 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
>> So, what is the immediate next step that you propose to fix or improve?
>> What is the most squeaky wheel, or the most modest gain that we can make?
>
> I think one of the biggest problems is that we don't have a working CI and
> that PRs are getting merged without running it. The other problem is that
> we have a lot of untested changes going into master.
>
> For a step by step plan I suggest:
>
>  - Setup a simple CI that makes it easy to test before a merge, e.g. like
> this [1]. The CI can be really simple, just 1 Erlang version and one OS -
> but let's get started somewhere. Do you have other ideas how to solve CI,
> PRs and the multi-repo-approach?

Who will setup and host this CI machine? I guess, we don't speak about
CI within ASF infra here.

Solution of multirepo PRs is simple: use GitHub API to subscript to PR
comments and changes. With special comment author sets references to
other PR's. Subscription to changes is required to catch PR updates.
Special comment is required to prevent false start when CI will try to
build PR without related ones. Also it should be able to catch new
references over time e.g. because author add missed tests to another
repository after CI green build.

>  - With a working CI system for the multi-repo setup merging should just
> happen with a green CI. Flaky tests should get fixed or deleted.

Need to define policy what to do if CI is red by a reason not related to PR.

>  - We would require that every change needs tests if not covered elsewhere
> already. This might include that untestable code has to be refactored
> first, a problem many legacy applications face (btw a good book on this is
> "Working Effectively with Legacy Code". This would automatically make our
> code more testable and increase test coverage over time in a
> very efficient way. This would also allow us to add features while slowly
> refactoring the code.

What tools will be used to make this requirement mandatory?

And the final note:  what is due to date (approximately) for all of this?


While CI is a dream, what we can do to tolerate this problem *now*?
For instance we can:
- Reviewer who said +1 must first run full tests locally and ensure they are ok;
- Reviewer must ensure that tests to the related changes are included;
and so on. Anything else?

--
,,,^..^,,,

Re: [PROPOSAL] Improving the quality of CouchDB

[Robert Kowalski <ro...@kowalski.gd>]

Thanks for the many answers!

> It sounds to me like the primary thrust of your argument is about
continuous
> integration--somehow making it more easy and more transparent for
> developers (seasoned and novice) to see the effects of their change. Is
> that right?

Yes, exactly!

> So, what is the immediate next step that you propose to fix or improve?
> What is the most squeaky wheel, or the most modest gain that we can make?

I think one of the biggest problems is that we don't have a working CI and
that PRs are getting merged without running it. The other problem is that
we have a lot of untested changes going into master.

For a step by step plan I suggest:

 - Setup a simple CI that makes it easy to test before a merge, e.g. like
this [1]. The CI can be really simple, just 1 Erlang version and one OS -
but let's get started somewhere. Do you have other ideas how to solve CI,
PRs and the multi-repo-approach?

 - With a working CI system for the multi-repo setup merging should just
happen with a green CI. Flaky tests should get fixed or deleted.

 - We would require that every change needs tests if not covered elsewhere
already. This might include that untestable code has to be refactored
first, a problem many legacy applications face (btw a good book on this is
"Working Effectively with Legacy Code". This would automatically make our
code more testable and increase test coverage over time in a
very efficient way. This would also allow us to add features while slowly
refactoring the code.

- Bonus step: we could make our lifes a bit easier. The video in [2] shows
how we could build a pipeline that just shows a merge button if the
testsuite was green. On a click on the button the Jenkins would merge the
PR. Why: merging over multiple repos right now is a lot of manual work,
let's automate most of our boring, daily tasks.


@Sebastian: do you plan to submit the integration test change? is the work
somewhere available?

[1] https://cloudup.com/cOgxRPbt9aP
[2] https://cloudup.com/c0VJDIoYqmI

On Tue, Sep 8, 2015 at 11:38 AM, Jason Smith <ja...@gmail.com>
wrote:

> Hi, Robert. Well, that was very long and I read every word! Thank you for
> your thoughtful assessment and feedback!
>
> You make many points across several different areas of concern. It sounds
> to me like the primary thrust of your argument is about continuous
> integration--somehow making it more easy and more transparent for
> developers (seasoned and novice) to see the effects of their change. Is
> that right?
>
> So, what is the immediate next step that you propose to fix or improve?
> What is the most squeaky wheel, or the most modest gain that we can make?
>
> Thanks again!
>
> On Mon, Sep 7, 2015 at 9:49 PM, Robert Kowalski <ro...@kowalski.gd> wrote:
>
> > Hi list,
> >
> > I had my first real programming job in a Web Agency. We were building
> > mobile websites for our external customers. The development process
> > followed a strict waterfall model, after you finished your project it was
> > thrown over the fence to the QA folks, which tested the website manually.
> > At some point, when all found bugs were fixed, the product got released.
> > Sometimes a bugfix took several iterations of going back and forth
> between
> > QA and dev. Sometimes the QA folks forgot to test if the bug was really
> > fixed after the third iteration as humans make errors, especially when it
> > is a boring task.
> >
> > After a lifetime of 2 years, the customer usually wanted a redesign, and
> > the next developer throwed everything of the old website away, as not
> much
> > of it was reusable. When something broke during the life of such an
> > application/website, the company and developers noticed that by the
> > customer calling them.
> >
> > These days I think often about my time in my first job and the customers
> > calling the company regarding bugs. Right now the Erlang core of CouchDB
> > reminds me quite often how our customers called us when something broke.
> > For CouchDB 2 it is not the users that call us right now, but PouchDB and
> > Fauxton when their testsuites gets red or incredible slow.
> >
> > When I was starting to work on Fauxton we were in a somehow similar
> > situation, a merge usually had 2-3 fixes on the same or next day, because
> > something somehow broke. A proper review for a change took incredible
> much
> > time and every pull request needed a lot of iterations. The most evil
> > changes were the ones where you fixed a bug in one area, and a completely
> > different area broke. These bugs were so evil because they were
> surprising
> > and unexpected.
> >
> > After a real big issue we tried out that every change would need unit
> > and/or integration tests for several weeks.
> >
> > Writing unit tests was very hard these days, most of the code was not
> > designed to get tested afterwards. So most of the tests written in that
> > time were Selenium based integration tests. This was better than nothing
> > but they were slow. Nevertheless, they caught a lot of bugs, even before
> > code was sent to the review. When we made a switch to React and a
> different
> > architecture for our components and their communication, better and
> easier
> > unit/functional testing was one of the many design goals.
> >
> > Writing good Selenium tests is incredible hard, as you basically face the
> > same problems you have in the distributed systems world in the world of
> > modern UIs. State and data changes asynchronously and you never know when
> > the result of your request, which changes the UI is returned. It even
> gets
> > harder by putting this on top of an eventually consistent system :)
> >
> > Sometimes we also pair-programmed in a video call in front of a test
> where
> > coming up with a good test was the problem, as we also had to share and
> > gain knowledge in the team how to write tests. One mistake we made was of
> > not fixing flaky tests immediately and adding more and more flaky tests
> and
> > just restarting the CI for a few weeks. At some point our CI system just
> > collapsed and it took the team a long time to fix those, partly also
> > because tests were a new topic and it is hard to understand why Selenium
> > tests fail in an async JS application with a cluster beneath it which is
> > eventually consistent. Additionally it decreased the confidence in a red
> > tests signaling a bug.
> >
> > After fixing the testsuite in this early crisis we've got pretty stable
> > tests. Additionally reviews take much less time. Reviews got even faster
> > when we added a pre-check for our coding guidelines to the testsuite. If
> > you take a look at a day in the time when we began writing tests, we
> > sometimes lost an hour to write those tests, because it was hard. Looking
> > just at this short timeframe it felt like we were losing time. But if we
> > take a look at a longer timeframe we did not decrease in our speed to
> > deliver features for several reasons: less bugfixes needed after code got
> > on master, less tickets and communication because of less bugs, faster
> > reviews because many bugs are caught before the reviewer takes a look and
> > so on. Some people in the team still hate writing tests but also everyone
> > on the team realized and also mentions that we have no other options.
> >
> > Right now I am seeing the same pattern we observed some time ago in
> > Fauxton for CouchDB's Erlang core :
> >
> >  - not noticing something broke (if you break it, will you even notice?)
> >  - eventually a user in the broader sense notices something broke
> >  - a lot of patching to fix patches where stuff broke in another place
> >  - regressions
> >  - it takes time to review code properly because of a large api surface
> > and a lot of functionality
> >
> > I would like to propose that every PR needs unit, functional AND/OR
> > integration tests for CouchDB's Erlang core, too - given it is not
> covered
> > by an existing test that run. Maybe we could try the new approach for 3
> > months and then decide if we keep it.
> >
> > Most of CouchDB's bugs are getting noticed by interacting with the HTTP
> > API. A simple integration test that CouchDB compiles and the bug got
> fixed
> > / the feature works would be enough as a minimum.
> >
> > Let's take a look what we have for testing now:
> >
> > - Erlang unit tests
> > - Erlang functional tests
> > - integration tests written in Erlang, which use ibrowse [1]
> >
> > - integration tests in JavaScript which test the backdoor ports
> >
> > - Fauxton's Integration tests simulating a user and their browsers
> > - PouchDB's integration tests
> >
> > So there are a lot of different ways to write tests. One problem for
> > testing is our multi-repository approach. The multiple PRs over different
> > repos must somehow get united again. Here is how we could solve it as a
> > first step [2]. The testsuite is kicked off once all PRs are opened by
> > putting the branchnames for the repositories that should not be master
> into
> > the config fields.
> >
> > When the testsuite is green another button could appear, to automate the
> > merge of all repos involved int he change, but that would we an
> additional
> > feature [3].
> >
> > The reviewer and the submitter of the PR would be both responsible that a
> > merge is covered by tests and the testsuite is green.
> >
> > Important is that it is a first step and depending how creative we get it
> > can indefinitely improve, like with Fauxton where a lot changed in the
> very
> > short timeframe since we are requiring tests. The important step is that
> we
> > somehow start at some point, and everybody is welcome and invited (and
> > should feel empowered) to make the processes how we write software and
> > tests better.
> >
> > Flaky tests are a problem and should get fixed immediately, as an
> > unreliable testsuite does not make much sense (remember the crying wolf
> kid
> > story). We learned that quite painfully in Fauxton.
> >
> > The change will need some discipline and maybe also stepping temporarily
> > out of our comfort zone, but I am certain we will gain speed, improve our
> > daily workflows and improve CouchDB's Quality over the long term. I
> think a
> > 3 month timeframe to try the new approach would be a good time to find
> out
> > if we are improving or things get worse.
> >
> >
> > Best,
> > Robert :)
> >
> >
> > [1]
> >
> https://github.com/apache/couchdb-chttpd/blob/master/test/chttpd_db_test.erl#L67
> > [2] https://cloudup.com/cOgxRPbt9aP
> > [3] https://cloudup.com/c0VJDIoYqmI
> >
>

Re: [PROPOSAL] Improving the quality of CouchDB

[Jason Smith <ja...@gmail.com>]

Hi, Robert. Well, that was very long and I read every word! Thank you for
your thoughtful assessment and feedback!

You make many points across several different areas of concern. It sounds
to me like the primary thrust of your argument is about continuous
integration--somehow making it more easy and more transparent for
developers (seasoned and novice) to see the effects of their change. Is
that right?

So, what is the immediate next step that you propose to fix or improve?
What is the most squeaky wheel, or the most modest gain that we can make?

Thanks again!

On Mon, Sep 7, 2015 at 9:49 PM, Robert Kowalski <ro...@kowalski.gd> wrote:

> Hi list,
>
> I had my first real programming job in a Web Agency. We were building
> mobile websites for our external customers. The development process
> followed a strict waterfall model, after you finished your project it was
> thrown over the fence to the QA folks, which tested the website manually.
> At some point, when all found bugs were fixed, the product got released.
> Sometimes a bugfix took several iterations of going back and forth between
> QA and dev. Sometimes the QA folks forgot to test if the bug was really
> fixed after the third iteration as humans make errors, especially when it
> is a boring task.
>
> After a lifetime of 2 years, the customer usually wanted a redesign, and
> the next developer throwed everything of the old website away, as not much
> of it was reusable. When something broke during the life of such an
> application/website, the company and developers noticed that by the
> customer calling them.
>
> These days I think often about my time in my first job and the customers
> calling the company regarding bugs. Right now the Erlang core of CouchDB
> reminds me quite often how our customers called us when something broke.
> For CouchDB 2 it is not the users that call us right now, but PouchDB and
> Fauxton when their testsuites gets red or incredible slow.
>
> When I was starting to work on Fauxton we were in a somehow similar
> situation, a merge usually had 2-3 fixes on the same or next day, because
> something somehow broke. A proper review for a change took incredible much
> time and every pull request needed a lot of iterations. The most evil
> changes were the ones where you fixed a bug in one area, and a completely
> different area broke. These bugs were so evil because they were surprising
> and unexpected.
>
> After a real big issue we tried out that every change would need unit
> and/or integration tests for several weeks.
>
> Writing unit tests was very hard these days, most of the code was not
> designed to get tested afterwards. So most of the tests written in that
> time were Selenium based integration tests. This was better than nothing
> but they were slow. Nevertheless, they caught a lot of bugs, even before
> code was sent to the review. When we made a switch to React and a different
> architecture for our components and their communication, better and easier
> unit/functional testing was one of the many design goals.
>
> Writing good Selenium tests is incredible hard, as you basically face the
> same problems you have in the distributed systems world in the world of
> modern UIs. State and data changes asynchronously and you never know when
> the result of your request, which changes the UI is returned. It even gets
> harder by putting this on top of an eventually consistent system :)
>
> Sometimes we also pair-programmed in a video call in front of a test where
> coming up with a good test was the problem, as we also had to share and
> gain knowledge in the team how to write tests. One mistake we made was of
> not fixing flaky tests immediately and adding more and more flaky tests and
> just restarting the CI for a few weeks. At some point our CI system just
> collapsed and it took the team a long time to fix those, partly also
> because tests were a new topic and it is hard to understand why Selenium
> tests fail in an async JS application with a cluster beneath it which is
> eventually consistent. Additionally it decreased the confidence in a red
> tests signaling a bug.
>
> After fixing the testsuite in this early crisis we've got pretty stable
> tests. Additionally reviews take much less time. Reviews got even faster
> when we added a pre-check for our coding guidelines to the testsuite. If
> you take a look at a day in the time when we began writing tests, we
> sometimes lost an hour to write those tests, because it was hard. Looking
> just at this short timeframe it felt like we were losing time. But if we
> take a look at a longer timeframe we did not decrease in our speed to
> deliver features for several reasons: less bugfixes needed after code got
> on master, less tickets and communication because of less bugs, faster
> reviews because many bugs are caught before the reviewer takes a look and
> so on. Some people in the team still hate writing tests but also everyone
> on the team realized and also mentions that we have no other options.
>
> Right now I am seeing the same pattern we observed some time ago in
> Fauxton for CouchDB's Erlang core :
>
>  - not noticing something broke (if you break it, will you even notice?)
>  - eventually a user in the broader sense notices something broke
>  - a lot of patching to fix patches where stuff broke in another place
>  - regressions
>  - it takes time to review code properly because of a large api surface
> and a lot of functionality
>
> I would like to propose that every PR needs unit, functional AND/OR
> integration tests for CouchDB's Erlang core, too - given it is not covered
> by an existing test that run. Maybe we could try the new approach for 3
> months and then decide if we keep it.
>
> Most of CouchDB's bugs are getting noticed by interacting with the HTTP
> API. A simple integration test that CouchDB compiles and the bug got fixed
> / the feature works would be enough as a minimum.
>
> Let's take a look what we have for testing now:
>
> - Erlang unit tests
> - Erlang functional tests
> - integration tests written in Erlang, which use ibrowse [1]
>
> - integration tests in JavaScript which test the backdoor ports
>
> - Fauxton's Integration tests simulating a user and their browsers
> - PouchDB's integration tests
>
> So there are a lot of different ways to write tests. One problem for
> testing is our multi-repository approach. The multiple PRs over different
> repos must somehow get united again. Here is how we could solve it as a
> first step [2]. The testsuite is kicked off once all PRs are opened by
> putting the branchnames for the repositories that should not be master into
> the config fields.
>
> When the testsuite is green another button could appear, to automate the
> merge of all repos involved int he change, but that would we an additional
> feature [3].
>
> The reviewer and the submitter of the PR would be both responsible that a
> merge is covered by tests and the testsuite is green.
>
> Important is that it is a first step and depending how creative we get it
> can indefinitely improve, like with Fauxton where a lot changed in the very
> short timeframe since we are requiring tests. The important step is that we
> somehow start at some point, and everybody is welcome and invited (and
> should feel empowered) to make the processes how we write software and
> tests better.
>
> Flaky tests are a problem and should get fixed immediately, as an
> unreliable testsuite does not make much sense (remember the crying wolf kid
> story). We learned that quite painfully in Fauxton.
>
> The change will need some discipline and maybe also stepping temporarily
> out of our comfort zone, but I am certain we will gain speed, improve our
> daily workflows and improve CouchDB's Quality over the long term. I think a
> 3 month timeframe to try the new approach would be a good time to find out
> if we are improving or things get worse.
>
>
> Best,
> Robert :)
>
>
> [1]
> https://github.com/apache/couchdb-chttpd/blob/master/test/chttpd_db_test.erl#L67
> [2] https://cloudup.com/cOgxRPbt9aP
> [3] https://cloudup.com/c0VJDIoYqmI
>