What to run/test during a CI build

[Bastian Krol <ba...@tu-dortmund.de>]

Hi folks,


 > I fixed one more issue that prevented us from building CouchDB from 
within a release tarball

this line from Jan in the release testing thread reminded me of another 
question I pondered for the CI topic.

Right now we clone/update the repo from git and then run "./configure && 
make all check dist" and that's it. I wonder if that is enough? 
Currently I am not doing anything with the binary or the tarball that is 
produced in this step.

My guess would be that the one following would yield a more valuable 
feedback:

* Start CouchDB via the binary that has been produced by make all and at 
least run some simple smoke tests against that instance -- or is 
something like that done during make all/make check?
* Or, run make dist first to only produce a tarball, then use that 
tarball to actually build the CouchDB binary - instead of building 
directly from the git checkout. (This would catch the kind of issues 
mentioned by Jan above, I suppose).
* Some combination of the above, for example, build the binary from a 
dist tarball, start it, run smoke tests.


What do you think?


Disclaimer: I didn't take the time yet to really check the Makefile (and 
also my make skills are close to non-existing), so if the Makefile 
already covers parts of this, my bad.


Kind regards, merry Christmas & happy holidays

    Bastian

Re: What to run/test during a CI build

[Jan Lehnardt <ja...@apache.org>]

> On 25 Dec 2015, at 23:02, Bastian Krol <ba...@tu-dortmund.de> wrote:
> 
>> So worth to extend this chain to:
>> 
>> "./configure && make all check dist && tar -x
>> apache-couchdb-2.0-some-hash.tar.gz && cd apache-couchdb-2.0-some-hash
>> && ./configure && make all check" - not real command, but with fixing
>> tar and cd part it will be close to.
> 
> Wouldn't it be enough then to only `make dist` in the git repo (without make all check) and leave the `make all check` part for the second make run, when running in the directory where we extracted the tarball? Otherwise we are duplicating the work of the all & check targets. Since we will be running on builds.a.o we should try to minimize build times when possible, I think.

That sounds like a good idea!

> 
>> If it's possible to test "make install" on CI into some fake root -
>> that would be also helpful.
> 
> make install should be possible whenever we run the build in a Docker container (so, for all Linux flavors). We'll have to see for other OSes. How would you verify that the make install was successful?


Fauxton has a set of smoke tests for a set up node/cluster: https://github.com/apache/couchdb-fauxton/blob/master/app/addons/verifyinstall/resources.js (adapted from Futon, which might be a bit easier to follow: https://github.com/apache/couchdb/blob/1.x.x/share/www/verify_install.html#L47). These should be easy to adapt, or extracted, so we can run them as part of CI.

Best
Jan
--

Re: What to run/test during a CI build

[Bastian Krol <ba...@tu-dortmund.de>]

> So worth to extend this chain to:
>
> "./configure && make all check dist && tar -x
> apache-couchdb-2.0-some-hash.tar.gz && cd apache-couchdb-2.0-some-hash
> && ./configure && make all check" - not real command, but with fixing
> tar and cd part it will be close to.

Wouldn't it be enough then to only `make dist` in the git repo (without 
make all check) and leave the `make all check` part for the second make 
run, when running in the directory where we extracted the tarball? 
Otherwise we are duplicating the work of the all & check targets. Since 
we will be running on builds.a.o we should try to minimize build times 
when possible, I think.

> If it's possible to test "make install" on CI into some fake root -
> that would be also helpful.

make install should be possible whenever we run the build in a Docker 
container (so, for all Linux flavors). We'll have to see for other OSes. 
How would you verify that the make install was successful?

Kind regards

   Bastian

Re: What to run/test during a CI build

[Alexander Shorin <kx...@gmail.com>]

Hi Bastian,

On Fri, Dec 25, 2015 at 1:37 AM, Bastian Krol
<ba...@tu-dortmund.de> wrote:
> Right now we clone/update the repo from git and then run "./configure &&
> make all check dist" and that's it. I wonder if that is enough? Currently I
> am not doing anything with the binary or the tarball that is produced in
> this step.

We found yesterday the case when "make all check dist" passes
successfully, but produced release is broken (because we leave some
bits which triggers git info lookup while we strip all the .git
everywhere). So worth to extend this chain to:

"./configure && make all check dist && tar -x
apache-couchdb-2.0-some-hash.tar.gz && cd apache-couchdb-2.0-some-hash
&& ./configure && make all check" - not real command, but with fixing
tar and cd part it will be close to.

If it's possible to test "make install" on CI into some fake root -
that would be also helpful.

--
,,,^..^,,,

Re: What to run/test during a CI build

[Jan Lehnardt <ja...@apache.org>]

Heya Bastian,

if they are still relevant, here are a few answers for the “Open Questions” in the README:


> There is no CentOS/RHEL there, shouldn't it be added?

Yes it should.


> Do we run a CouchDB build on all combinations on each commit? This would probably be too much for the ASF Infra build systems. Do we build them once a day? We need to find a good balance between early feedback and resource consumption here.


Eventually, I would like a build in all combinations on all commits and all PRs / branches (unless a branch opts out proactively). I’m sure we can drum up the computing resources required, when we need them.

For the time being, I’m happy with whatever in-between, as long as we have some CI :)


> Do we even want to build the master branch or some other branch/tag? I guess the master branch would be most interesting for now, but not entirely sure. Also, it might make sense to make the branch/tag parameterizable so we could also use this to create releases from a specific tag etc.

Yup, I’d love this. Ideally, we only ever vote on releases that come from a CI’d release channel and eventually declare one a stable release instead of handcrafting tarballs.


> What exactly do we do in each Jenkins build? Just build CouchDB? Also build docs? Start CouchDB? Run some test suite?

I think we covered this earlier in the thread.


> The build is currently triggered as the CMD in the Dockerfile via the script build-ci.sh. Is that okay? If we need more steps (beyond simply building CouchDB) we would need to add it to build-ci.sh.


Sounds okay to me.

Best
Jan
--



> On 24 Dec 2015, at 23:37, Bastian Krol <ba...@tu-dortmund.de> wrote:
> 
> Hi folks,
> 
> 
> > I fixed one more issue that prevented us from building CouchDB from within a release tarball
> 
> this line from Jan in the release testing thread reminded me of another question I pondered for the CI topic.
> 
> Right now we clone/update the repo from git and then run "./configure && make all check dist" and that's it. I wonder if that is enough? Currently I am not doing anything with the binary or the tarball that is produced in this step.
> 
> My guess would be that the one following would yield a more valuable feedback:
> 
> * Start CouchDB via the binary that has been produced by make all and at least run some simple smoke tests against that instance -- or is something like that done during make all/make check?
> * Or, run make dist first to only produce a tarball, then use that tarball to actually build the CouchDB binary - instead of building directly from the git checkout. (This would catch the kind of issues mentioned by Jan above, I suppose).
> * Some combination of the above, for example, build the binary from a dist tarball, start it, run smoke tests.
> 
> 
> What do you think?
> 
> 
> Disclaimer: I didn't take the time yet to really check the Makefile (and also my make skills are close to non-existing), so if the Makefile already covers parts of this, my bad.
> 
> 
> Kind regards, merry Christmas & happy holidays
> 
>   Bastian