A tale of two Dockers: couchdb-ci and couchdb-docker

[Joan Touzet <wo...@apache.org>]

Hey everyone,

I had a brief private exchange with Garren about Docker, and figured it
would be better if everyone could benefit from the email chain. So here
it is!

The couchdb-docker repository has a self-contained, ready-to-run CouchDB
configuration. The intent for this is to be used in development
scenarios, for instance when the Fauxton team is improving Fauxton and
needs a backing CouchDB against which to test everything. The details
are here:

    https://github.com/apache/couchdb-docker

Currently we use Docker a little differently for the Jenkins CI runs.
There is some useful overlap between the two approaches.

Current our CI setup is split across Travis and Jenkins. We use Travis
for (slightly faster) smoke tests across 4 varieties of Erlang, and one
variety of Linux (Ubuntu). The builds run "make check". Each of these
builds takes about 15 minutes, and they tend to run in parallel.

Jenkins, on the other hand, tests within custom-built Docker images
across 2 varieties of Erlang (right now the OS default, and 18.3) and
across multiple OS variants: Debian x1, Ubuntu x3, CentOS x2) and will
soon include macOS and FreeBSD builds. Builds on Jenkins are more
involved - we run 'make dist' to create a tarball, then run 'make check'
from that tarball. The intent is to simulate the full release process.
Builds take about 20 minutes as a result.

The Docker images are provisioned using Ansible to keep setup steps
consistent across different OSes. The details are here:

    https://github.com/apache/couchdb-ci

Images are pushed up to the couchdbdev/ namespace at Docker Cloud by
hand, since they don't change very often. I considered having these
under the apache/ Docker namespace but since they are more
dev-consumable rather than end-user consumable it didn't make much
sense. Garren, klaemo and I all have access to the couchdbdev/
organization.

Also, I've added the packaging tools to the images for Debian & Ubuntu,
with CentOS builds not far behind. Most developers won't need these,
either, but we definitely need them if we intend to build packages for
each check-in -- which is a direction I'd like to go.

Note that the Docker remote build stuff is transitioning to a paid
model, see https://docs.docker.com/docker-cloud/builds/automated-build/
and specifically the text "This hosted build service is free while it is
in Beta." (Docker Hub is being replaced by Docker Cloud.) We need to be
careful about leveraging a service that is about to disappear; we may
have to start building those Docker images and publishing them from
Apache Jenkins in the same way that I intend to start publishing daily
packages as well.

I'm very concerned about pushing forward with building anything
automatically (Docker images, snap packages, Debian/Ubuntu/CentOS
packages, etc.) until the test suite is fixed. I've been begging for
help on the mailing lists to no avail. I don't feel right with
suggesting anyone run anything "nightly" or "direct from master" if any
tests fail.  We need to eject the failing tests, or better yet, fix them
so they are deterministic. We can't keep limping along as a project with
a test suite that no one trusts.

Apache has a namespace at the Docker Cloud repository. Eventually, we'd
love to see the end-user Docker image show up at apache/couchdb. Infra
will be happy to help with this once we're ready to go with a semi-
official image.

P.S. There is actually a *third* docker, and that's the Dockerfile
contained within the actual couchdb repository itself. klaemo originally
intended this to be used for development work on CouchDB itself. My
proposal is to remove this Dockerfile in favour of the couchdb-ci
images.

Thanks for reading my little essay!

-Joan

Re: A tale of two Dockers: couchdb-ci and couchdb-docker

[Garren Smith <ga...@apache.org>]

We use a CouchDB docker container for testing PouchDB and Fauxton. So my
usage and requirements for docker instances are pretty narrow.
I just need a development image that is up to date with master. Joan
created this one https://hub.docker.com/r/couchdbdev/debian-8-dev/ based
off the couchdb-docker/dev Dockerfile.
This is perfect for our needs.

Cheers
Garren

On Fri, Apr 28, 2017 at 11:20 PM, Joan Touzet <wo...@apache.org> wrote:

> ----- Original Message -----
> > From: "Daniel Munch" <da...@gmail.com>
> >
> > Now for me it feels a little as if couchdb could go into the same
> > direction. While couchdb-ci has already all the necessary build and
> > test steps, those are sort of duplicated in couchdb-docker, only that
> > in the latter it's a little harder to create all sorts of
> > permutations
> > of os/erlang version. For me, ideally, couchdb-ci would be able to
> > create the end-user image just as another form of packaging. This
> > would also mean that couchdb-docker could disappear at some point.
> > With the build infrastructure already contained in couchdb-ci, the
> > Dockerfiles representing the actual "runtime images" will become much
> > simpler and may find their place in couchb-ci as well.
>
> Thanks Daniel, this is good food for thought. You should note that
> we actually have a third repo, couchdb-pkg, that is used for all the
> packaging content, just to make things confusing ;) Right now it includes
> the Debian/Ubuntu (.deb) control files and the snap content, but will
> soon include the CentOS packaging stuff as well.
>
> Indeed, if you look at couchdb-docker right now, you'll see that the
> image installs all of the build chain tooling, then builds CouchDB,
> then *uninstalls* the tools to shrink the resulting image. We may
> very well be able to chain that part onto the couchdb-ci images
> rather than duplicating the work. As for the other points you raise,
> I need to think about it more.
>
> -Joan
>

Re: A tale of two Dockers: couchdb-ci and couchdb-docker

[Joan Touzet <wo...@apache.org>]

----- Original Message -----
> From: "Daniel Munch" <da...@gmail.com>
> 
> Now for me it feels a little as if couchdb could go into the same
> direction. While couchdb-ci has already all the necessary build and
> test steps, those are sort of duplicated in couchdb-docker, only that
> in the latter it's a little harder to create all sorts of
> permutations
> of os/erlang version. For me, ideally, couchdb-ci would be able to
> create the end-user image just as another form of packaging. This
> would also mean that couchdb-docker could disappear at some point.
> With the build infrastructure already contained in couchdb-ci, the
> Dockerfiles representing the actual "runtime images" will become much
> simpler and may find their place in couchb-ci as well.

Thanks Daniel, this is good food for thought. You should note that
we actually have a third repo, couchdb-pkg, that is used for all the
packaging content, just to make things confusing ;) Right now it includes
the Debian/Ubuntu (.deb) control files and the snap content, but will
soon include the CentOS packaging stuff as well.

Indeed, if you look at couchdb-docker right now, you'll see that the
image installs all of the build chain tooling, then builds CouchDB,
then *uninstalls* the tools to shrink the resulting image. We may
very well be able to chain that part onto the couchdb-ci images
rather than duplicating the work. As for the other points you raise,
I need to think about it more.

-Joan

Re: A tale of two Dockers: couchdb-ci and couchdb-docker

[Daniel Munch <da...@gmail.com>]

Hi Joan, hi everybody,

thank you for this small tale! There are multiple points I'd like to
add some opinions from my side, but first and foremost thank you for
the effort of getting tests and packing automated again! It's not an
easy task and requires a good amount of discipline, and more
important, help from all the sides.

But let's get back to Docker. At my company we're using docker
extensively, and the "tale of two dockers" sort of became a standard
for us: It merely translates into having a Docker image which contains
all the build-tools necessary to build a project and run the tests and
the "runtime image", a minimal image containing only the artefacts
from the compilation step and the necessary libraries/runtimes needed
for running the project. Not only this helps in keeping runtime images
lean, but also the Dockerfiles itself are more comprehensible. Note
that using this approach Docker Hub was never an option for us since
it pushes you into the "do-everything-in-Dockerfile" way of doing.

Now for me it feels a little as if couchdb could go into the same
direction. While couchdb-ci has already all the necessary build and
test steps, those are sort of duplicated in couchdb-docker, only that
in the latter it's a little harder to create all sorts of permutations
of os/erlang version. For me, ideally, couchdb-ci would be able to
create the end-user image just as another form of packaging. This
would also mean that couchdb-docker could disappear at some point.
With the build infrastructure already contained in couchdb-ci, the
Dockerfiles representing the actual "runtime images" will become much
simpler and may find their place in couchb-ci as well.

Hope I could add some valuable information, thanks for reading and
thanks again Joan for the great work around the build and testing
automation!

On Fri, Apr 28, 2017 at 6:52 PM, Joan Touzet <wo...@apache.org> wrote:
> Hey everyone,
>
> I had a brief private exchange with Garren about Docker, and figured it
> would be better if everyone could benefit from the email chain. So here
> it is!
>
> The couchdb-docker repository has a self-contained, ready-to-run CouchDB
> configuration. The intent for this is to be used in development
> scenarios, for instance when the Fauxton team is improving Fauxton and
> needs a backing CouchDB against which to test everything. The details
> are here:
>
>     https://github.com/apache/couchdb-docker
>
> Currently we use Docker a little differently for the Jenkins CI runs.
> There is some useful overlap between the two approaches.
>
> Current our CI setup is split across Travis and Jenkins. We use Travis
> for (slightly faster) smoke tests across 4 varieties of Erlang, and one
> variety of Linux (Ubuntu). The builds run "make check". Each of these
> builds takes about 15 minutes, and they tend to run in parallel.
>
> Jenkins, on the other hand, tests within custom-built Docker images
> across 2 varieties of Erlang (right now the OS default, and 18.3) and
> across multiple OS variants: Debian x1, Ubuntu x3, CentOS x2) and will
> soon include macOS and FreeBSD builds. Builds on Jenkins are more
> involved - we run 'make dist' to create a tarball, then run 'make check'
> from that tarball. The intent is to simulate the full release process.
> Builds take about 20 minutes as a result.
>
> The Docker images are provisioned using Ansible to keep setup steps
> consistent across different OSes. The details are here:
>
>     https://github.com/apache/couchdb-ci
>
> Images are pushed up to the couchdbdev/ namespace at Docker Cloud by
> hand, since they don't change very often. I considered having these
> under the apache/ Docker namespace but since they are more
> dev-consumable rather than end-user consumable it didn't make much
> sense. Garren, klaemo and I all have access to the couchdbdev/
> organization.
>
> Also, I've added the packaging tools to the images for Debian & Ubuntu,
> with CentOS builds not far behind. Most developers won't need these,
> either, but we definitely need them if we intend to build packages for
> each check-in -- which is a direction I'd like to go.
>
> Note that the Docker remote build stuff is transitioning to a paid
> model, see https://docs.docker.com/docker-cloud/builds/automated-build/
> and specifically the text "This hosted build service is free while it is
> in Beta." (Docker Hub is being replaced by Docker Cloud.) We need to be
> careful about leveraging a service that is about to disappear; we may
> have to start building those Docker images and publishing them from
> Apache Jenkins in the same way that I intend to start publishing daily
> packages as well.
>
> I'm very concerned about pushing forward with building anything
> automatically (Docker images, snap packages, Debian/Ubuntu/CentOS
> packages, etc.) until the test suite is fixed. I've been begging for
> help on the mailing lists to no avail. I don't feel right with
> suggesting anyone run anything "nightly" or "direct from master" if any
> tests fail.  We need to eject the failing tests, or better yet, fix them
> so they are deterministic. We can't keep limping along as a project with
> a test suite that no one trusts.
>
> Apache has a namespace at the Docker Cloud repository. Eventually, we'd
> love to see the end-user Docker image show up at apache/couchdb. Infra
> will be happy to help with this once we're ready to go with a semi-
> official image.
>
> P.S. There is actually a *third* docker, and that's the Dockerfile
> contained within the actual couchdb repository itself. klaemo originally
> intended this to be used for development work on CouchDB itself. My
> proposal is to remove this Dockerfile in favour of the couchdb-ci
> images.
>
> Thanks for reading my little essay!
>
> -Joan