You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@brooklyn.apache.org by Thomas Bouron <th...@cloudsoftcorp.com> on 2017/11/14 15:43:30 UTC
[PROPOSAL] Using docker for jenkins builds
Hi Brooklyners!
Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by using
the RPM package. This is still on hold because we don't build RPM package
of SNAPSHOT versions and therefore, our vagrant install script won't work
for bleeding edge versions.
For the record, RPM are not built on SNAPSHOT because INFRA does not offer
centos/RHEL machines as Jenkins slaves.
But that is coming to an end. INFRA did install (recently-ish) docker on
all Jenkins slaves which means that a new world is opened to us! In the
past weeks, I have created a clone[2] of the `brooklyn-dist-master`[3] job
and configured it to use a custom docker image I created[4]. It is based on
`maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` tag).
As my test was successful, I decided to go a little further by creating 2
other tags for this image:
- `client, based on `maven:alpine` with the `go` binary
- `latest` which is the combination of the last 2.
My plan is to migrate all jenkins jobs to use docker. I'll donate the
`Dockerfile` to each git repo so that jenkins will be able to build and
publish them on docker hub instead of using my account/image. Once we have
everything dockerised, we can look at improving current builds, like using
jenkins pipelines rather than relying on upstream<->downstream projects'
relationships.
It also makes the integration/live tests rise from the ashes. This is the
perfect opportunity to fix and make then part of our CI.
WDYT? Even if it's obvious, the main advantage here is the portability of
the build environment. It also means that we can use any jenkins slaves,
potentially saving us a lot of time when waiting for a build executor.
Best.
[1] https://github.com/apache/brooklyn-dist/pull/105
[2]
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
[3] https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master/
[4] https://hub.docker.com/r/tbouron/brooklyn-build/
--
Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Thomas Bouron <th...@cloudsoftcorp.com>.
Hi Richard.
Indeed, this will work but I guess we cannot guarantee SLAs ;)
I think there is a bit of confusion because I mixed things, let me clarify.
There are two aspect to what I'm doing:
1. Build all Brooklyn projects into Docker
2. Replace the 2 jobs we have per project (to build master + PRs) by a
single pipeline per project that take care of all. This solution is really
elegant because everything is nicely organised + the configuration lives
within git (I don't think I need to explain the benefits of that)
"retest this please" is currently available for all Brooklyn PRs and its
loss is induced by #2, but has no link with #1. In fact #2 sits on top of
#1.
But you're right, I should ask infra to add this feature. Will open a JIRA
for it.
Best.
On Tue, 24 Jul 2018 at 10:12 Richard Downer <ri...@apache.org> wrote:
> Loss of "retest this please" isn't a deal breaker IMO. Replace it with
> "@rdowner retest this please" and it will have the same effect, if somewhat
> slower ;-)
>
> Did we have "retest this please" available before? If so then I'd hope
> there's a way to get it working with the Docker-based build, especially
> since the Docker path is the one recommended by Infra. If there's a loss of
> functionality there then hopefully we could persuade Infra to get this
> feature working again.
>
> Richard.
>
>
> On Tue, 24 Jul 2018 at 09:34, Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com>
> wrote:
>
> > Hi all.
> >
> > Just a quick update about the status of this work: all projects now build
> > within docker containers. It has been a bit painful to set up but it's
> all
> > good now.
> >
> > Also, the concept I mentioned in my previous email works like a charm.
> > However, I discovered yesterday a limitation: Brooklyn GitHub repos are
> > owned by Apache, therefore we don't have the permissions to add webhooks.
> > This means that the feature to re-trigger a build from a PR comment by
> > writing "retest this please" won't work. On top of that, it seems that
> even
> > the DSL command to set this up is missing from the Apache
> > Jenkins installation.
> >
> > Knowing this, are we happy to move anyway, or is it a deal breaker? WDYT?
> >
> > Best.
> >
> > On Tue, 1 May 2018 at 14:33 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com
> > >
> > wrote:
> >
> > > Hi all.
> > >
> > > Continuing my work on this topic, I just pushed a PR [1] in
> > > brooklyn-library that adds a Jenkinsfile. If you are not familiar with
> > what
> > > it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL.
> > That
> > > allows you to capture the configuration / best practices under version
> > > control so it's easier to iterate over and is repeatable. This
> particular
> > > Jenkinsfile implements the current configuration we have + a fix for
> > > permission issues we recently had (see INFRA ticket [2])
> > >
> > > To make this work, we still need to manually create the multibranch
> > > pipeline job, and pointing it toward the right GitHub repo. I'll do
> that
> > > once the PR is merge. But the bonus point of this approach is that it
> > > automatically handles PR builds, with very nice views so that is 2
> birds
> > > from one stone.
> > >
> > > Why not using a seed instead that would avoid doing any kind of manual
> > > config? That is a good question and the answer is: we are running on
> the
> > > apache infra, and I don't think we have the power of using Jenkins
> seed.
> > > Now if I'm wrong, it's fine as we can just add the seed on top of the
> > > pipelines.
> > >
> > > Anyway, I tested my Jenkinsfile locally with brooklyn-library so it
> > should
> > > work nicely. I was planning to test with this project first, validate
> the
> > > concept, then do the same for the remaining git submodules. All I need
> > now,
> > > is a review on [1] + merge :)
> > >
> > > Best.
> > >
> > > [1] https://github.com/apache/brooklyn-library/pull/153
> > > [2]
> > >
> >
> https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
> > >
> > > On Wed, 15 Nov 2017 at 16:42 Thomas Bouron <
> > > thomas.bouron@cloudsoftcorp.com> wrote:
> > >
> > >> Looks like we start to reach a consensus :)
> > >>
> > >> I duplicated all our jobs and configured them to use docker instead:
> > >> - brooklyn-server =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/
> > >> - brooklyn-library =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
> > >> - brooklyn-client =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/
> > >> - brooklyn-ui =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/
> > >> - brooklyn-dist =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
> > >> - brooklyn =>
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/
> > >>
> > >> I only "disabled" the other jobs by removing all triggers, just in
> case
> > >> what I did breaks everything then we can go back.
> > >>
> > >> While I was on the jenkins, I noticed some `incubator-brooklyn-*`
> jobs,
> > I
> > >> guess those can be deleted isn't it? Also, anyone know the purpose of
> > >> brooklyn-master-windows[1] ?
> > >>
> > >> ---
> > >> In terms of next steps, I'll monitor to see if the new jobs behave as
> > >> expected. If yes, I'll then move on to:
> > >> - migrate the PR's jobs to use docker
> > >> - add the `Dockerfile` within each git submodule
> > >> - leverage the Jenkins' pipeline feature by adding the config directly
> > in
> > >> each git submodule
> > >> - create new job to build and publish docs/website automatically
> > >>
> > >> Best
> > >>
> > >> [1]
> > >>
> >
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/
> > >>
> > >> On Wed, 15 Nov 2017 at 11:18 John McCabe <jo...@johnmccabe.net> wrote:
> > >>
> > >>> Huge +1 on consistency/portability.
> > >>>
> > >>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org>
> > >>> wrote:
> > >>>
> > >>> > +1 I echo what Richard says about consistency ... also makes
> > debugging
> > >>> > build failures easier
> > >>> >
> > >>> >
> > >>> >
> > >>> > On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org>
> > >>> wrote:
> > >>> >
> > >>> > > +1
> > >>> > >
> > >>> > > Having seen some of the conversations on the builds@apache.org
> > >>> list, it
> > >>> > > does seem like this is the way to go as far as getting the right
> > >>> build
> > >>> > > environment on the Apache Jenkins cluster. This is the best
> > solution
> > >>> for
> > >>> > > getting the `rpm` and `deb` built regularly.
> > >>> > >
> > >>> > > Build consistency is also a great driver. I can see this being
> > >>> useful for
> > >>> > > the release process too, as a release build would have an
> identical
> > >>> > > environment to that used to build the snapshots.
> > >>> > >
> > >>> > > Richard.
> > >>> > >
> > >>> > >
> > >>> > > On 14 November 2017 at 15:43, Thomas Bouron
> > >>> > <thomas.bouron@cloudsoftcorp.c
> > >>> > > om
> > >>> > > > wrote:
> > >>> > >
> > >>> > > > Hi Brooklyners!
> > >>> > > >
> > >>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant`
> > by
> > >>> > using
> > >>> > > > the RPM package. This is still on hold because we don't build
> RPM
> > >>> > package
> > >>> > > > of SNAPSHOT versions and therefore, our vagrant install script
> > >>> won't
> > >>> > work
> > >>> > > > for bleeding edge versions.
> > >>> > > > For the record, RPM are not built on SNAPSHOT because INFRA
> does
> > >>> not
> > >>> > > offer
> > >>> > > > centos/RHEL machines as Jenkins slaves.
> > >>> > > >
> > >>> > > > But that is coming to an end. INFRA did install (recently-ish)
> > >>> docker
> > >>> > on
> > >>> > > > all Jenkins slaves which means that a new world is opened to
> us!
> > >>> In the
> > >>> > > > past weeks, I have created a clone[2] of the
> > >>> `brooklyn-dist-master`[3]
> > >>> > > job
> > >>> > > > and configured it to use a custom docker image I created[4]. It
> > is
> > >>> > based
> > >>> > > on
> > >>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the
> `dist`
> > >>> tag).
> > >>> > > > As my test was successful, I decided to go a little further by
> > >>> > creating 2
> > >>> > > > other tags for this image:
> > >>> > > > - `client, based on `maven:alpine` with the `go` binary
> > >>> > > > - `latest` which is the combination of the last 2.
> > >>> > > >
> > >>> > > > My plan is to migrate all jenkins jobs to use docker. I'll
> donate
> > >>> the
> > >>> > > > `Dockerfile` to each git repo so that jenkins will be able to
> > >>> build and
> > >>> > > > publish them on docker hub instead of using my account/image.
> > Once
> > >>> we
> > >>> > > have
> > >>> > > > everything dockerised, we can look at improving current builds,
> > >>> like
> > >>> > > using
> > >>> > > > jenkins pipelines rather than relying on upstream<->downstream
> > >>> > projects'
> > >>> > > > relationships.
> > >>> > > >
> > >>> > > > It also makes the integration/live tests rise from the ashes.
> > This
> > >>> is
> > >>> > the
> > >>> > > > perfect opportunity to fix and make then part of our CI.
> > >>> > > >
> > >>> > > > WDYT? Even if it's obvious, the main advantage here is the
> > >>> portability
> > >>> > of
> > >>> > > > the build environment. It also means that we can use any
> jenkins
> > >>> > slaves,
> > >>> > > > potentially saving us a lot of time when waiting for a build
> > >>> executor.
> > >>> > > >
> > >>> > > > Best.
> > >>> > > >
> > >>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105
> > >>> > > > [2]
> > >>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/
> > >>> > > > brooklyn-dist-master-docker/
> > >>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> > >>> > > > brooklyn-dist-master/
> > >>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> > >>> > > > --
> > >>> > > >
> > >>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft
> Corporation
> > •
> > >>> > > > https://cloudsoft.io/
> > >>> > > > Github: https://github.com/tbouron
> > >>> > > > Twitter: https://twitter.com/eltibouron
> > >>> > > >
> > >>> > >
> > >>> >
> > >>>
> > >> --
> > >>
> > >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > >> https://cloudsoft.io/
> > >> Github: https://github.com/tbouron
> > >> Twitter: https://twitter.com/eltibouron
> > >>
> > > --
> > >
> > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > > https://cloudsoft.io/
> > > Github: https://github.com/tbouron
> > > Twitter: https://twitter.com/eltibouron
> > >
> > --
> >
> > Thomas Bouron
> > Senior Software Engineer
> >
> > *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
> >
> > GitHub: https://github.com/tbouron
> > Twitter: https://twitter.com/eltibouron
> >
> > Need a hand with AWS? Get a Free Consultation.
> >
>
--
Thomas Bouron
Senior Software Engineer
*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Need a hand with AWS? Get a Free Consultation.
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Richard Downer <ri...@apache.org>.
Loss of "retest this please" isn't a deal breaker IMO. Replace it with
"@rdowner retest this please" and it will have the same effect, if somewhat
slower ;-)
Did we have "retest this please" available before? If so then I'd hope
there's a way to get it working with the Docker-based build, especially
since the Docker path is the one recommended by Infra. If there's a loss of
functionality there then hopefully we could persuade Infra to get this
feature working again.
Richard.
On Tue, 24 Jul 2018 at 09:34, Thomas Bouron <th...@cloudsoftcorp.com>
wrote:
> Hi all.
>
> Just a quick update about the status of this work: all projects now build
> within docker containers. It has been a bit painful to set up but it's all
> good now.
>
> Also, the concept I mentioned in my previous email works like a charm.
> However, I discovered yesterday a limitation: Brooklyn GitHub repos are
> owned by Apache, therefore we don't have the permissions to add webhooks.
> This means that the feature to re-trigger a build from a PR comment by
> writing "retest this please" won't work. On top of that, it seems that even
> the DSL command to set this up is missing from the Apache
> Jenkins installation.
>
> Knowing this, are we happy to move anyway, or is it a deal breaker? WDYT?
>
> Best.
>
> On Tue, 1 May 2018 at 14:33 Thomas Bouron <thomas.bouron@cloudsoftcorp.com
> >
> wrote:
>
> > Hi all.
> >
> > Continuing my work on this topic, I just pushed a PR [1] in
> > brooklyn-library that adds a Jenkinsfile. If you are not familiar with
> what
> > it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL.
> That
> > allows you to capture the configuration / best practices under version
> > control so it's easier to iterate over and is repeatable. This particular
> > Jenkinsfile implements the current configuration we have + a fix for
> > permission issues we recently had (see INFRA ticket [2])
> >
> > To make this work, we still need to manually create the multibranch
> > pipeline job, and pointing it toward the right GitHub repo. I'll do that
> > once the PR is merge. But the bonus point of this approach is that it
> > automatically handles PR builds, with very nice views so that is 2 birds
> > from one stone.
> >
> > Why not using a seed instead that would avoid doing any kind of manual
> > config? That is a good question and the answer is: we are running on the
> > apache infra, and I don't think we have the power of using Jenkins seed.
> > Now if I'm wrong, it's fine as we can just add the seed on top of the
> > pipelines.
> >
> > Anyway, I tested my Jenkinsfile locally with brooklyn-library so it
> should
> > work nicely. I was planning to test with this project first, validate the
> > concept, then do the same for the remaining git submodules. All I need
> now,
> > is a review on [1] + merge :)
> >
> > Best.
> >
> > [1] https://github.com/apache/brooklyn-library/pull/153
> > [2]
> >
> https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
> >
> > On Wed, 15 Nov 2017 at 16:42 Thomas Bouron <
> > thomas.bouron@cloudsoftcorp.com> wrote:
> >
> >> Looks like we start to reach a consensus :)
> >>
> >> I duplicated all our jobs and configured them to use docker instead:
> >> - brooklyn-server =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/
> >> - brooklyn-library =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
> >> - brooklyn-client =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/
> >> - brooklyn-ui =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/
> >> - brooklyn-dist =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
> >> - brooklyn =>
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/
> >>
> >> I only "disabled" the other jobs by removing all triggers, just in case
> >> what I did breaks everything then we can go back.
> >>
> >> While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs,
> I
> >> guess those can be deleted isn't it? Also, anyone know the purpose of
> >> brooklyn-master-windows[1] ?
> >>
> >> ---
> >> In terms of next steps, I'll monitor to see if the new jobs behave as
> >> expected. If yes, I'll then move on to:
> >> - migrate the PR's jobs to use docker
> >> - add the `Dockerfile` within each git submodule
> >> - leverage the Jenkins' pipeline feature by adding the config directly
> in
> >> each git submodule
> >> - create new job to build and publish docs/website automatically
> >>
> >> Best
> >>
> >> [1]
> >>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/
> >>
> >> On Wed, 15 Nov 2017 at 11:18 John McCabe <jo...@johnmccabe.net> wrote:
> >>
> >>> Huge +1 on consistency/portability.
> >>>
> >>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org>
> >>> wrote:
> >>>
> >>> > +1 I echo what Richard says about consistency ... also makes
> debugging
> >>> > build failures easier
> >>> >
> >>> >
> >>> >
> >>> > On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org>
> >>> wrote:
> >>> >
> >>> > > +1
> >>> > >
> >>> > > Having seen some of the conversations on the builds@apache.org
> >>> list, it
> >>> > > does seem like this is the way to go as far as getting the right
> >>> build
> >>> > > environment on the Apache Jenkins cluster. This is the best
> solution
> >>> for
> >>> > > getting the `rpm` and `deb` built regularly.
> >>> > >
> >>> > > Build consistency is also a great driver. I can see this being
> >>> useful for
> >>> > > the release process too, as a release build would have an identical
> >>> > > environment to that used to build the snapshots.
> >>> > >
> >>> > > Richard.
> >>> > >
> >>> > >
> >>> > > On 14 November 2017 at 15:43, Thomas Bouron
> >>> > <thomas.bouron@cloudsoftcorp.c
> >>> > > om
> >>> > > > wrote:
> >>> > >
> >>> > > > Hi Brooklyners!
> >>> > > >
> >>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant`
> by
> >>> > using
> >>> > > > the RPM package. This is still on hold because we don't build RPM
> >>> > package
> >>> > > > of SNAPSHOT versions and therefore, our vagrant install script
> >>> won't
> >>> > work
> >>> > > > for bleeding edge versions.
> >>> > > > For the record, RPM are not built on SNAPSHOT because INFRA does
> >>> not
> >>> > > offer
> >>> > > > centos/RHEL machines as Jenkins slaves.
> >>> > > >
> >>> > > > But that is coming to an end. INFRA did install (recently-ish)
> >>> docker
> >>> > on
> >>> > > > all Jenkins slaves which means that a new world is opened to us!
> >>> In the
> >>> > > > past weeks, I have created a clone[2] of the
> >>> `brooklyn-dist-master`[3]
> >>> > > job
> >>> > > > and configured it to use a custom docker image I created[4]. It
> is
> >>> > based
> >>> > > on
> >>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist`
> >>> tag).
> >>> > > > As my test was successful, I decided to go a little further by
> >>> > creating 2
> >>> > > > other tags for this image:
> >>> > > > - `client, based on `maven:alpine` with the `go` binary
> >>> > > > - `latest` which is the combination of the last 2.
> >>> > > >
> >>> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate
> >>> the
> >>> > > > `Dockerfile` to each git repo so that jenkins will be able to
> >>> build and
> >>> > > > publish them on docker hub instead of using my account/image.
> Once
> >>> we
> >>> > > have
> >>> > > > everything dockerised, we can look at improving current builds,
> >>> like
> >>> > > using
> >>> > > > jenkins pipelines rather than relying on upstream<->downstream
> >>> > projects'
> >>> > > > relationships.
> >>> > > >
> >>> > > > It also makes the integration/live tests rise from the ashes.
> This
> >>> is
> >>> > the
> >>> > > > perfect opportunity to fix and make then part of our CI.
> >>> > > >
> >>> > > > WDYT? Even if it's obvious, the main advantage here is the
> >>> portability
> >>> > of
> >>> > > > the build environment. It also means that we can use any jenkins
> >>> > slaves,
> >>> > > > potentially saving us a lot of time when waiting for a build
> >>> executor.
> >>> > > >
> >>> > > > Best.
> >>> > > >
> >>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105
> >>> > > > [2]
> >>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/
> >>> > > > brooklyn-dist-master-docker/
> >>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> >>> > > > brooklyn-dist-master/
> >>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> >>> > > > --
> >>> > > >
> >>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation
> •
> >>> > > > https://cloudsoft.io/
> >>> > > > Github: https://github.com/tbouron
> >>> > > > Twitter: https://twitter.com/eltibouron
> >>> > > >
> >>> > >
> >>> >
> >>>
> >> --
> >>
> >> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> >> https://cloudsoft.io/
> >> Github: https://github.com/tbouron
> >> Twitter: https://twitter.com/eltibouron
> >>
> > --
> >
> > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > https://cloudsoft.io/
> > Github: https://github.com/tbouron
> > Twitter: https://twitter.com/eltibouron
> >
> --
>
> Thomas Bouron
> Senior Software Engineer
>
> *Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
>
> GitHub: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
> Need a hand with AWS? Get a Free Consultation.
>
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Thomas Bouron <th...@cloudsoftcorp.com>.
Hi all.
Just a quick update about the status of this work: all projects now build
within docker containers. It has been a bit painful to set up but it's all
good now.
Also, the concept I mentioned in my previous email works like a charm.
However, I discovered yesterday a limitation: Brooklyn GitHub repos are
owned by Apache, therefore we don't have the permissions to add webhooks.
This means that the feature to re-trigger a build from a PR comment by
writing "retest this please" won't work. On top of that, it seems that even
the DSL command to set this up is missing from the Apache
Jenkins installation.
Knowing this, are we happy to move anyway, or is it a deal breaker? WDYT?
Best.
On Tue, 1 May 2018 at 14:33 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:
> Hi all.
>
> Continuing my work on this topic, I just pushed a PR [1] in
> brooklyn-library that adds a Jenkinsfile. If you are not familiar with what
> it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL. That
> allows you to capture the configuration / best practices under version
> control so it's easier to iterate over and is repeatable. This particular
> Jenkinsfile implements the current configuration we have + a fix for
> permission issues we recently had (see INFRA ticket [2])
>
> To make this work, we still need to manually create the multibranch
> pipeline job, and pointing it toward the right GitHub repo. I'll do that
> once the PR is merge. But the bonus point of this approach is that it
> automatically handles PR builds, with very nice views so that is 2 birds
> from one stone.
>
> Why not using a seed instead that would avoid doing any kind of manual
> config? That is a good question and the answer is: we are running on the
> apache infra, and I don't think we have the power of using Jenkins seed.
> Now if I'm wrong, it's fine as we can just add the seed on top of the
> pipelines.
>
> Anyway, I tested my Jenkinsfile locally with brooklyn-library so it should
> work nicely. I was planning to test with this project first, validate the
> concept, then do the same for the remaining git submodules. All I need now,
> is a review on [1] + merge :)
>
> Best.
>
> [1] https://github.com/apache/brooklyn-library/pull/153
> [2]
> https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
>
> On Wed, 15 Nov 2017 at 16:42 Thomas Bouron <
> thomas.bouron@cloudsoftcorp.com> wrote:
>
>> Looks like we start to reach a consensus :)
>>
>> I duplicated all our jobs and configured them to use docker instead:
>> - brooklyn-server =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/
>> - brooklyn-library =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
>> - brooklyn-client =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/
>> - brooklyn-ui =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/
>> - brooklyn-dist =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
>> - brooklyn =>
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/
>>
>> I only "disabled" the other jobs by removing all triggers, just in case
>> what I did breaks everything then we can go back.
>>
>> While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs, I
>> guess those can be deleted isn't it? Also, anyone know the purpose of
>> brooklyn-master-windows[1] ?
>>
>> ---
>> In terms of next steps, I'll monitor to see if the new jobs behave as
>> expected. If yes, I'll then move on to:
>> - migrate the PR's jobs to use docker
>> - add the `Dockerfile` within each git submodule
>> - leverage the Jenkins' pipeline feature by adding the config directly in
>> each git submodule
>> - create new job to build and publish docs/website automatically
>>
>> Best
>>
>> [1]
>> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/
>>
>> On Wed, 15 Nov 2017 at 11:18 John McCabe <jo...@johnmccabe.net> wrote:
>>
>>> Huge +1 on consistency/portability.
>>>
>>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org>
>>> wrote:
>>>
>>> > +1 I echo what Richard says about consistency ... also makes debugging
>>> > build failures easier
>>> >
>>> >
>>> >
>>> > On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org>
>>> wrote:
>>> >
>>> > > +1
>>> > >
>>> > > Having seen some of the conversations on the builds@apache.org
>>> list, it
>>> > > does seem like this is the way to go as far as getting the right
>>> build
>>> > > environment on the Apache Jenkins cluster. This is the best solution
>>> for
>>> > > getting the `rpm` and `deb` built regularly.
>>> > >
>>> > > Build consistency is also a great driver. I can see this being
>>> useful for
>>> > > the release process too, as a release build would have an identical
>>> > > environment to that used to build the snapshots.
>>> > >
>>> > > Richard.
>>> > >
>>> > >
>>> > > On 14 November 2017 at 15:43, Thomas Bouron
>>> > <thomas.bouron@cloudsoftcorp.c
>>> > > om
>>> > > > wrote:
>>> > >
>>> > > > Hi Brooklyners!
>>> > > >
>>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by
>>> > using
>>> > > > the RPM package. This is still on hold because we don't build RPM
>>> > package
>>> > > > of SNAPSHOT versions and therefore, our vagrant install script
>>> won't
>>> > work
>>> > > > for bleeding edge versions.
>>> > > > For the record, RPM are not built on SNAPSHOT because INFRA does
>>> not
>>> > > offer
>>> > > > centos/RHEL machines as Jenkins slaves.
>>> > > >
>>> > > > But that is coming to an end. INFRA did install (recently-ish)
>>> docker
>>> > on
>>> > > > all Jenkins slaves which means that a new world is opened to us!
>>> In the
>>> > > > past weeks, I have created a clone[2] of the
>>> `brooklyn-dist-master`[3]
>>> > > job
>>> > > > and configured it to use a custom docker image I created[4]. It is
>>> > based
>>> > > on
>>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist`
>>> tag).
>>> > > > As my test was successful, I decided to go a little further by
>>> > creating 2
>>> > > > other tags for this image:
>>> > > > - `client, based on `maven:alpine` with the `go` binary
>>> > > > - `latest` which is the combination of the last 2.
>>> > > >
>>> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate
>>> the
>>> > > > `Dockerfile` to each git repo so that jenkins will be able to
>>> build and
>>> > > > publish them on docker hub instead of using my account/image. Once
>>> we
>>> > > have
>>> > > > everything dockerised, we can look at improving current builds,
>>> like
>>> > > using
>>> > > > jenkins pipelines rather than relying on upstream<->downstream
>>> > projects'
>>> > > > relationships.
>>> > > >
>>> > > > It also makes the integration/live tests rise from the ashes. This
>>> is
>>> > the
>>> > > > perfect opportunity to fix and make then part of our CI.
>>> > > >
>>> > > > WDYT? Even if it's obvious, the main advantage here is the
>>> portability
>>> > of
>>> > > > the build environment. It also means that we can use any jenkins
>>> > slaves,
>>> > > > potentially saving us a lot of time when waiting for a build
>>> executor.
>>> > > >
>>> > > > Best.
>>> > > >
>>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105
>>> > > > [2]
>>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/
>>> > > > brooklyn-dist-master-docker/
>>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
>>> > > > brooklyn-dist-master/
>>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
>>> > > > --
>>> > > >
>>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
>>> > > > https://cloudsoft.io/
>>> > > > Github: https://github.com/tbouron
>>> > > > Twitter: https://twitter.com/eltibouron
>>> > > >
>>> > >
>>> >
>>>
>> --
>>
>> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
>> https://cloudsoft.io/
>> Github: https://github.com/tbouron
>> Twitter: https://twitter.com/eltibouron
>>
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
--
Thomas Bouron
Senior Software Engineer
*Cloudsoft <https://cloudsoft.io/> *| Bringing Business to the Cloud
GitHub: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Need a hand with AWS? Get a Free Consultation.
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Thomas Bouron <th...@cloudsoftcorp.com>.
Hi all.
Continuing my work on this topic, I just pushed a PR [1] in
brooklyn-library that adds a Jenkinsfile. If you are not familiar with what
it is, a Jenkinfile describes a Jenkins job by using the pipeline DSL. That
allows you to capture the configuration / best practices under version
control so it's easier to iterate over and is repeatable. This particular
Jenkinsfile implements the current configuration we have + a fix for
permission issues we recently had (see INFRA ticket [2])
To make this work, we still need to manually create the multibranch
pipeline job, and pointing it toward the right GitHub repo. I'll do that
once the PR is merge. But the bonus point of this approach is that it
automatically handles PR builds, with very nice views so that is 2 birds
from one stone.
Why not using a seed instead that would avoid doing any kind of manual
config? That is a good question and the answer is: we are running on the
apache infra, and I don't think we have the power of using Jenkins seed.
Now if I'm wrong, it's fine as we can just add the seed on top of the
pipelines.
Anyway, I tested my Jenkinsfile locally with brooklyn-library so it should
work nicely. I was planning to test with this project first, validate the
concept, then do the same for the remaining git submodules. All I need now,
is a review on [1] + merge :)
Best.
[1] https://github.com/apache/brooklyn-library/pull/153
[2]
https://issues.apache.org/jira/browse/INFRA-16417?page=com.atlassian.jira.plugin.system.issuetabpanels%3Aall-tabpanel
On Wed, 15 Nov 2017 at 16:42 Thomas Bouron <th...@cloudsoftcorp.com>
wrote:
> Looks like we start to reach a consensus :)
>
> I duplicated all our jobs and configured them to use docker instead:
> - brooklyn-server =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/
> - brooklyn-library =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
> - brooklyn-client =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/
> - brooklyn-ui =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/
> - brooklyn-dist =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
> - brooklyn =>
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/
>
> I only "disabled" the other jobs by removing all triggers, just in case
> what I did breaks everything then we can go back.
>
> While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs, I
> guess those can be deleted isn't it? Also, anyone know the purpose of
> brooklyn-master-windows[1] ?
>
> ---
> In terms of next steps, I'll monitor to see if the new jobs behave as
> expected. If yes, I'll then move on to:
> - migrate the PR's jobs to use docker
> - add the `Dockerfile` within each git submodule
> - leverage the Jenkins' pipeline feature by adding the config directly in
> each git submodule
> - create new job to build and publish docs/website automatically
>
> Best
>
> [1]
> https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/
>
> On Wed, 15 Nov 2017 at 11:18 John McCabe <jo...@johnmccabe.net> wrote:
>
>> Huge +1 on consistency/portability.
>>
>> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org> wrote:
>>
>> > +1 I echo what Richard says about consistency ... also makes debugging
>> > build failures easier
>> >
>> >
>> >
>> > On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org>
>> wrote:
>> >
>> > > +1
>> > >
>> > > Having seen some of the conversations on the builds@apache.org list,
>> it
>> > > does seem like this is the way to go as far as getting the right build
>> > > environment on the Apache Jenkins cluster. This is the best solution
>> for
>> > > getting the `rpm` and `deb` built regularly.
>> > >
>> > > Build consistency is also a great driver. I can see this being useful
>> for
>> > > the release process too, as a release build would have an identical
>> > > environment to that used to build the snapshots.
>> > >
>> > > Richard.
>> > >
>> > >
>> > > On 14 November 2017 at 15:43, Thomas Bouron
>> > <thomas.bouron@cloudsoftcorp.c
>> > > om
>> > > > wrote:
>> > >
>> > > > Hi Brooklyners!
>> > > >
>> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by
>> > using
>> > > > the RPM package. This is still on hold because we don't build RPM
>> > package
>> > > > of SNAPSHOT versions and therefore, our vagrant install script won't
>> > work
>> > > > for bleeding edge versions.
>> > > > For the record, RPM are not built on SNAPSHOT because INFRA does not
>> > > offer
>> > > > centos/RHEL machines as Jenkins slaves.
>> > > >
>> > > > But that is coming to an end. INFRA did install (recently-ish)
>> docker
>> > on
>> > > > all Jenkins slaves which means that a new world is opened to us! In
>> the
>> > > > past weeks, I have created a clone[2] of the
>> `brooklyn-dist-master`[3]
>> > > job
>> > > > and configured it to use a custom docker image I created[4]. It is
>> > based
>> > > on
>> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist`
>> tag).
>> > > > As my test was successful, I decided to go a little further by
>> > creating 2
>> > > > other tags for this image:
>> > > > - `client, based on `maven:alpine` with the `go` binary
>> > > > - `latest` which is the combination of the last 2.
>> > > >
>> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate
>> the
>> > > > `Dockerfile` to each git repo so that jenkins will be able to build
>> and
>> > > > publish them on docker hub instead of using my account/image. Once
>> we
>> > > have
>> > > > everything dockerised, we can look at improving current builds, like
>> > > using
>> > > > jenkins pipelines rather than relying on upstream<->downstream
>> > projects'
>> > > > relationships.
>> > > >
>> > > > It also makes the integration/live tests rise from the ashes. This
>> is
>> > the
>> > > > perfect opportunity to fix and make then part of our CI.
>> > > >
>> > > > WDYT? Even if it's obvious, the main advantage here is the
>> portability
>> > of
>> > > > the build environment. It also means that we can use any jenkins
>> > slaves,
>> > > > potentially saving us a lot of time when waiting for a build
>> executor.
>> > > >
>> > > > Best.
>> > > >
>> > > > [1] https://github.com/apache/brooklyn-dist/pull/105
>> > > > [2]
>> > > > https://builds.apache.org/view/B/view/Brooklyn/job/
>> > > > brooklyn-dist-master-docker/
>> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
>> > > > brooklyn-dist-master/
>> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
>> > > > --
>> > > >
>> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
>> > > > https://cloudsoft.io/
>> > > > Github: https://github.com/tbouron
>> > > > Twitter: https://twitter.com/eltibouron
>> > > >
>> > >
>> >
>>
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>
--
Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Thomas Bouron <th...@cloudsoftcorp.com>.
Looks like we start to reach a consensus :)
I duplicated all our jobs and configured them to use docker instead:
- brooklyn-server =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-server-master-docker/
- brooklyn-library =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-library-master-docker/
- brooklyn-client =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-client-master-docker/
- brooklyn-ui =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-ui-master-docker/
- brooklyn-dist =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-dist-master-docker/
- brooklyn =>
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-build-docker/
I only "disabled" the other jobs by removing all triggers, just in case
what I did breaks everything then we can go back.
While I was on the jenkins, I noticed some `incubator-brooklyn-*` jobs, I
guess those can be deleted isn't it? Also, anyone know the purpose of
brooklyn-master-windows[1] ?
---
In terms of next steps, I'll monitor to see if the new jobs behave as
expected. If yes, I'll then move on to:
- migrate the PR's jobs to use docker
- add the `Dockerfile` within each git submodule
- leverage the Jenkins' pipeline feature by adding the config directly in
each git submodule
- create new job to build and publish docs/website automatically
Best
[1]
https://builds.apache.org/view/B/view/Brooklyn/job/brooklyn-master-windows/
On Wed, 15 Nov 2017 at 11:18 John McCabe <jo...@johnmccabe.net> wrote:
> Huge +1 on consistency/portability.
>
> On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org> wrote:
>
> > +1 I echo what Richard says about consistency ... also makes debugging
> > build failures easier
> >
> >
> >
> > On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org> wrote:
> >
> > > +1
> > >
> > > Having seen some of the conversations on the builds@apache.org list,
> it
> > > does seem like this is the way to go as far as getting the right build
> > > environment on the Apache Jenkins cluster. This is the best solution
> for
> > > getting the `rpm` and `deb` built regularly.
> > >
> > > Build consistency is also a great driver. I can see this being useful
> for
> > > the release process too, as a release build would have an identical
> > > environment to that used to build the snapshots.
> > >
> > > Richard.
> > >
> > >
> > > On 14 November 2017 at 15:43, Thomas Bouron
> > <thomas.bouron@cloudsoftcorp.c
> > > om
> > > > wrote:
> > >
> > > > Hi Brooklyners!
> > > >
> > > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by
> > using
> > > > the RPM package. This is still on hold because we don't build RPM
> > package
> > > > of SNAPSHOT versions and therefore, our vagrant install script won't
> > work
> > > > for bleeding edge versions.
> > > > For the record, RPM are not built on SNAPSHOT because INFRA does not
> > > offer
> > > > centos/RHEL machines as Jenkins slaves.
> > > >
> > > > But that is coming to an end. INFRA did install (recently-ish) docker
> > on
> > > > all Jenkins slaves which means that a new world is opened to us! In
> the
> > > > past weeks, I have created a clone[2] of the
> `brooklyn-dist-master`[3]
> > > job
> > > > and configured it to use a custom docker image I created[4]. It is
> > based
> > > on
> > > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist`
> tag).
> > > > As my test was successful, I decided to go a little further by
> > creating 2
> > > > other tags for this image:
> > > > - `client, based on `maven:alpine` with the `go` binary
> > > > - `latest` which is the combination of the last 2.
> > > >
> > > > My plan is to migrate all jenkins jobs to use docker. I'll donate the
> > > > `Dockerfile` to each git repo so that jenkins will be able to build
> and
> > > > publish them on docker hub instead of using my account/image. Once we
> > > have
> > > > everything dockerised, we can look at improving current builds, like
> > > using
> > > > jenkins pipelines rather than relying on upstream<->downstream
> > projects'
> > > > relationships.
> > > >
> > > > It also makes the integration/live tests rise from the ashes. This is
> > the
> > > > perfect opportunity to fix and make then part of our CI.
> > > >
> > > > WDYT? Even if it's obvious, the main advantage here is the
> portability
> > of
> > > > the build environment. It also means that we can use any jenkins
> > slaves,
> > > > potentially saving us a lot of time when waiting for a build
> executor.
> > > >
> > > > Best.
> > > >
> > > > [1] https://github.com/apache/brooklyn-dist/pull/105
> > > > [2]
> > > > https://builds.apache.org/view/B/view/Brooklyn/job/
> > > > brooklyn-dist-master-docker/
> > > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> > > > brooklyn-dist-master/
> > > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> > > > --
> > > >
> > > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > > > https://cloudsoft.io/
> > > > Github: https://github.com/tbouron
> > > > Twitter: https://twitter.com/eltibouron
> > > >
> > >
> >
>
--
Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
https://cloudsoft.io/
Github: https://github.com/tbouron
Twitter: https://twitter.com/eltibouron
Re: [PROPOSAL] Using docker for jenkins builds
Posted by John McCabe <jo...@johnmccabe.net>.
Huge +1 on consistency/portability.
On Tue, 14 Nov 2017 at 16:40 Mark McKenna <m4...@apache.org> wrote:
> +1 I echo what Richard says about consistency ... also makes debugging
> build failures easier
>
>
>
> On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org> wrote:
>
> > +1
> >
> > Having seen some of the conversations on the builds@apache.org list, it
> > does seem like this is the way to go as far as getting the right build
> > environment on the Apache Jenkins cluster. This is the best solution for
> > getting the `rpm` and `deb` built regularly.
> >
> > Build consistency is also a great driver. I can see this being useful for
> > the release process too, as a release build would have an identical
> > environment to that used to build the snapshots.
> >
> > Richard.
> >
> >
> > On 14 November 2017 at 15:43, Thomas Bouron
> <thomas.bouron@cloudsoftcorp.c
> > om
> > > wrote:
> >
> > > Hi Brooklyners!
> > >
> > > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by
> using
> > > the RPM package. This is still on hold because we don't build RPM
> package
> > > of SNAPSHOT versions and therefore, our vagrant install script won't
> work
> > > for bleeding edge versions.
> > > For the record, RPM are not built on SNAPSHOT because INFRA does not
> > offer
> > > centos/RHEL machines as Jenkins slaves.
> > >
> > > But that is coming to an end. INFRA did install (recently-ish) docker
> on
> > > all Jenkins slaves which means that a new world is opened to us! In the
> > > past weeks, I have created a clone[2] of the `brooklyn-dist-master`[3]
> > job
> > > and configured it to use a custom docker image I created[4]. It is
> based
> > on
> > > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` tag).
> > > As my test was successful, I decided to go a little further by
> creating 2
> > > other tags for this image:
> > > - `client, based on `maven:alpine` with the `go` binary
> > > - `latest` which is the combination of the last 2.
> > >
> > > My plan is to migrate all jenkins jobs to use docker. I'll donate the
> > > `Dockerfile` to each git repo so that jenkins will be able to build and
> > > publish them on docker hub instead of using my account/image. Once we
> > have
> > > everything dockerised, we can look at improving current builds, like
> > using
> > > jenkins pipelines rather than relying on upstream<->downstream
> projects'
> > > relationships.
> > >
> > > It also makes the integration/live tests rise from the ashes. This is
> the
> > > perfect opportunity to fix and make then part of our CI.
> > >
> > > WDYT? Even if it's obvious, the main advantage here is the portability
> of
> > > the build environment. It also means that we can use any jenkins
> slaves,
> > > potentially saving us a lot of time when waiting for a build executor.
> > >
> > > Best.
> > >
> > > [1] https://github.com/apache/brooklyn-dist/pull/105
> > > [2]
> > > https://builds.apache.org/view/B/view/Brooklyn/job/
> > > brooklyn-dist-master-docker/
> > > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> > > brooklyn-dist-master/
> > > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> > > --
> > >
> > > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > > https://cloudsoft.io/
> > > Github: https://github.com/tbouron
> > > Twitter: https://twitter.com/eltibouron
> > >
> >
>
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Mark McKenna <m4...@apache.org>.
+1 I echo what Richard says about consistency ... also makes debugging
build failures easier
On 14 November 2017 at 15:48, Richard Downer <ri...@apache.org> wrote:
> +1
>
> Having seen some of the conversations on the builds@apache.org list, it
> does seem like this is the way to go as far as getting the right build
> environment on the Apache Jenkins cluster. This is the best solution for
> getting the `rpm` and `deb` built regularly.
>
> Build consistency is also a great driver. I can see this being useful for
> the release process too, as a release build would have an identical
> environment to that used to build the snapshots.
>
> Richard.
>
>
> On 14 November 2017 at 15:43, Thomas Bouron <thomas.bouron@cloudsoftcorp.c
> om
> > wrote:
>
> > Hi Brooklyners!
> >
> > Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by using
> > the RPM package. This is still on hold because we don't build RPM package
> > of SNAPSHOT versions and therefore, our vagrant install script won't work
> > for bleeding edge versions.
> > For the record, RPM are not built on SNAPSHOT because INFRA does not
> offer
> > centos/RHEL machines as Jenkins slaves.
> >
> > But that is coming to an end. INFRA did install (recently-ish) docker on
> > all Jenkins slaves which means that a new world is opened to us! In the
> > past weeks, I have created a clone[2] of the `brooklyn-dist-master`[3]
> job
> > and configured it to use a custom docker image I created[4]. It is based
> on
> > `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` tag).
> > As my test was successful, I decided to go a little further by creating 2
> > other tags for this image:
> > - `client, based on `maven:alpine` with the `go` binary
> > - `latest` which is the combination of the last 2.
> >
> > My plan is to migrate all jenkins jobs to use docker. I'll donate the
> > `Dockerfile` to each git repo so that jenkins will be able to build and
> > publish them on docker hub instead of using my account/image. Once we
> have
> > everything dockerised, we can look at improving current builds, like
> using
> > jenkins pipelines rather than relying on upstream<->downstream projects'
> > relationships.
> >
> > It also makes the integration/live tests rise from the ashes. This is the
> > perfect opportunity to fix and make then part of our CI.
> >
> > WDYT? Even if it's obvious, the main advantage here is the portability of
> > the build environment. It also means that we can use any jenkins slaves,
> > potentially saving us a lot of time when waiting for a build executor.
> >
> > Best.
> >
> > [1] https://github.com/apache/brooklyn-dist/pull/105
> > [2]
> > https://builds.apache.org/view/B/view/Brooklyn/job/
> > brooklyn-dist-master-docker/
> > [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> > brooklyn-dist-master/
> > [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> > --
> >
> > Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> > https://cloudsoft.io/
> > Github: https://github.com/tbouron
> > Twitter: https://twitter.com/eltibouron
> >
>
Re: [PROPOSAL] Using docker for jenkins builds
Posted by Richard Downer <ri...@apache.org>.
+1
Having seen some of the conversations on the builds@apache.org list, it
does seem like this is the way to go as far as getting the right build
environment on the Apache Jenkins cluster. This is the best solution for
getting the `rpm` and `deb` built regularly.
Build consistency is also a great driver. I can see this being useful for
the release process too, as a release build would have an identical
environment to that used to build the snapshots.
Richard.
On 14 November 2017 at 15:43, Thomas Bouron <thomas.bouron@cloudsoftcorp.com
> wrote:
> Hi Brooklyners!
>
> Few months ago, I pushed a PR[1] to simplify `brooklyn-vagrant` by using
> the RPM package. This is still on hold because we don't build RPM package
> of SNAPSHOT versions and therefore, our vagrant install script won't work
> for bleeding edge versions.
> For the record, RPM are not built on SNAPSHOT because INFRA does not offer
> centos/RHEL machines as Jenkins slaves.
>
> But that is coming to an end. INFRA did install (recently-ish) docker on
> all Jenkins slaves which means that a new world is opened to us! In the
> past weeks, I have created a clone[2] of the `brooklyn-dist-master`[3] job
> and configured it to use a custom docker image I created[4]. It is based on
> `maven:alpine` and adds `rpm` and `dpkg` binaries (for the `dist` tag).
> As my test was successful, I decided to go a little further by creating 2
> other tags for this image:
> - `client, based on `maven:alpine` with the `go` binary
> - `latest` which is the combination of the last 2.
>
> My plan is to migrate all jenkins jobs to use docker. I'll donate the
> `Dockerfile` to each git repo so that jenkins will be able to build and
> publish them on docker hub instead of using my account/image. Once we have
> everything dockerised, we can look at improving current builds, like using
> jenkins pipelines rather than relying on upstream<->downstream projects'
> relationships.
>
> It also makes the integration/live tests rise from the ashes. This is the
> perfect opportunity to fix and make then part of our CI.
>
> WDYT? Even if it's obvious, the main advantage here is the portability of
> the build environment. It also means that we can use any jenkins slaves,
> potentially saving us a lot of time when waiting for a build executor.
>
> Best.
>
> [1] https://github.com/apache/brooklyn-dist/pull/105
> [2]
> https://builds.apache.org/view/B/view/Brooklyn/job/
> brooklyn-dist-master-docker/
> [3] https://builds.apache.org/view/B/view/Brooklyn/job/
> brooklyn-dist-master/
> [4] https://hub.docker.com/r/tbouron/brooklyn-build/
> --
>
> Thomas Bouron • Senior Software Engineer @ Cloudsoft Corporation •
> https://cloudsoft.io/
> Github: https://github.com/tbouron
> Twitter: https://twitter.com/eltibouron
>