CI for Brooklyn

[Andrew Kennedy <an...@cloudsoftcorp.com>]

Hi.

What should we do to set up continuous integration of Brooklyn, now that
we've moved to the ASF? I know there's a Jenkins cluster managed by Infra,
so we can get set up there, probably. What about pushing -SNAPSHOT Jar
files to Sonatype, as we used to do from our Jenkins server?

Andrew.
-- 
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;

Re: CI for Brooklyn

[Richard Downer <ri...@apache.org>]

Svet,

I've switched off the "abort the build if it's stuck" option - once
we've got a run of successful builds and know the typical time, I can
switch it back on again.

Richard.

On 27 September 2014 10:44, Svetoslav Neykov
<sv...@cloudsoftcorp.com> wrote:
> Richard,
>
> Could you look into extending the timeout period for the build. A lot of builds failed around the 20th minute due to this restriction.
>
> Svet.
>
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Friday, September 26, 2014 11:23 PM
> To: dev@brooklyn.incubator.apache.org
> Subject: Re: CI for Brooklyn
>
> On Thu, Sep 25, 2014 at 11:36 AM, Richard Downer <ri...@apache.org> wrote:
>> Andrew,
>>
>> Kicking this thread back into life. I notice that I don't have the
>> option to modify our Jenkins build job - I assume this is because I
>> don't have the necessary Jenkins privileges. Are you able to give this
>> privilege to all of the Brooklyn committers?
>>
>
> Any officer can give them the requisite karma. Though, concept of least-privilege suggests we should only grant to those who actually wish to exercise those privileges.
>
>> If I can get privileges, I can look into sending Jenkins build
>> failures to the list, as we discussed earlier in this thread.
>>
>
> Done - you now have karma for that. (login to builds.a.o with your LDAP creds)
>
> --David
>

RE: CI for Brooklyn

[Svetoslav Neykov <sv...@cloudsoftcorp.com>]

Richard,

Could you look into extending the timeout period for the build. A lot of builds failed around the 20th minute due to this restriction.

Svet.

-----Original Message-----
From: David Nalley [mailto:david@gnsa.us] 
Sent: Friday, September 26, 2014 11:23 PM
To: dev@brooklyn.incubator.apache.org
Subject: Re: CI for Brooklyn

On Thu, Sep 25, 2014 at 11:36 AM, Richard Downer <ri...@apache.org> wrote:
> Andrew,
>
> Kicking this thread back into life. I notice that I don't have the 
> option to modify our Jenkins build job - I assume this is because I 
> don't have the necessary Jenkins privileges. Are you able to give this 
> privilege to all of the Brooklyn committers?
>

Any officer can give them the requisite karma. Though, concept of least-privilege suggests we should only grant to those who actually wish to exercise those privileges.

> If I can get privileges, I can look into sending Jenkins build 
> failures to the list, as we discussed earlier in this thread.
>

Done - you now have karma for that. (login to builds.a.o with your LDAP creds)

--David

Re: CI for Brooklyn

[David Nalley <da...@gnsa.us>]

On Thu, Sep 25, 2014 at 11:36 AM, Richard Downer <ri...@apache.org> wrote:
> Andrew,
>
> Kicking this thread back into life. I notice that I don't have the
> option to modify our Jenkins build job - I assume this is because I
> don't have the necessary Jenkins privileges. Are you able to give this
> privilege to all of the Brooklyn committers?
>

Any officer can give them the requisite karma. Though, concept of
least-privilege suggests we should only grant to those who actually
wish to exercise those privileges.

> If I can get privileges, I can look into sending Jenkins build
> failures to the list, as we discussed earlier in this thread.
>

Done - you now have karma for that. (login to builds.a.o with your LDAP creds)

--David

Re: CI for Brooklyn

[Richard Downer <ri...@apache.org>]

Andrew,

Kicking this thread back into life. I notice that I don't have the
option to modify our Jenkins build job - I assume this is because I
don't have the necessary Jenkins privileges. Are you able to give this
privilege to all of the Brooklyn committers?

If I can get privileges, I can look into sending Jenkins build
failures to the list, as we discussed earlier in this thread.

Thanks
Richard.


On 1 July 2014 10:20, Aled Sage <al...@gmail.com> wrote:
> +1 to e-mailing failures to dev@brooklyn.
>
> +1 to Richard's observation - we should not run integration tests on shared
> infrastructure; let's continue to use the jenkins box that cloudsoft
> contribute.
>
> Andrew, have you looked at setting up buildhive to run against all pull
> requests on https://github.com/apache/incubator-brooklyn?
> I really miss that! PRs are sometimes getting merged without anyone checking
> if the tests still pass!
>
> Aled
>
>
>
> On 01/07/2014 09:59, Richard Downer wrote:
>>
>> On 30 June 2014 21:01, Andrew Kennedy <an...@cloudsoftcorp.com>
>> wrote:
>>>
>>> Any objections (or other ideas) about sending results of failing builds
>>> on
>>> Jenkins to the list?
>>
>> +1
>> It simply needs to be done... it's a pain because it generates noise,
>> but the alternative - that nobody ever looks at Jenkins - is worse.
>>
>> One other comment, I see you added a note to the Jenkins job saying
>> you were intending to run the integration tests as part of the job.
>> Since the integration tests can be starting processes, opening ports,
>> etc. (and a buggy entity or test could leave these behind), is it wise
>> to run this on shared infrastructure?
>>
>> Richard.
>
>

Re: CI for Brooklyn

[Aled Sage <al...@gmail.com>]

+1 to e-mailing failures to dev@brooklyn.

+1 to Richard's observation - we should not run integration tests on 
shared infrastructure; let's continue to use the jenkins box that 
cloudsoft contribute.

Andrew, have you looked at setting up buildhive to run against all pull 
requests on https://github.com/apache/incubator-brooklyn?
I really miss that! PRs are sometimes getting merged without anyone 
checking if the tests still pass!

Aled


On 01/07/2014 09:59, Richard Downer wrote:
> On 30 June 2014 21:01, Andrew Kennedy <an...@cloudsoftcorp.com> wrote:
>> Any objections (or other ideas) about sending results of failing builds on
>> Jenkins to the list?
> +1
> It simply needs to be done... it's a pain because it generates noise,
> but the alternative - that nobody ever looks at Jenkins - is worse.
>
> One other comment, I see you added a note to the Jenkins job saying
> you were intending to run the integration tests as part of the job.
> Since the integration tests can be starting processes, opening ports,
> etc. (and a buggy entity or test could leave these behind), is it wise
> to run this on shared infrastructure?
>
> Richard.

Re: CI for Brooklyn

[Richard Downer <ri...@apache.org>]

On 30 June 2014 21:01, Andrew Kennedy <an...@cloudsoftcorp.com> wrote:
> Any objections (or other ideas) about sending results of failing builds on
> Jenkins to the list?

+1
It simply needs to be done... it's a pain because it generates noise,
but the alternative - that nobody ever looks at Jenkins - is worse.

One other comment, I see you added a note to the Jenkins job saying
you were intending to run the integration tests as part of the job.
Since the integration tests can be starting processes, opening ports,
etc. (and a buggy entity or test could leave these behind), is it wise
to run this on shared infrastructure?

Richard.

Re: CI for Brooklyn

[Andrew Kennedy <an...@cloudsoftcorp.com>]

Any objections (or other ideas) about sending results of failing builds on
Jenkins to the list?

Andrew.

-- 
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;


On 30 June 2014 18:45, Andrew Kennedy <an...@cloudsoftcorp.com>
wrote:

> OK,
>
> I've added a basic Maven build of the master branch to the ASF Jenkins
> server, and placed it in a 'Brooklyn' view. You can view the results here:
>
> - https://builds.apache.org/job/incubator-brooklyn-master-build/
> - https://builds.apache.org/view/A-D/view/Brooklyn/
>
> Any new job that starts with `incubator-brooklyn-` will be included in the
> list of jobs in the Brooklyn view.
>
> I have also updated the parent POM file to inherit the latest ASF parent
> (version 14) POM file, which is a pre-requisite for the next step. Once I
> have set things up for the ASF Nexus, we can start pushing -SNAPSHOT build
> artifacts there.
>
> Andrew.
>
> --
> -- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;
>
>
> On 26 June 2014 19:38, Andrew Kennedy <an...@cloudsoftcorp.com>
> wrote:
>
>> I actually already have access, from Qpid, so I'll set something up and
>> report back.
>>
>> Andrew.
>> --
>> -- andrew kennedy ? cloudsoft & software engineer : @grkvlt ;
>> On 26 Jun 2014 17:22, "Richard Downer" <ri...@apache.org> wrote:
>>
>>> So it seems like a good place to start would be builds.a.o. According
>>> to the Jenkins page on the Apache wiki:
>>> https://wiki.apache.org/general/Jenkins
>>> ...someone who is a PMC chair can grant Jenkins access to committers.
>>> Do we have any PMC chairs amongst our mentors who could grant this to
>>> the Brooklyn committers, or should I take this to incubator-general?
>>>
>>> On 24 June 2014 03:23, David Nalley <da...@gnsa.us> wrote:
>>> > So it depends on who you advertise to? If you note that they are
>>> > snapshots, not releases, may eat babies, don't promote that as the way
>>> > to consume Brooklyn, you're probably ok. To the 'general user' of your
>>> > product, you would promote releases that have been voted on by the
>>> > PMC.
>>>
>>> This point is noted, and I also saw the mammoth thread on
>>> legal-general along those lines recently. To avoid controversy we
>>> wouldn't link to snapshot builds on the main parts of the website.
>>> Maybe going off-topic slightly but I think that a "developers" corner
>>> of the website may be called for, where working with bleeding-edge
>>> code, and the breakages that come with it, is expected.
>>>
>>> Richard.
>>>
>>
>

Re: CI for Brooklyn

[Andrew Kennedy <an...@cloudsoftcorp.com>]

OK,

I've added a basic Maven build of the master branch to the ASF Jenkins
server, and placed it in a 'Brooklyn' view. You can view the results here:

- https://builds.apache.org/job/incubator-brooklyn-master-build/
- https://builds.apache.org/view/A-D/view/Brooklyn/

Any new job that starts with `incubator-brooklyn-` will be included in the
list of jobs in the Brooklyn view.

I have also updated the parent POM file to inherit the latest ASF parent
(version 14) POM file, which is a pre-requisite for the next step. Once I
have set things up for the ASF Nexus, we can start pushing -SNAPSHOT build
artifacts there.

Andrew.

-- 
-- andrew kennedy ? engineer : http://cloudsoftcorp.com/developers/ ;


On 26 June 2014 19:38, Andrew Kennedy <an...@cloudsoftcorp.com>
wrote:

> I actually already have access, from Qpid, so I'll set something up and
> report back.
>
> Andrew.
> --
> -- andrew kennedy ? cloudsoft & software engineer : @grkvlt ;
> On 26 Jun 2014 17:22, "Richard Downer" <ri...@apache.org> wrote:
>
>> So it seems like a good place to start would be builds.a.o. According
>> to the Jenkins page on the Apache wiki:
>> https://wiki.apache.org/general/Jenkins
>> ...someone who is a PMC chair can grant Jenkins access to committers.
>> Do we have any PMC chairs amongst our mentors who could grant this to
>> the Brooklyn committers, or should I take this to incubator-general?
>>
>> On 24 June 2014 03:23, David Nalley <da...@gnsa.us> wrote:
>> > So it depends on who you advertise to? If you note that they are
>> > snapshots, not releases, may eat babies, don't promote that as the way
>> > to consume Brooklyn, you're probably ok. To the 'general user' of your
>> > product, you would promote releases that have been voted on by the
>> > PMC.
>>
>> This point is noted, and I also saw the mammoth thread on
>> legal-general along those lines recently. To avoid controversy we
>> wouldn't link to snapshot builds on the main parts of the website.
>> Maybe going off-topic slightly but I think that a "developers" corner
>> of the website may be called for, where working with bleeding-edge
>> code, and the breakages that come with it, is expected.
>>
>> Richard.
>>
>

Re: CI for Brooklyn

[Andrew Kennedy <an...@cloudsoftcorp.com>]

I actually already have access, from Qpid, so I'll set something up and
report back.

Andrew.
-- 
-- andrew kennedy ? cloudsoft & software engineer : @grkvlt ;
On 26 Jun 2014 17:22, "Richard Downer" <ri...@apache.org> wrote:

> So it seems like a good place to start would be builds.a.o. According
> to the Jenkins page on the Apache wiki:
> https://wiki.apache.org/general/Jenkins
> ...someone who is a PMC chair can grant Jenkins access to committers.
> Do we have any PMC chairs amongst our mentors who could grant this to
> the Brooklyn committers, or should I take this to incubator-general?
>
> On 24 June 2014 03:23, David Nalley <da...@gnsa.us> wrote:
> > So it depends on who you advertise to? If you note that they are
> > snapshots, not releases, may eat babies, don't promote that as the way
> > to consume Brooklyn, you're probably ok. To the 'general user' of your
> > product, you would promote releases that have been voted on by the
> > PMC.
>
> This point is noted, and I also saw the mammoth thread on
> legal-general along those lines recently. To avoid controversy we
> wouldn't link to snapshot builds on the main parts of the website.
> Maybe going off-topic slightly but I think that a "developers" corner
> of the website may be called for, where working with bleeding-edge
> code, and the breakages that come with it, is expected.
>
> Richard.
>

Re: CI for Brooklyn

[Richard Downer <ri...@apache.org>]

So it seems like a good place to start would be builds.a.o. According
to the Jenkins page on the Apache wiki:
https://wiki.apache.org/general/Jenkins
...someone who is a PMC chair can grant Jenkins access to committers.
Do we have any PMC chairs amongst our mentors who could grant this to
the Brooklyn committers, or should I take this to incubator-general?

On 24 June 2014 03:23, David Nalley <da...@gnsa.us> wrote:
> So it depends on who you advertise to? If you note that they are
> snapshots, not releases, may eat babies, don't promote that as the way
> to consume Brooklyn, you're probably ok. To the 'general user' of your
> product, you would promote releases that have been voted on by the
> PMC.

This point is noted, and I also saw the mammoth thread on
legal-general along those lines recently. To avoid controversy we
wouldn't link to snapshot builds on the main parts of the website.
Maybe going off-topic slightly but I think that a "developers" corner
of the website may be called for, where working with bleeding-edge
code, and the breakages that come with it, is expected.

Richard.

Re: CI for Brooklyn

[David Nalley <da...@gnsa.us>]

On Mon, Jun 23, 2014 at 6:25 PM, Alex Heneveld
<al...@cloudsoftcorp.com> wrote:
>
> builds.a.o pushing to repository.a.o synching to maven central sounds good
> to me!
>
> any issues doing this with snapshots?

You can push snapshots from jenkins, but not releases themselves.
(votes and all need to take place, etc)

> ideally i'd like to see snapshot
> artifacts from a healthy builds pushed as soon as they are available.
> that's CI!  :)

Indeed - but lets talk a bit lower.

> no requirement to keep them as they can be regenerated but i
> wondered if there is any guidance in this area.  (in particular i noticed
> most other projects don't seem to have recent snapshots here, i wondered if
> there was a reason!)
>

So it depends on who you advertise to? If you note that they are
snapshots, not releases, may eat babies, don't promote that as the way
to consume Brooklyn, you're probably ok. To the 'general user' of your
product, you would promote releases that have been voted on by the
PMC.


> as for cloudbees ... i find it useful but not reliable so i'd be inclined to
> keep it but non-canonical.  does that make sense?
>

yep.

Re: CI for Brooklyn

[Alex Heneveld <al...@cloudsoftcorp.com>]

builds.a.o pushing to repository.a.o synching to maven central sounds 
good to me!

any issues doing this with snapshots?  ideally i'd like to see snapshot 
artifacts from a healthy builds pushed as soon as they are available.  
that's CI!  :)  no requirement to keep them as they can be regenerated 
but i wondered if there is any guidance in this area.  (in particular i 
noticed most other projects don't seem to have recent snapshots here, i 
wondered if there was a reason!)

as for cloudbees ... i find it useful but not reliable so i'd be 
inclined to keep it but non-canonical.  does that make sense?

--alex


On 23/06/2014 20:51, David Nalley wrote:
> On Mon, Jun 23, 2014 at 2:04 PM, David Nalley <da...@gnsa.us> wrote:
>> On Mon, Jun 23, 2014 at 11:28 AM, Andrew Kennedy
>> <an...@cloudsoftcorp.com> wrote:
>>> Hi.
>>>
>>> What should we do to set up continuous integration of Brooklyn, now that
>>> we've moved to the ASF? I know there's a Jenkins cluster managed by Infra,
>>> so we can get set up there, probably.
>> Yes; builds.a.o
> Just to be clear, there are other options.
> Cloudbees, travis-ci, etc.
> There's also more than one CI platform at the ASF.
>
>>> What about pushing -SNAPSHOT Jar
>>> files to Sonatype, as we used to do from our Jenkins server?
>>>
>> You can push to the ASF's repository.a.o - which syncs with maven central.
>>
>> --David

Re: CI for Brooklyn

[David Nalley <da...@gnsa.us>]

On Mon, Jun 23, 2014 at 2:04 PM, David Nalley <da...@gnsa.us> wrote:
> On Mon, Jun 23, 2014 at 11:28 AM, Andrew Kennedy
> <an...@cloudsoftcorp.com> wrote:
>> Hi.
>>
>> What should we do to set up continuous integration of Brooklyn, now that
>> we've moved to the ASF? I know there's a Jenkins cluster managed by Infra,
>> so we can get set up there, probably.
>
> Yes; builds.a.o

Just to be clear, there are other options.
Cloudbees, travis-ci, etc.
There's also more than one CI platform at the ASF.

>
>> What about pushing -SNAPSHOT Jar
>> files to Sonatype, as we used to do from our Jenkins server?
>>
>
> You can push to the ASF's repository.a.o - which syncs with maven central.
>
> --David

Re: CI for Brooklyn

[David Nalley <da...@gnsa.us>]

On Mon, Jun 23, 2014 at 11:28 AM, Andrew Kennedy
<an...@cloudsoftcorp.com> wrote:
> Hi.
>
> What should we do to set up continuous integration of Brooklyn, now that
> we've moved to the ASF? I know there's a Jenkins cluster managed by Infra,
> so we can get set up there, probably.

Yes; builds.a.o

> What about pushing -SNAPSHOT Jar
> files to Sonatype, as we used to do from our Jenkins server?
>

You can push to the ASF's repository.a.o - which syncs with maven central.

--David