You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@crunch.apache.org by Matthias Friedrich <ma...@mafr.de> on 2012/07/29 10:02:29 UTC
Continuous Integration: Next Steps
Hi,
over the last three weeks we've been working hard on making Crunch
ready for CI. It turned out to be much more work than initially
expected and I cursed myself more than once for starting this. In the
end it took four people and many patches to pull this off (thanks a
lot for your help, Josh, Rahul, and Gabriel!). In case you're
interested, these were the issues that lead us to CI readiness:
* CRUNCH-8: Refactor Crunch's build into a multi-module Maven project
* CRUNCH-17: Split unit tests from integration tests
* CRUNCH-18: Improve build stability
* CRUNCH-21: Make temporary directory configurable
* CRUNCH-24: Make test suite suiteable for continuous integration
* CRUNCH-26: Make Scrunch tests suiteable for continuous integration
We can now start with basic jobs to detect broken commits by running
the unit and integration test suites. I suggest starting with an
on-commit build using "mvn package" and a nightly build using "mvn
verify" (which runs the integration test suite). Running the
integration tests on-commit would be great but may be too expensive.
To get started we need access to Apache's Jenkins instance [1] which
apparently only PMC chairs can grant [2]. I'm new to Apache processes,
but perhaps a mentor can help us out here?
Next steps:
CI gives us the ability to regularly run reports to check code
quality. We already started preparations for this (Maven site setup,
checkstyle config and basic reports), but I think the best thing would
be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
bundles all kinds of code metrics in a nice interface and points out
where refactoring time is best spent. It's the most convenient way of
tracking code quality over time and catches many issues human
reviewers usually overlook.
Additional to reports, it's also quite common to deploy nightly
SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
sites. I'm not sure if this is possible at Apache for legal reasons
though.
Please let me know where you think we should go next. I know this
whole CI episode has been a nuisance, so thanks again for your help.
Regards,
Matthias
[1] https://builds.apache.org
[2] http://wiki.apache.org/general/Jenkins
[3] https://analysis.apache.org
Re: Continuous Integration: Next Steps
Posted by Matthias Friedrich <ma...@mafr.de>.
Works, thank you!
Right now there's not a lot I can do though, all build slots are
either blocked or the machines are down. I'll set things up as soon
as the service has stabilized.
Regards,
Matthias
On Sunday, 2012-08-05, Brock Noland wrote:
> done!
>
> brock@minotaur:~$ list_appgroups.pl --user=mafr
> hudson-jobadmin
>
>
>
> On Sun, Aug 5, 2012 at 7:42 PM, Josh Wills <jw...@cloudera.com> wrote:
>
> > Ya know, Matthias understands this stuff. Brock, can you give Matthias
> > (mafr) admin access to Jenkins?
> >
> > J
> >
> > On Sun, Aug 5, 2012 at 5:23 PM, Josh Wills <jw...@cloudera.com> wrote:
> >
> >> Thanks Brock. I edited it to become incubator-crunch, and it's running
> >> now: https://builds.apache.org/job/incubator-crunch/
> >>
> >> My Jenkins knowledge is also near-zero, so if anyone has any settings
> >> they would like me to enable, please let me know.
> >>
> >> On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <br...@cloudera.com> wrote:
> >>
> >>> Hmm. My experience with Jenkins is nearly zero. I just created
> >>> "crunch-trunk" job, can you edit it?
> >>>
> >>> https://builds.apache.org/job/crunch-trunk/configure
> >>>
> >>>
> >>>
> >>> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <jo...@gmail.com> wrote:
> >>>
> >>> > I get "jwills is missing the Create permission"
> >>> >
> >>> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com>
> >>> wrote:
> >>> > > According to ldap, we are both in the "hudson-jobadmin" group. There
> >>> is
> >>> > no
> >>> > > "New Job" link in the upper left? The link goes here:
> >>> > > https://builds.apache.org/view/All/newJob
> >>> > >
> >>> > > Brock
> >>> > >
> >>> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com>
> >>> wrote:
> >>> > >
> >>> > >> Hey Brock,
> >>> > >>
> >>> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
> >>> > >> that I can create a new job-- I just have a dashboard that lists
> >>> Build
> >>> > >> History/Project Relationship/etc. Is there some admin bit I'm
> >>> missing?
> >>> > >>
> >>> > >> J
> >>> > >>
> >>> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com>
> >>> > wrote:
> >>> > >> > Done!
> >>> > >> >
> >>> > >> > Brock
> >>> > >> >
> >>> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com>
> >>> > wrote:
> >>> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all
> >>> of
> >>> > the
> >>> > >> PMC
> >>> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
> >>> > Jenkins
> >>> > >> >> access so I can setup CI?
> >>> > >> >>
> >>> > >> >> J
> >>> > >> >>
> >>> > >> >>
> >>> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com>
> >>> wrote:
> >>> > >> >>>
> >>> > >> >>> I think sonar will be a good way forward.
> >>> > >> >>>
> >>> > >> >>> As for maven repo for snapshots, there is one snapshots repo
> >>> from
> >>> > >> Apache
> >>> > >> >>> at
> >>> https://repository.apache.org/content/repositories/snapshots/.
> >>> > >> >>> I think this is good enough for the purpose and released
> >>> artifacts
> >>> > can
> >>> > >> be
> >>> > >> >>> uploaded to maven repos.
> >>> > >> >>>
> >>> > >> >>> regards
> >>> > >> >>> Rahul
> >>> > >> >>>
> >>> > >> >>>
> >>> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
> >>> > >> >>>>
> >>> > >> >>>> Hi,
> >>> > >> >>>>
> >>> > >> >>>> over the last three weeks we've been working hard on making
> >>> Crunch
> >>> > >> >>>> ready for CI. It turned out to be much more work than initially
> >>> > >> >>>> expected and I cursed myself more than once for starting this.
> >>> In
> >>> > the
> >>> > >> >>>> end it took four people and many patches to pull this off
> >>> (thanks a
> >>> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
> >>> > >> >>>> interested, these were the issues that lead us to CI readiness:
> >>> > >> >>>>
> >>> > >> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
> >>> > >> project
> >>> > >> >>>> * CRUNCH-17: Split unit tests from integration tests
> >>> > >> >>>> * CRUNCH-18: Improve build stability
> >>> > >> >>>> * CRUNCH-21: Make temporary directory configurable
> >>> > >> >>>> * CRUNCH-24: Make test suite suiteable for continuous
> >>> integration
> >>> > >> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous
> >>> > integration
> >>> > >> >>>>
> >>> > >> >>>> We can now start with basic jobs to detect broken commits by
> >>> > running
> >>> > >> >>>> the unit and integration test suites. I suggest starting with
> >>> an
> >>> > >> >>>> on-commit build using "mvn package" and a nightly build using
> >>> "mvn
> >>> > >> >>>> verify" (which runs the integration test suite). Running the
> >>> > >> >>>> integration tests on-commit would be great but may be too
> >>> > expensive.
> >>> > >> >>>>
> >>> > >> >>>> To get started we need access to Apache's Jenkins instance [1]
> >>> > which
> >>> > >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
> >>> > processes,
> >>> > >> >>>> but perhaps a mentor can help us out here?
> >>> > >> >>>>
> >>> > >> >>>>
> >>> > >> >>>> Next steps:
> >>> > >> >>>>
> >>> > >> >>>> CI gives us the ability to regularly run reports to check code
> >>> > >> >>>> quality. We already started preparations for this (Maven site
> >>> > setup,
> >>> > >> >>>> checkstyle config and basic reports), but I think the best
> >>> thing
> >>> > would
> >>> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at
> >>> work,
> >>> > it
> >>> > >> >>>> bundles all kinds of code metrics in a nice interface and
> >>> points
> >>> > out
> >>> > >> >>>> where refactoring time is best spent. It's the most convenient
> >>> way
> >>> > of
> >>> > >> >>>> tracking code quality over time and catches many issues human
> >>> > >> >>>> reviewers usually overlook.
> >>> > >> >>>>
> >>> > >> >>>> Additional to reports, it's also quite common to deploy nightly
> >>> > >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
> >>> > Maven
> >>> > >> >>>> sites. I'm not sure if this is possible at Apache for legal
> >>> reasons
> >>> > >> >>>> though.
> >>> > >> >>>>
> >>> > >> >>>>
> >>> > >> >>>> Please let me know where you think we should go next. I know
> >>> this
> >>> > >> >>>> whole CI episode has been a nuisance, so thanks again for your
> >>> > help.
> >>> > >> >>>>
> >>> > >> >>>> Regards,
> >>> > >> >>>> Matthias
> >>> > >> >>>>
> >>> > >> >>>> [1] https://builds.apache.org
> >>> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
> >>> > >> >>>> [3] https://analysis.apache.org
> >>> > >> >>>
> >>> > >> >>>
> >>> > >> >>
> >>> > >> >>
> >>> > >> >>
> >>> > >> >> --
> >>> > >> >> Director of Data Science
> >>> > >> >> Cloudera
> >>> > >> >> Twitter: @josh_wills
> >>> > >> >>
> >>> > >> >
> >>> > >> >
> >>> > >> >
> >>> > >> > --
> >>> > >> > Apache MRUnit - Unit testing MapReduce -
> >>> > >> http://incubator.apache.org/mrunit/
> >>> > >>
> >>> > >>
> >>> > >>
> >>> > >> --
> >>> > >> Director of Data Science
> >>> > >> Cloudera
> >>> > >> Twitter: @josh_wills
> >>> > >>
> >>> > >
> >>> > >
> >>> > >
> >>> > > --
> >>> > > Apache MRUnit - Unit testing MapReduce -
> >>> > http://incubator.apache.org/mrunit/
> >>> >
> >>>
> >>>
> >>>
> >>> --
> >>> Apache MRUnit - Unit testing MapReduce -
> >>> http://incubator.apache.org/mrunit/
> >>>
> >>
> >>
> >>
> >> --
> >> Director of Data Science
> >> Cloudera <http://www.cloudera.com>
> >> Twitter: @josh_wills <http://twitter.com/josh_wills>
> >>
> >>
> >
> >
> > --
> > Director of Data Science
> > Cloudera <http://www.cloudera.com>
> > Twitter: @josh_wills <http://twitter.com/josh_wills>
> >
> >
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Brock Noland <br...@cloudera.com>.
done!
brock@minotaur:~$ list_appgroups.pl --user=mafr
hudson-jobadmin
On Sun, Aug 5, 2012 at 7:42 PM, Josh Wills <jw...@cloudera.com> wrote:
> Ya know, Matthias understands this stuff. Brock, can you give Matthias
> (mafr) admin access to Jenkins?
>
> J
>
> On Sun, Aug 5, 2012 at 5:23 PM, Josh Wills <jw...@cloudera.com> wrote:
>
>> Thanks Brock. I edited it to become incubator-crunch, and it's running
>> now: https://builds.apache.org/job/incubator-crunch/
>>
>> My Jenkins knowledge is also near-zero, so if anyone has any settings
>> they would like me to enable, please let me know.
>>
>> On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <br...@cloudera.com> wrote:
>>
>>> Hmm. My experience with Jenkins is nearly zero. I just created
>>> "crunch-trunk" job, can you edit it?
>>>
>>> https://builds.apache.org/job/crunch-trunk/configure
>>>
>>>
>>>
>>> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <jo...@gmail.com> wrote:
>>>
>>> > I get "jwills is missing the Create permission"
>>> >
>>> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com>
>>> wrote:
>>> > > According to ldap, we are both in the "hudson-jobadmin" group. There
>>> is
>>> > no
>>> > > "New Job" link in the upper left? The link goes here:
>>> > > https://builds.apache.org/view/All/newJob
>>> > >
>>> > > Brock
>>> > >
>>> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com>
>>> wrote:
>>> > >
>>> > >> Hey Brock,
>>> > >>
>>> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
>>> > >> that I can create a new job-- I just have a dashboard that lists
>>> Build
>>> > >> History/Project Relationship/etc. Is there some admin bit I'm
>>> missing?
>>> > >>
>>> > >> J
>>> > >>
>>> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com>
>>> > wrote:
>>> > >> > Done!
>>> > >> >
>>> > >> > Brock
>>> > >> >
>>> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com>
>>> > wrote:
>>> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all
>>> of
>>> > the
>>> > >> PMC
>>> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
>>> > Jenkins
>>> > >> >> access so I can setup CI?
>>> > >> >>
>>> > >> >> J
>>> > >> >>
>>> > >> >>
>>> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com>
>>> wrote:
>>> > >> >>>
>>> > >> >>> I think sonar will be a good way forward.
>>> > >> >>>
>>> > >> >>> As for maven repo for snapshots, there is one snapshots repo
>>> from
>>> > >> Apache
>>> > >> >>> at
>>> https://repository.apache.org/content/repositories/snapshots/.
>>> > >> >>> I think this is good enough for the purpose and released
>>> artifacts
>>> > can
>>> > >> be
>>> > >> >>> uploaded to maven repos.
>>> > >> >>>
>>> > >> >>> regards
>>> > >> >>> Rahul
>>> > >> >>>
>>> > >> >>>
>>> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>>> > >> >>>>
>>> > >> >>>> Hi,
>>> > >> >>>>
>>> > >> >>>> over the last three weeks we've been working hard on making
>>> Crunch
>>> > >> >>>> ready for CI. It turned out to be much more work than initially
>>> > >> >>>> expected and I cursed myself more than once for starting this.
>>> In
>>> > the
>>> > >> >>>> end it took four people and many patches to pull this off
>>> (thanks a
>>> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>>> > >> >>>> interested, these were the issues that lead us to CI readiness:
>>> > >> >>>>
>>> > >> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
>>> > >> project
>>> > >> >>>> * CRUNCH-17: Split unit tests from integration tests
>>> > >> >>>> * CRUNCH-18: Improve build stability
>>> > >> >>>> * CRUNCH-21: Make temporary directory configurable
>>> > >> >>>> * CRUNCH-24: Make test suite suiteable for continuous
>>> integration
>>> > >> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous
>>> > integration
>>> > >> >>>>
>>> > >> >>>> We can now start with basic jobs to detect broken commits by
>>> > running
>>> > >> >>>> the unit and integration test suites. I suggest starting with
>>> an
>>> > >> >>>> on-commit build using "mvn package" and a nightly build using
>>> "mvn
>>> > >> >>>> verify" (which runs the integration test suite). Running the
>>> > >> >>>> integration tests on-commit would be great but may be too
>>> > expensive.
>>> > >> >>>>
>>> > >> >>>> To get started we need access to Apache's Jenkins instance [1]
>>> > which
>>> > >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
>>> > processes,
>>> > >> >>>> but perhaps a mentor can help us out here?
>>> > >> >>>>
>>> > >> >>>>
>>> > >> >>>> Next steps:
>>> > >> >>>>
>>> > >> >>>> CI gives us the ability to regularly run reports to check code
>>> > >> >>>> quality. We already started preparations for this (Maven site
>>> > setup,
>>> > >> >>>> checkstyle config and basic reports), but I think the best
>>> thing
>>> > would
>>> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at
>>> work,
>>> > it
>>> > >> >>>> bundles all kinds of code metrics in a nice interface and
>>> points
>>> > out
>>> > >> >>>> where refactoring time is best spent. It's the most convenient
>>> way
>>> > of
>>> > >> >>>> tracking code quality over time and catches many issues human
>>> > >> >>>> reviewers usually overlook.
>>> > >> >>>>
>>> > >> >>>> Additional to reports, it's also quite common to deploy nightly
>>> > >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
>>> > Maven
>>> > >> >>>> sites. I'm not sure if this is possible at Apache for legal
>>> reasons
>>> > >> >>>> though.
>>> > >> >>>>
>>> > >> >>>>
>>> > >> >>>> Please let me know where you think we should go next. I know
>>> this
>>> > >> >>>> whole CI episode has been a nuisance, so thanks again for your
>>> > help.
>>> > >> >>>>
>>> > >> >>>> Regards,
>>> > >> >>>> Matthias
>>> > >> >>>>
>>> > >> >>>> [1] https://builds.apache.org
>>> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
>>> > >> >>>> [3] https://analysis.apache.org
>>> > >> >>>
>>> > >> >>>
>>> > >> >>
>>> > >> >>
>>> > >> >>
>>> > >> >> --
>>> > >> >> Director of Data Science
>>> > >> >> Cloudera
>>> > >> >> Twitter: @josh_wills
>>> > >> >>
>>> > >> >
>>> > >> >
>>> > >> >
>>> > >> > --
>>> > >> > Apache MRUnit - Unit testing MapReduce -
>>> > >> http://incubator.apache.org/mrunit/
>>> > >>
>>> > >>
>>> > >>
>>> > >> --
>>> > >> Director of Data Science
>>> > >> Cloudera
>>> > >> Twitter: @josh_wills
>>> > >>
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > Apache MRUnit - Unit testing MapReduce -
>>> > http://incubator.apache.org/mrunit/
>>> >
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Josh Wills <jw...@cloudera.com>.
Ya know, Matthias understands this stuff. Brock, can you give Matthias
(mafr) admin access to Jenkins?
J
On Sun, Aug 5, 2012 at 5:23 PM, Josh Wills <jw...@cloudera.com> wrote:
> Thanks Brock. I edited it to become incubator-crunch, and it's running
> now: https://builds.apache.org/job/incubator-crunch/
>
> My Jenkins knowledge is also near-zero, so if anyone has any settings they
> would like me to enable, please let me know.
>
> On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <br...@cloudera.com> wrote:
>
>> Hmm. My experience with Jenkins is nearly zero. I just created
>> "crunch-trunk" job, can you edit it?
>>
>> https://builds.apache.org/job/crunch-trunk/configure
>>
>>
>>
>> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <jo...@gmail.com> wrote:
>>
>> > I get "jwills is missing the Create permission"
>> >
>> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com>
>> wrote:
>> > > According to ldap, we are both in the "hudson-jobadmin" group. There
>> is
>> > no
>> > > "New Job" link in the upper left? The link goes here:
>> > > https://builds.apache.org/view/All/newJob
>> > >
>> > > Brock
>> > >
>> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com>
>> wrote:
>> > >
>> > >> Hey Brock,
>> > >>
>> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
>> > >> that I can create a new job-- I just have a dashboard that lists
>> Build
>> > >> History/Project Relationship/etc. Is there some admin bit I'm
>> missing?
>> > >>
>> > >> J
>> > >>
>> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com>
>> > wrote:
>> > >> > Done!
>> > >> >
>> > >> > Brock
>> > >> >
>> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com>
>> > wrote:
>> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all of
>> > the
>> > >> PMC
>> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
>> > Jenkins
>> > >> >> access so I can setup CI?
>> > >> >>
>> > >> >> J
>> > >> >>
>> > >> >>
>> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com>
>> wrote:
>> > >> >>>
>> > >> >>> I think sonar will be a good way forward.
>> > >> >>>
>> > >> >>> As for maven repo for snapshots, there is one snapshots repo from
>> > >> Apache
>> > >> >>> at https://repository.apache.org/content/repositories/snapshots/
>> .
>> > >> >>> I think this is good enough for the purpose and released
>> artifacts
>> > can
>> > >> be
>> > >> >>> uploaded to maven repos.
>> > >> >>>
>> > >> >>> regards
>> > >> >>> Rahul
>> > >> >>>
>> > >> >>>
>> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>> > >> >>>>
>> > >> >>>> Hi,
>> > >> >>>>
>> > >> >>>> over the last three weeks we've been working hard on making
>> Crunch
>> > >> >>>> ready for CI. It turned out to be much more work than initially
>> > >> >>>> expected and I cursed myself more than once for starting this.
>> In
>> > the
>> > >> >>>> end it took four people and many patches to pull this off
>> (thanks a
>> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>> > >> >>>> interested, these were the issues that lead us to CI readiness:
>> > >> >>>>
>> > >> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
>> > >> project
>> > >> >>>> * CRUNCH-17: Split unit tests from integration tests
>> > >> >>>> * CRUNCH-18: Improve build stability
>> > >> >>>> * CRUNCH-21: Make temporary directory configurable
>> > >> >>>> * CRUNCH-24: Make test suite suiteable for continuous
>> integration
>> > >> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous
>> > integration
>> > >> >>>>
>> > >> >>>> We can now start with basic jobs to detect broken commits by
>> > running
>> > >> >>>> the unit and integration test suites. I suggest starting with an
>> > >> >>>> on-commit build using "mvn package" and a nightly build using
>> "mvn
>> > >> >>>> verify" (which runs the integration test suite). Running the
>> > >> >>>> integration tests on-commit would be great but may be too
>> > expensive.
>> > >> >>>>
>> > >> >>>> To get started we need access to Apache's Jenkins instance [1]
>> > which
>> > >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
>> > processes,
>> > >> >>>> but perhaps a mentor can help us out here?
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> Next steps:
>> > >> >>>>
>> > >> >>>> CI gives us the ability to regularly run reports to check code
>> > >> >>>> quality. We already started preparations for this (Maven site
>> > setup,
>> > >> >>>> checkstyle config and basic reports), but I think the best thing
>> > would
>> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at
>> work,
>> > it
>> > >> >>>> bundles all kinds of code metrics in a nice interface and points
>> > out
>> > >> >>>> where refactoring time is best spent. It's the most convenient
>> way
>> > of
>> > >> >>>> tracking code quality over time and catches many issues human
>> > >> >>>> reviewers usually overlook.
>> > >> >>>>
>> > >> >>>> Additional to reports, it's also quite common to deploy nightly
>> > >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
>> > Maven
>> > >> >>>> sites. I'm not sure if this is possible at Apache for legal
>> reasons
>> > >> >>>> though.
>> > >> >>>>
>> > >> >>>>
>> > >> >>>> Please let me know where you think we should go next. I know
>> this
>> > >> >>>> whole CI episode has been a nuisance, so thanks again for your
>> > help.
>> > >> >>>>
>> > >> >>>> Regards,
>> > >> >>>> Matthias
>> > >> >>>>
>> > >> >>>> [1] https://builds.apache.org
>> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
>> > >> >>>> [3] https://analysis.apache.org
>> > >> >>>
>> > >> >>>
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> >> --
>> > >> >> Director of Data Science
>> > >> >> Cloudera
>> > >> >> Twitter: @josh_wills
>> > >> >>
>> > >> >
>> > >> >
>> > >> >
>> > >> > --
>> > >> > Apache MRUnit - Unit testing MapReduce -
>> > >> http://incubator.apache.org/mrunit/
>> > >>
>> > >>
>> > >>
>> > >> --
>> > >> Director of Data Science
>> > >> Cloudera
>> > >> Twitter: @josh_wills
>> > >>
>> > >
>> > >
>> > >
>> > > --
>> > > Apache MRUnit - Unit testing MapReduce -
>> > http://incubator.apache.org/mrunit/
>> >
>>
>>
>>
>> --
>> Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>
>
>
> --
> Director of Data Science
> Cloudera <http://www.cloudera.com>
> Twitter: @josh_wills <http://twitter.com/josh_wills>
>
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Continuous Integration: Next Steps
Posted by Josh Wills <jw...@cloudera.com>.
Thanks Brock. I edited it to become incubator-crunch, and it's running now:
https://builds.apache.org/job/incubator-crunch/
My Jenkins knowledge is also near-zero, so if anyone has any settings they
would like me to enable, please let me know.
On Thu, Aug 2, 2012 at 7:14 AM, Brock Noland <br...@cloudera.com> wrote:
> Hmm. My experience with Jenkins is nearly zero. I just created
> "crunch-trunk" job, can you edit it?
>
> https://builds.apache.org/job/crunch-trunk/configure
>
>
>
> On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <jo...@gmail.com> wrote:
>
> > I get "jwills is missing the Create permission"
> >
> > On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com> wrote:
> > > According to ldap, we are both in the "hudson-jobadmin" group. There is
> > no
> > > "New Job" link in the upper left? The link goes here:
> > > https://builds.apache.org/view/All/newJob
> > >
> > > Brock
> > >
> > > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com>
> wrote:
> > >
> > >> Hey Brock,
> > >>
> > >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
> > >> that I can create a new job-- I just have a dashboard that lists Build
> > >> History/Project Relationship/etc. Is there some admin bit I'm missing?
> > >>
> > >> J
> > >>
> > >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com>
> > wrote:
> > >> > Done!
> > >> >
> > >> > Brock
> > >> >
> > >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com>
> > wrote:
> > >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all of
> > the
> > >> PMC
> > >> >> chairs on our teams-- guys, could one of your grant me (jwills)
> > Jenkins
> > >> >> access so I can setup CI?
> > >> >>
> > >> >> J
> > >> >>
> > >> >>
> > >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
> > >> >>>
> > >> >>> I think sonar will be a good way forward.
> > >> >>>
> > >> >>> As for maven repo for snapshots, there is one snapshots repo from
> > >> Apache
> > >> >>> at https://repository.apache.org/content/repositories/snapshots/.
> > >> >>> I think this is good enough for the purpose and released artifacts
> > can
> > >> be
> > >> >>> uploaded to maven repos.
> > >> >>>
> > >> >>> regards
> > >> >>> Rahul
> > >> >>>
> > >> >>>
> > >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
> > >> >>>>
> > >> >>>> Hi,
> > >> >>>>
> > >> >>>> over the last three weeks we've been working hard on making
> Crunch
> > >> >>>> ready for CI. It turned out to be much more work than initially
> > >> >>>> expected and I cursed myself more than once for starting this. In
> > the
> > >> >>>> end it took four people and many patches to pull this off
> (thanks a
> > >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
> > >> >>>> interested, these were the issues that lead us to CI readiness:
> > >> >>>>
> > >> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
> > >> project
> > >> >>>> * CRUNCH-17: Split unit tests from integration tests
> > >> >>>> * CRUNCH-18: Improve build stability
> > >> >>>> * CRUNCH-21: Make temporary directory configurable
> > >> >>>> * CRUNCH-24: Make test suite suiteable for continuous
> integration
> > >> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous
> > integration
> > >> >>>>
> > >> >>>> We can now start with basic jobs to detect broken commits by
> > running
> > >> >>>> the unit and integration test suites. I suggest starting with an
> > >> >>>> on-commit build using "mvn package" and a nightly build using
> "mvn
> > >> >>>> verify" (which runs the integration test suite). Running the
> > >> >>>> integration tests on-commit would be great but may be too
> > expensive.
> > >> >>>>
> > >> >>>> To get started we need access to Apache's Jenkins instance [1]
> > which
> > >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
> > processes,
> > >> >>>> but perhaps a mentor can help us out here?
> > >> >>>>
> > >> >>>>
> > >> >>>> Next steps:
> > >> >>>>
> > >> >>>> CI gives us the ability to regularly run reports to check code
> > >> >>>> quality. We already started preparations for this (Maven site
> > setup,
> > >> >>>> checkstyle config and basic reports), but I think the best thing
> > would
> > >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work,
> > it
> > >> >>>> bundles all kinds of code metrics in a nice interface and points
> > out
> > >> >>>> where refactoring time is best spent. It's the most convenient
> way
> > of
> > >> >>>> tracking code quality over time and catches many issues human
> > >> >>>> reviewers usually overlook.
> > >> >>>>
> > >> >>>> Additional to reports, it's also quite common to deploy nightly
> > >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
> > Maven
> > >> >>>> sites. I'm not sure if this is possible at Apache for legal
> reasons
> > >> >>>> though.
> > >> >>>>
> > >> >>>>
> > >> >>>> Please let me know where you think we should go next. I know this
> > >> >>>> whole CI episode has been a nuisance, so thanks again for your
> > help.
> > >> >>>>
> > >> >>>> Regards,
> > >> >>>> Matthias
> > >> >>>>
> > >> >>>> [1] https://builds.apache.org
> > >> >>>> [2] http://wiki.apache.org/general/Jenkins
> > >> >>>> [3] https://analysis.apache.org
> > >> >>>
> > >> >>>
> > >> >>
> > >> >>
> > >> >>
> > >> >> --
> > >> >> Director of Data Science
> > >> >> Cloudera
> > >> >> Twitter: @josh_wills
> > >> >>
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > Apache MRUnit - Unit testing MapReduce -
> > >> http://incubator.apache.org/mrunit/
> > >>
> > >>
> > >>
> > >> --
> > >> Director of Data Science
> > >> Cloudera
> > >> Twitter: @josh_wills
> > >>
> > >
> > >
> > >
> > > --
> > > Apache MRUnit - Unit testing MapReduce -
> > http://incubator.apache.org/mrunit/
> >
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Continuous Integration: Next Steps
Posted by Brock Noland <br...@cloudera.com>.
Hmm. My experience with Jenkins is nearly zero. I just created
"crunch-trunk" job, can you edit it?
https://builds.apache.org/job/crunch-trunk/configure
On Thu, Aug 2, 2012 at 9:03 AM, Josh Wills <jo...@gmail.com> wrote:
> I get "jwills is missing the Create permission"
>
> On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com> wrote:
> > According to ldap, we are both in the "hudson-jobadmin" group. There is
> no
> > "New Job" link in the upper left? The link goes here:
> > https://builds.apache.org/view/All/newJob
> >
> > Brock
> >
> > On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com> wrote:
> >
> >> Hey Brock,
> >>
> >> I can log in to Jenkins at builds.apache.org, but it doesn't appear
> >> that I can create a new job-- I just have a dashboard that lists Build
> >> History/Project Relationship/etc. Is there some admin bit I'm missing?
> >>
> >> J
> >>
> >> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com>
> wrote:
> >> > Done!
> >> >
> >> > Brock
> >> >
> >> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com>
> wrote:
> >> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all of
> the
> >> PMC
> >> >> chairs on our teams-- guys, could one of your grant me (jwills)
> Jenkins
> >> >> access so I can setup CI?
> >> >>
> >> >> J
> >> >>
> >> >>
> >> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
> >> >>>
> >> >>> I think sonar will be a good way forward.
> >> >>>
> >> >>> As for maven repo for snapshots, there is one snapshots repo from
> >> Apache
> >> >>> at https://repository.apache.org/content/repositories/snapshots/.
> >> >>> I think this is good enough for the purpose and released artifacts
> can
> >> be
> >> >>> uploaded to maven repos.
> >> >>>
> >> >>> regards
> >> >>> Rahul
> >> >>>
> >> >>>
> >> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
> >> >>>>
> >> >>>> Hi,
> >> >>>>
> >> >>>> over the last three weeks we've been working hard on making Crunch
> >> >>>> ready for CI. It turned out to be much more work than initially
> >> >>>> expected and I cursed myself more than once for starting this. In
> the
> >> >>>> end it took four people and many patches to pull this off (thanks a
> >> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
> >> >>>> interested, these were the issues that lead us to CI readiness:
> >> >>>>
> >> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
> >> project
> >> >>>> * CRUNCH-17: Split unit tests from integration tests
> >> >>>> * CRUNCH-18: Improve build stability
> >> >>>> * CRUNCH-21: Make temporary directory configurable
> >> >>>> * CRUNCH-24: Make test suite suiteable for continuous integration
> >> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous
> integration
> >> >>>>
> >> >>>> We can now start with basic jobs to detect broken commits by
> running
> >> >>>> the unit and integration test suites. I suggest starting with an
> >> >>>> on-commit build using "mvn package" and a nightly build using "mvn
> >> >>>> verify" (which runs the integration test suite). Running the
> >> >>>> integration tests on-commit would be great but may be too
> expensive.
> >> >>>>
> >> >>>> To get started we need access to Apache's Jenkins instance [1]
> which
> >> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache
> processes,
> >> >>>> but perhaps a mentor can help us out here?
> >> >>>>
> >> >>>>
> >> >>>> Next steps:
> >> >>>>
> >> >>>> CI gives us the ability to regularly run reports to check code
> >> >>>> quality. We already started preparations for this (Maven site
> setup,
> >> >>>> checkstyle config and basic reports), but I think the best thing
> would
> >> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work,
> it
> >> >>>> bundles all kinds of code metrics in a nice interface and points
> out
> >> >>>> where refactoring time is best spent. It's the most convenient way
> of
> >> >>>> tracking code quality over time and catches many issues human
> >> >>>> reviewers usually overlook.
> >> >>>>
> >> >>>> Additional to reports, it's also quite common to deploy nightly
> >> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date
> Maven
> >> >>>> sites. I'm not sure if this is possible at Apache for legal reasons
> >> >>>> though.
> >> >>>>
> >> >>>>
> >> >>>> Please let me know where you think we should go next. I know this
> >> >>>> whole CI episode has been a nuisance, so thanks again for your
> help.
> >> >>>>
> >> >>>> Regards,
> >> >>>> Matthias
> >> >>>>
> >> >>>> [1] https://builds.apache.org
> >> >>>> [2] http://wiki.apache.org/general/Jenkins
> >> >>>> [3] https://analysis.apache.org
> >> >>>
> >> >>>
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Director of Data Science
> >> >> Cloudera
> >> >> Twitter: @josh_wills
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Apache MRUnit - Unit testing MapReduce -
> >> http://incubator.apache.org/mrunit/
> >>
> >>
> >>
> >> --
> >> Director of Data Science
> >> Cloudera
> >> Twitter: @josh_wills
> >>
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Josh Wills <jo...@gmail.com>.
I get "jwills is missing the Create permission"
On Thu, Aug 2, 2012 at 6:49 AM, Brock Noland <br...@cloudera.com> wrote:
> According to ldap, we are both in the "hudson-jobadmin" group. There is no
> "New Job" link in the upper left? The link goes here:
> https://builds.apache.org/view/All/newJob
>
> Brock
>
> On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com> wrote:
>
>> Hey Brock,
>>
>> I can log in to Jenkins at builds.apache.org, but it doesn't appear
>> that I can create a new job-- I just have a dashboard that lists Build
>> History/Project Relationship/etc. Is there some admin bit I'm missing?
>>
>> J
>>
>> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com> wrote:
>> > Done!
>> >
>> > Brock
>> >
>> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com> wrote:
>> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all of the
>> PMC
>> >> chairs on our teams-- guys, could one of your grant me (jwills) Jenkins
>> >> access so I can setup CI?
>> >>
>> >> J
>> >>
>> >>
>> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
>> >>>
>> >>> I think sonar will be a good way forward.
>> >>>
>> >>> As for maven repo for snapshots, there is one snapshots repo from
>> Apache
>> >>> at https://repository.apache.org/content/repositories/snapshots/.
>> >>> I think this is good enough for the purpose and released artifacts can
>> be
>> >>> uploaded to maven repos.
>> >>>
>> >>> regards
>> >>> Rahul
>> >>>
>> >>>
>> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>> >>>>
>> >>>> Hi,
>> >>>>
>> >>>> over the last three weeks we've been working hard on making Crunch
>> >>>> ready for CI. It turned out to be much more work than initially
>> >>>> expected and I cursed myself more than once for starting this. In the
>> >>>> end it took four people and many patches to pull this off (thanks a
>> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>> >>>> interested, these were the issues that lead us to CI readiness:
>> >>>>
>> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
>> project
>> >>>> * CRUNCH-17: Split unit tests from integration tests
>> >>>> * CRUNCH-18: Improve build stability
>> >>>> * CRUNCH-21: Make temporary directory configurable
>> >>>> * CRUNCH-24: Make test suite suiteable for continuous integration
>> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
>> >>>>
>> >>>> We can now start with basic jobs to detect broken commits by running
>> >>>> the unit and integration test suites. I suggest starting with an
>> >>>> on-commit build using "mvn package" and a nightly build using "mvn
>> >>>> verify" (which runs the integration test suite). Running the
>> >>>> integration tests on-commit would be great but may be too expensive.
>> >>>>
>> >>>> To get started we need access to Apache's Jenkins instance [1] which
>> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
>> >>>> but perhaps a mentor can help us out here?
>> >>>>
>> >>>>
>> >>>> Next steps:
>> >>>>
>> >>>> CI gives us the ability to regularly run reports to check code
>> >>>> quality. We already started preparations for this (Maven site setup,
>> >>>> checkstyle config and basic reports), but I think the best thing would
>> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
>> >>>> bundles all kinds of code metrics in a nice interface and points out
>> >>>> where refactoring time is best spent. It's the most convenient way of
>> >>>> tracking code quality over time and catches many issues human
>> >>>> reviewers usually overlook.
>> >>>>
>> >>>> Additional to reports, it's also quite common to deploy nightly
>> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
>> >>>> sites. I'm not sure if this is possible at Apache for legal reasons
>> >>>> though.
>> >>>>
>> >>>>
>> >>>> Please let me know where you think we should go next. I know this
>> >>>> whole CI episode has been a nuisance, so thanks again for your help.
>> >>>>
>> >>>> Regards,
>> >>>> Matthias
>> >>>>
>> >>>> [1] https://builds.apache.org
>> >>>> [2] http://wiki.apache.org/general/Jenkins
>> >>>> [3] https://analysis.apache.org
>> >>>
>> >>>
>> >>
>> >>
>> >>
>> >> --
>> >> Director of Data Science
>> >> Cloudera
>> >> Twitter: @josh_wills
>> >>
>> >
>> >
>> >
>> > --
>> > Apache MRUnit - Unit testing MapReduce -
>> http://incubator.apache.org/mrunit/
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera
>> Twitter: @josh_wills
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Brock Noland <br...@cloudera.com>.
According to ldap, we are both in the "hudson-jobadmin" group. There is no
"New Job" link in the upper left? The link goes here:
https://builds.apache.org/view/All/newJob
Brock
On Thu, Aug 2, 2012 at 12:13 AM, Josh Wills <jw...@cloudera.com> wrote:
> Hey Brock,
>
> I can log in to Jenkins at builds.apache.org, but it doesn't appear
> that I can create a new job-- I just have a dashboard that lists Build
> History/Project Relationship/etc. Is there some admin bit I'm missing?
>
> J
>
> On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com> wrote:
> > Done!
> >
> > Brock
> >
> > On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com> wrote:
> >> Explicitly emailing Brock, Tom, and Arun, who I believe are all of the
> PMC
> >> chairs on our teams-- guys, could one of your grant me (jwills) Jenkins
> >> access so I can setup CI?
> >>
> >> J
> >>
> >>
> >> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
> >>>
> >>> I think sonar will be a good way forward.
> >>>
> >>> As for maven repo for snapshots, there is one snapshots repo from
> Apache
> >>> at https://repository.apache.org/content/repositories/snapshots/.
> >>> I think this is good enough for the purpose and released artifacts can
> be
> >>> uploaded to maven repos.
> >>>
> >>> regards
> >>> Rahul
> >>>
> >>>
> >>> On 29-07-2012 13:32, Matthias Friedrich wrote:
> >>>>
> >>>> Hi,
> >>>>
> >>>> over the last three weeks we've been working hard on making Crunch
> >>>> ready for CI. It turned out to be much more work than initially
> >>>> expected and I cursed myself more than once for starting this. In the
> >>>> end it took four people and many patches to pull this off (thanks a
> >>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
> >>>> interested, these were the issues that lead us to CI readiness:
> >>>>
> >>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven
> project
> >>>> * CRUNCH-17: Split unit tests from integration tests
> >>>> * CRUNCH-18: Improve build stability
> >>>> * CRUNCH-21: Make temporary directory configurable
> >>>> * CRUNCH-24: Make test suite suiteable for continuous integration
> >>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
> >>>>
> >>>> We can now start with basic jobs to detect broken commits by running
> >>>> the unit and integration test suites. I suggest starting with an
> >>>> on-commit build using "mvn package" and a nightly build using "mvn
> >>>> verify" (which runs the integration test suite). Running the
> >>>> integration tests on-commit would be great but may be too expensive.
> >>>>
> >>>> To get started we need access to Apache's Jenkins instance [1] which
> >>>> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
> >>>> but perhaps a mentor can help us out here?
> >>>>
> >>>>
> >>>> Next steps:
> >>>>
> >>>> CI gives us the ability to regularly run reports to check code
> >>>> quality. We already started preparations for this (Maven site setup,
> >>>> checkstyle config and basic reports), but I think the best thing would
> >>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
> >>>> bundles all kinds of code metrics in a nice interface and points out
> >>>> where refactoring time is best spent. It's the most convenient way of
> >>>> tracking code quality over time and catches many issues human
> >>>> reviewers usually overlook.
> >>>>
> >>>> Additional to reports, it's also quite common to deploy nightly
> >>>> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
> >>>> sites. I'm not sure if this is possible at Apache for legal reasons
> >>>> though.
> >>>>
> >>>>
> >>>> Please let me know where you think we should go next. I know this
> >>>> whole CI episode has been a nuisance, so thanks again for your help.
> >>>>
> >>>> Regards,
> >>>> Matthias
> >>>>
> >>>> [1] https://builds.apache.org
> >>>> [2] http://wiki.apache.org/general/Jenkins
> >>>> [3] https://analysis.apache.org
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Director of Data Science
> >> Cloudera
> >> Twitter: @josh_wills
> >>
> >
> >
> >
> > --
> > Apache MRUnit - Unit testing MapReduce -
> http://incubator.apache.org/mrunit/
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Josh Wills <jw...@cloudera.com>.
Hey Brock,
I can log in to Jenkins at builds.apache.org, but it doesn't appear
that I can create a new job-- I just have a dashboard that lists Build
History/Project Relationship/etc. Is there some admin bit I'm missing?
J
On Tue, Jul 31, 2012 at 7:40 AM, Brock Noland <br...@cloudera.com> wrote:
> Done!
>
> Brock
>
> On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com> wrote:
>> Explicitly emailing Brock, Tom, and Arun, who I believe are all of the PMC
>> chairs on our teams-- guys, could one of your grant me (jwills) Jenkins
>> access so I can setup CI?
>>
>> J
>>
>>
>> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
>>>
>>> I think sonar will be a good way forward.
>>>
>>> As for maven repo for snapshots, there is one snapshots repo from Apache
>>> at https://repository.apache.org/content/repositories/snapshots/.
>>> I think this is good enough for the purpose and released artifacts can be
>>> uploaded to maven repos.
>>>
>>> regards
>>> Rahul
>>>
>>>
>>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>>>>
>>>> Hi,
>>>>
>>>> over the last three weeks we've been working hard on making Crunch
>>>> ready for CI. It turned out to be much more work than initially
>>>> expected and I cursed myself more than once for starting this. In the
>>>> end it took four people and many patches to pull this off (thanks a
>>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>>>> interested, these were the issues that lead us to CI readiness:
>>>>
>>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven project
>>>> * CRUNCH-17: Split unit tests from integration tests
>>>> * CRUNCH-18: Improve build stability
>>>> * CRUNCH-21: Make temporary directory configurable
>>>> * CRUNCH-24: Make test suite suiteable for continuous integration
>>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
>>>>
>>>> We can now start with basic jobs to detect broken commits by running
>>>> the unit and integration test suites. I suggest starting with an
>>>> on-commit build using "mvn package" and a nightly build using "mvn
>>>> verify" (which runs the integration test suite). Running the
>>>> integration tests on-commit would be great but may be too expensive.
>>>>
>>>> To get started we need access to Apache's Jenkins instance [1] which
>>>> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
>>>> but perhaps a mentor can help us out here?
>>>>
>>>>
>>>> Next steps:
>>>>
>>>> CI gives us the ability to regularly run reports to check code
>>>> quality. We already started preparations for this (Maven site setup,
>>>> checkstyle config and basic reports), but I think the best thing would
>>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
>>>> bundles all kinds of code metrics in a nice interface and points out
>>>> where refactoring time is best spent. It's the most convenient way of
>>>> tracking code quality over time and catches many issues human
>>>> reviewers usually overlook.
>>>>
>>>> Additional to reports, it's also quite common to deploy nightly
>>>> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
>>>> sites. I'm not sure if this is possible at Apache for legal reasons
>>>> though.
>>>>
>>>>
>>>> Please let me know where you think we should go next. I know this
>>>> whole CI episode has been a nuisance, so thanks again for your help.
>>>>
>>>> Regards,
>>>> Matthias
>>>>
>>>> [1] https://builds.apache.org
>>>> [2] http://wiki.apache.org/general/Jenkins
>>>> [3] https://analysis.apache.org
>>>
>>>
>>
>>
>>
>> --
>> Director of Data Science
>> Cloudera
>> Twitter: @josh_wills
>>
>
>
>
> --
> Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
--
Director of Data Science
Cloudera
Twitter: @josh_wills
Re: Continuous Integration: Next Steps
Posted by Brock Noland <br...@cloudera.com>.
Done!
Brock
On Tue, Jul 31, 2012 at 9:36 AM, Josh Wills <jw...@cloudera.com> wrote:
> Explicitly emailing Brock, Tom, and Arun, who I believe are all of the PMC
> chairs on our teams-- guys, could one of your grant me (jwills) Jenkins
> access so I can setup CI?
>
> J
>
>
> On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
>>
>> I think sonar will be a good way forward.
>>
>> As for maven repo for snapshots, there is one snapshots repo from Apache
>> at https://repository.apache.org/content/repositories/snapshots/.
>> I think this is good enough for the purpose and released artifacts can be
>> uploaded to maven repos.
>>
>> regards
>> Rahul
>>
>>
>> On 29-07-2012 13:32, Matthias Friedrich wrote:
>>>
>>> Hi,
>>>
>>> over the last three weeks we've been working hard on making Crunch
>>> ready for CI. It turned out to be much more work than initially
>>> expected and I cursed myself more than once for starting this. In the
>>> end it took four people and many patches to pull this off (thanks a
>>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>>> interested, these were the issues that lead us to CI readiness:
>>>
>>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven project
>>> * CRUNCH-17: Split unit tests from integration tests
>>> * CRUNCH-18: Improve build stability
>>> * CRUNCH-21: Make temporary directory configurable
>>> * CRUNCH-24: Make test suite suiteable for continuous integration
>>> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
>>>
>>> We can now start with basic jobs to detect broken commits by running
>>> the unit and integration test suites. I suggest starting with an
>>> on-commit build using "mvn package" and a nightly build using "mvn
>>> verify" (which runs the integration test suite). Running the
>>> integration tests on-commit would be great but may be too expensive.
>>>
>>> To get started we need access to Apache's Jenkins instance [1] which
>>> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
>>> but perhaps a mentor can help us out here?
>>>
>>>
>>> Next steps:
>>>
>>> CI gives us the ability to regularly run reports to check code
>>> quality. We already started preparations for this (Maven site setup,
>>> checkstyle config and basic reports), but I think the best thing would
>>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
>>> bundles all kinds of code metrics in a nice interface and points out
>>> where refactoring time is best spent. It's the most convenient way of
>>> tracking code quality over time and catches many issues human
>>> reviewers usually overlook.
>>>
>>> Additional to reports, it's also quite common to deploy nightly
>>> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
>>> sites. I'm not sure if this is possible at Apache for legal reasons
>>> though.
>>>
>>>
>>> Please let me know where you think we should go next. I know this
>>> whole CI episode has been a nuisance, so thanks again for your help.
>>>
>>> Regards,
>>> Matthias
>>>
>>> [1] https://builds.apache.org
>>> [2] http://wiki.apache.org/general/Jenkins
>>> [3] https://analysis.apache.org
>>
>>
>
>
>
> --
> Director of Data Science
> Cloudera
> Twitter: @josh_wills
>
--
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/
Re: Continuous Integration: Next Steps
Posted by Josh Wills <jw...@cloudera.com>.
Explicitly emailing Brock, Tom, and Arun, who I believe are all of the PMC
chairs on our teams-- guys, could one of your grant me (jwills) Jenkins
access so I can setup CI?
J
On Mon, Jul 30, 2012 at 12:59 AM, Rahul <rs...@xebia.com> wrote:
> I think sonar will be a good way forward.
>
> As for maven repo for snapshots, there is one snapshots repo from Apache
> at https://repository.apache.org/**content/repositories/**snapshots/<https://repository.apache.org/content/repositories/snapshots/>
> .
> I think this is good enough for the purpose and released artifacts can be
> uploaded to maven repos.
>
> regards
> Rahul
>
>
> On 29-07-2012 13:32, Matthias Friedrich wrote:
>
>> Hi,
>>
>> over the last three weeks we've been working hard on making Crunch
>> ready for CI. It turned out to be much more work than initially
>> expected and I cursed myself more than once for starting this. In the
>> end it took four people and many patches to pull this off (thanks a
>> lot for your help, Josh, Rahul, and Gabriel!). In case you're
>> interested, these were the issues that lead us to CI readiness:
>>
>> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven project
>> * CRUNCH-17: Split unit tests from integration tests
>> * CRUNCH-18: Improve build stability
>> * CRUNCH-21: Make temporary directory configurable
>> * CRUNCH-24: Make test suite suiteable for continuous integration
>> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
>>
>> We can now start with basic jobs to detect broken commits by running
>> the unit and integration test suites. I suggest starting with an
>> on-commit build using "mvn package" and a nightly build using "mvn
>> verify" (which runs the integration test suite). Running the
>> integration tests on-commit would be great but may be too expensive.
>>
>> To get started we need access to Apache's Jenkins instance [1] which
>> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
>> but perhaps a mentor can help us out here?
>>
>>
>> Next steps:
>>
>> CI gives us the ability to regularly run reports to check code
>> quality. We already started preparations for this (Maven site setup,
>> checkstyle config and basic reports), but I think the best thing would
>> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
>> bundles all kinds of code metrics in a nice interface and points out
>> where refactoring time is best spent. It's the most convenient way of
>> tracking code quality over time and catches many issues human
>> reviewers usually overlook.
>>
>> Additional to reports, it's also quite common to deploy nightly
>> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
>> sites. I'm not sure if this is possible at Apache for legal reasons
>> though.
>>
>>
>> Please let me know where you think we should go next. I know this
>> whole CI episode has been a nuisance, so thanks again for your help.
>>
>> Regards,
>> Matthias
>>
>> [1] https://builds.apache.org
>> [2] http://wiki.apache.org/**general/Jenkins<http://wiki.apache.org/general/Jenkins>
>> [3] https://analysis.apache.org
>>
>
>
--
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>
Re: Continuous Integration: Next Steps
Posted by Rahul <rs...@xebia.com>.
I think sonar will be a good way forward.
As for maven repo for snapshots, there is one snapshots repo from Apache
at https://repository.apache.org/content/repositories/snapshots/.
I think this is good enough for the purpose and released artifacts can
be uploaded to maven repos.
regards
Rahul
On 29-07-2012 13:32, Matthias Friedrich wrote:
> Hi,
>
> over the last three weeks we've been working hard on making Crunch
> ready for CI. It turned out to be much more work than initially
> expected and I cursed myself more than once for starting this. In the
> end it took four people and many patches to pull this off (thanks a
> lot for your help, Josh, Rahul, and Gabriel!). In case you're
> interested, these were the issues that lead us to CI readiness:
>
> * CRUNCH-8: Refactor Crunch's build into a multi-module Maven project
> * CRUNCH-17: Split unit tests from integration tests
> * CRUNCH-18: Improve build stability
> * CRUNCH-21: Make temporary directory configurable
> * CRUNCH-24: Make test suite suiteable for continuous integration
> * CRUNCH-26: Make Scrunch tests suiteable for continuous integration
>
> We can now start with basic jobs to detect broken commits by running
> the unit and integration test suites. I suggest starting with an
> on-commit build using "mvn package" and a nightly build using "mvn
> verify" (which runs the integration test suite). Running the
> integration tests on-commit would be great but may be too expensive.
>
> To get started we need access to Apache's Jenkins instance [1] which
> apparently only PMC chairs can grant [2]. I'm new to Apache processes,
> but perhaps a mentor can help us out here?
>
>
> Next steps:
>
> CI gives us the ability to regularly run reports to check code
> quality. We already started preparations for this (Maven site setup,
> checkstyle config and basic reports), but I think the best thing would
> be to use Apache's Sonar instance [3]. I use Sonar a lot at work, it
> bundles all kinds of code metrics in a nice interface and points out
> where refactoring time is best spent. It's the most convenient way of
> tracking code quality over time and catches many issues human
> reviewers usually overlook.
>
> Additional to reports, it's also quite common to deploy nightly
> SNAPSHOT builds to a Maven repository and to publish up-to-date Maven
> sites. I'm not sure if this is possible at Apache for legal reasons
> though.
>
>
> Please let me know where you think we should go next. I know this
> whole CI episode has been a nuisance, so thanks again for your help.
>
> Regards,
> Matthias
>
> [1] https://builds.apache.org
> [2] http://wiki.apache.org/general/Jenkins
> [3] https://analysis.apache.org