You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@mahout.apache.org by Tom Pierce <tc...@cloudera.com> on 2012/03/05 20:40:14 UTC
ongoing jenkins failures
Hi folks,
I spent a little time looking into our Jenkins failures. I am not
very familiar with Jenkins, so this is probably a little remedial -
pointers for other/better things to look at are appreciated!
We've let this go for a long time:
* Last stable build (#1218), 3 mo 3 days ago
Before that, it looks like we were having intermittent failures
(failures at 1196, 1208, 1217 and then 1219-).
Our "workspace" disk usage spiked sometime around build 1370 - going
up over 800MB. Prior to this spike, workspace disk usage was
flat-lined at or very near 0 - going back to build 1218. Since the
spike, it looks like we've been oscillating between 800+0MB workspace
usage. This is likely due to failures occurring at different points
(before or after scratch space is used?).
Between the 1218/1219 Mahout Jenkins reports, we got a report about
Mahout-Examples (#37) failing, with a shell trace, including this
evidence of a misconfiguration on the test box:
[trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
-DskipTests=true -U clean install
Error: JAVA_HOME is not defined correctly.
We cannot execute /home/hudson/tools/java/latest1.6/bin/java
Build step 'Invoke top-level Maven targets' marked build as failure
The last-good and first-bad builds are too old to have a full Jenkins
report, but I think these are the intervening commits (between 1218
and 1219):
http://svn.apache.org/viewvc?rev=1209476&view=rev
http://svn.apache.org/viewvc?rev=1209480&view=rev
http://svn.apache.org/viewvc?rev=1209577&view=rev
http://svn.apache.org/viewvc?rev=1209794&view=rev
Those are pretty small changes except for the last - which added a new
LDA implementation (MAHOUT-897).
Looking through a full batch of Jenkins console output (30MB!) did not
let me pin down the reason Jenkins deems the build unstable (and
certainly nothing that indicates the above changes are to blame). I
see lots of tests run (a few skipped, but no errors/failures), lots of
checkstyle/findbugs problems, but nothing that screams "failure here".
It does seem that the cause is related to the checkstyle/pmd/findbugs
results, due to this line toward the end:
Build step 'Report Violations' changed build result to UNSTABLE
Does anyone know how to trace this back to a specific condition/fault?
-tom
Re: ongoing jenkins failures
Posted by Lance Norskog <go...@gmail.com>.
Right. I'm used to Lucene which keeps patches for years before
committing them. Mahout's ruthless attitude to old patches makes this
more reasonable.
git diff -w and blame -w ignore whitespace. I make patches with diff -w.
Lance
On Thu, Mar 8, 2012 at 11:59 PM, Ted Dunning <te...@gmail.com> wrote:
> I don't think that we have many pending patches at the moment anyway.
>
> Let's get a big batch style points in the bag while Tom has the energy to do the work.
>
> Sent from my iPhone
>
> On Mar 8, 2012, at 9:44 PM, Jake Mannix <ja...@gmail.com> wrote:
>
>> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce <tc...@apache.org> wrote:
>>
>>> I could support this plan, but it might be hard to say something like "ok,
>>> this week we're lowering the threshold another 50 warnings - no more
>>> commits until we're passing again!". It's the sort of thing that is easy
>>> to keep deferring until you eventually forget you meant to do it.
>>>
>>> I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly and
>>> move on. Speaking of that, any feedback on MAHOUT-987?
>>
>>
>> Historically I've been a strong opponent of big batch "fix a ton of
>> whitespace" commits, because it makes for ugly commit/blame history,
>> potentially invalidates patches, and similarly makes for merging other
>> branches (i.e. on GitHub) painful.
>>
>> I'm not sure if I have enough strength left in me to hold to that argument,
>> as my "broken windows theory" sense is deeply offended by the way I've
>> already gone way past "might" bitbucket the "unstable Jenkins build"
>> emails, and into "am totally ignoring it because it's noise"-land.
>>
>> So I guess I'm surprisingly enough a +1 on this ticket, at present. We
>> should hear back from more of a quorum on a commit that hits so many files
>> in different places.
>>
>> -jake
>>
>>
>>>
>>>
>>> -tom
>>>
>>>
>>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>>
>>>> +1 I'd like to see Jenkins become a reliable health indication and
>>>> setting the fb/pmd/cs bar too low does us no service unless we are prepared
>>>> to take those warnings seriously. Is it possible to raise the bar to where
>>>> we are "ok" again and then agree to lower it periodically to get us to
>>>> improve our hygiene index?
>>>>
>>>>
>>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>>
>>>>> Well we already have that in a sense - all the tests still run and we
>>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>>> complaints.
>>>>>
>>>>> My concern would be having 2 separate Jenkins tasks would make it even
>>>>> easier to ignore the non-test warnings.
>>>>>
>>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>>>> let us all test against the same rules as Jenkins in a straightforward
>>>>> way.
>>>>>
>>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>>> habit.
>>>>>
>>>>> -tom
>>>>>
>>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>>>> wrote:
>>>>>>
>>>>>>> About Jenkins:
>>>>>>> Will it be good to create separate maven profiles executed using
>>>>>>> separate
>>>>>>> Jenkins jobs for
>>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>>
>>>>>> Is it the intended distinction between normal and quality build? if
>>>>>> yes, +1, seems reasonable.
>>>>>>
>>>>>
>>>>>
>>>>
>>>
--
Lance Norskog
goksron@gmail.com
Re: ongoing jenkins failures
Posted by Ted Dunning <te...@gmail.com>.
I don't think that we have many pending patches at the moment anyway.
Let's get a big batch style points in the bag while Tom has the energy to do the work.
Sent from my iPhone
On Mar 8, 2012, at 9:44 PM, Jake Mannix <ja...@gmail.com> wrote:
> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce <tc...@apache.org> wrote:
>
>> I could support this plan, but it might be hard to say something like "ok,
>> this week we're lowering the threshold another 50 warnings - no more
>> commits until we're passing again!". It's the sort of thing that is easy
>> to keep deferring until you eventually forget you meant to do it.
>>
>> I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly and
>> move on. Speaking of that, any feedback on MAHOUT-987?
>
>
> Historically I've been a strong opponent of big batch "fix a ton of
> whitespace" commits, because it makes for ugly commit/blame history,
> potentially invalidates patches, and similarly makes for merging other
> branches (i.e. on GitHub) painful.
>
> I'm not sure if I have enough strength left in me to hold to that argument,
> as my "broken windows theory" sense is deeply offended by the way I've
> already gone way past "might" bitbucket the "unstable Jenkins build"
> emails, and into "am totally ignoring it because it's noise"-land.
>
> So I guess I'm surprisingly enough a +1 on this ticket, at present. We
> should hear back from more of a quorum on a commit that hits so many files
> in different places.
>
> -jake
>
>
>>
>>
>> -tom
>>
>>
>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>
>>> +1 I'd like to see Jenkins become a reliable health indication and
>>> setting the fb/pmd/cs bar too low does us no service unless we are prepared
>>> to take those warnings seriously. Is it possible to raise the bar to where
>>> we are "ok" again and then agree to lower it periodically to get us to
>>> improve our hygiene index?
>>>
>>>
>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>
>>>> Well we already have that in a sense - all the tests still run and we
>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>> complaints.
>>>>
>>>> My concern would be having 2 separate Jenkins tasks would make it even
>>>> easier to ignore the non-test warnings.
>>>>
>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>>> let us all test against the same rules as Jenkins in a straightforward
>>>> way.
>>>>
>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>> habit.
>>>>
>>>> -tom
>>>>
>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>>> wrote:
>>>>>
>>>>>> About Jenkins:
>>>>>> Will it be good to create separate maven profiles executed using
>>>>>> separate
>>>>>> Jenkins jobs for
>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>
>>>>> Is it the intended distinction between normal and quality build? if
>>>>> yes, +1, seems reasonable.
>>>>>
>>>>
>>>>
>>>
>>
Re: ongoing jenkins failures
Posted by tom pierce <tc...@apache.org>.
Ok - consensus seems to be +1 with some preference for waiting until
feature freeze or at least not overlapping with another sweeping set of
changes (which makes a ton of sense).
I'll plan to do the mass patches in a few weeks then - and in the
meantime I'll see if I can make our "mvn checkstyle:checkstyle" (and
other rules) give the same set of warnings that Jenkins does.
If I'm going to bother with this housekeeping exercise, I hope we can
keep this up (otherwise I would be happy to simply crank our thresholds
up/checks down now and call it done). I'm not sure we'd have a
volunteer to do this chore for every release...
-tom
On 03/10/2012 11:45 AM, Jeff Eastman wrote:
> And +1 on turning off all whitespace checks. Unless we all use exactly
> the same formatter this will always be an issue. Personally, I think
> "close enough" is fine for Mahout.
>
> On 3/10/12 9:36 AM, Jeff Eastman wrote:
>> +0.5 on doing it now, unless it is going to be *right now*. I have a
>> pile of changes to get into 0.7 that will all need to be done in
>> parallel if we don't want to break clustering. This will entail a
>> number of big patch files and any wholesale reformatting once this
>> begins will be a PITA.
>>
>> +1 on right after code freeze, since forward progress won't be
>> impacted in the interim.
>>
>>
>> On 3/10/12 4:12 AM, Grant Ingersoll wrote:
>>> +1 on doing it now.
>>>
>>> Another approach we might also consider is simply deferring all
>>> reformatting to release time. In other words, all committers
>>> should try to keep things in line when committing, but then at
>>> release time we just do a bulk reformat. I think it's reasonable to
>>> ask people to upgrade their patches after a release anyway. Then,
>>> we could turn off the Jenkins formatting checks, as we know the code
>>> will be pretty for every release and hopefully it won't drift too
>>> much in the interim.
>>>
>>> My bikeshedding proposal,
>>> -Grant
>>>
>>> On Mar 9, 2012, at 12:44 AM, Jake Mannix wrote:
>>>
>>>> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce<tc...@apache.org> wrote:
>>>>
>>>>> I could support this plan, but it might be hard to say something
>>>>> like "ok,
>>>>> this week we're lowering the threshold another 50 warnings - no more
>>>>> commits until we're passing again!". It's the sort of thing that
>>>>> is easy
>>>>> to keep deferring until you eventually forget you meant to do it.
>>>>>
>>>>> I guess I'm a "just rip the band-aid off" kind of guy; suffer
>>>>> quickly and
>>>>> move on. Speaking of that, any feedback on MAHOUT-987?
>>>>
>>>> Historically I've been a strong opponent of big batch "fix a ton of
>>>> whitespace" commits, because it makes for ugly commit/blame history,
>>>> potentially invalidates patches, and similarly makes for merging other
>>>> branches (i.e. on GitHub) painful.
>>>>
>>>> I'm not sure if I have enough strength left in me to hold to that
>>>> argument,
>>>> as my "broken windows theory" sense is deeply offended by the way I've
>>>> already gone way past "might" bitbucket the "unstable Jenkins build"
>>>> emails, and into "am totally ignoring it because it's noise"-land.
>>>>
>>>> So I guess I'm surprisingly enough a +1 on this ticket, at
>>>> present. We
>>>> should hear back from more of a quorum on a commit that hits so
>>>> many files
>>>> in different places.
>>>>
>>>> -jake
>>>>
>>>>
>>>>>
>>>>> -tom
>>>>>
>>>>>
>>>>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>>>>
>>>>>> +1 I'd like to see Jenkins become a reliable health indication and
>>>>>> setting the fb/pmd/cs bar too low does us no service unless we
>>>>>> are prepared
>>>>>> to take those warnings seriously. Is it possible to raise the bar
>>>>>> to where
>>>>>> we are "ok" again and then agree to lower it periodically to get
>>>>>> us to
>>>>>> improve our hygiene index?
>>>>>>
>>>>>>
>>>>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>>>>
>>>>>>> Well we already have that in a sense - all the tests still run
>>>>>>> and we
>>>>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>>>>> complaints.
>>>>>>>
>>>>>>> My concern would be having 2 separate Jenkins tasks would make
>>>>>>> it even
>>>>>>> easier to ignore the non-test warnings.
>>>>>>>
>>>>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>>>>> counts go up, or drop those tasks from Jenkins entirely. This
>>>>>>> would
>>>>>>> let us all test against the same rules as Jenkins in a
>>>>>>> straightforward
>>>>>>> way.
>>>>>>>
>>>>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>>>>> habit.
>>>>>>>
>>>>>>> -tom
>>>>>>>
>>>>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh
>>>>>>>> Ranjan<pr...@xebia.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> About Jenkins:
>>>>>>>>> Will it be good to create separate maven profiles executed using
>>>>>>>>> separate
>>>>>>>>> Jenkins jobs for
>>>>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>>>>
>>>>>>>> Is it the intended distinction between normal and quality
>>>>>>>> build? if
>>>>>>>> yes, +1, seems reasonable.
>>>>>>>>
>>>>>>>
>>> --------------------------------------------
>>> Grant Ingersoll
>>> http://www.lucidimagination.com
>>>
>>>
>>>
>>>
>>
>
Re: ongoing jenkins failures
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
And +1 on turning off all whitespace checks. Unless we all use exactly
the same formatter this will always be an issue. Personally, I think
"close enough" is fine for Mahout.
On 3/10/12 9:36 AM, Jeff Eastman wrote:
> +0.5 on doing it now, unless it is going to be *right now*. I have a
> pile of changes to get into 0.7 that will all need to be done in
> parallel if we don't want to break clustering. This will entail a
> number of big patch files and any wholesale reformatting once this
> begins will be a PITA.
>
> +1 on right after code freeze, since forward progress won't be
> impacted in the interim.
>
>
> On 3/10/12 4:12 AM, Grant Ingersoll wrote:
>> +1 on doing it now.
>>
>> Another approach we might also consider is simply deferring all
>> reformatting to release time. In other words, all committers should
>> try to keep things in line when committing, but then at release time
>> we just do a bulk reformat. I think it's reasonable to ask people to
>> upgrade their patches after a release anyway. Then, we could turn
>> off the Jenkins formatting checks, as we know the code will be pretty
>> for every release and hopefully it won't drift too much in the interim.
>>
>> My bikeshedding proposal,
>> -Grant
>>
>> On Mar 9, 2012, at 12:44 AM, Jake Mannix wrote:
>>
>>> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce<tc...@apache.org> wrote:
>>>
>>>> I could support this plan, but it might be hard to say something
>>>> like "ok,
>>>> this week we're lowering the threshold another 50 warnings - no more
>>>> commits until we're passing again!". It's the sort of thing that
>>>> is easy
>>>> to keep deferring until you eventually forget you meant to do it.
>>>>
>>>> I guess I'm a "just rip the band-aid off" kind of guy; suffer
>>>> quickly and
>>>> move on. Speaking of that, any feedback on MAHOUT-987?
>>>
>>> Historically I've been a strong opponent of big batch "fix a ton of
>>> whitespace" commits, because it makes for ugly commit/blame history,
>>> potentially invalidates patches, and similarly makes for merging other
>>> branches (i.e. on GitHub) painful.
>>>
>>> I'm not sure if I have enough strength left in me to hold to that
>>> argument,
>>> as my "broken windows theory" sense is deeply offended by the way I've
>>> already gone way past "might" bitbucket the "unstable Jenkins build"
>>> emails, and into "am totally ignoring it because it's noise"-land.
>>>
>>> So I guess I'm surprisingly enough a +1 on this ticket, at present. We
>>> should hear back from more of a quorum on a commit that hits so many
>>> files
>>> in different places.
>>>
>>> -jake
>>>
>>>
>>>>
>>>> -tom
>>>>
>>>>
>>>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>>>
>>>>> +1 I'd like to see Jenkins become a reliable health indication and
>>>>> setting the fb/pmd/cs bar too low does us no service unless we are
>>>>> prepared
>>>>> to take those warnings seriously. Is it possible to raise the bar
>>>>> to where
>>>>> we are "ok" again and then agree to lower it periodically to get
>>>>> us to
>>>>> improve our hygiene index?
>>>>>
>>>>>
>>>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>>>
>>>>>> Well we already have that in a sense - all the tests still run
>>>>>> and we
>>>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>>>> complaints.
>>>>>>
>>>>>> My concern would be having 2 separate Jenkins tasks would make it
>>>>>> even
>>>>>> easier to ignore the non-test warnings.
>>>>>>
>>>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>>>>> let us all test against the same rules as Jenkins in a
>>>>>> straightforward
>>>>>> way.
>>>>>>
>>>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>>>> habit.
>>>>>>
>>>>>> -tom
>>>>>>
>>>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> About Jenkins:
>>>>>>>> Will it be good to create separate maven profiles executed using
>>>>>>>> separate
>>>>>>>> Jenkins jobs for
>>>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>>>
>>>>>>> Is it the intended distinction between normal and quality build? if
>>>>>>> yes, +1, seems reasonable.
>>>>>>>
>>>>>>
>> --------------------------------------------
>> Grant Ingersoll
>> http://www.lucidimagination.com
>>
>>
>>
>>
>
Re: ongoing jenkins failures
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
+0.5 on doing it now, unless it is going to be *right now*. I have a
pile of changes to get into 0.7 that will all need to be done in
parallel if we don't want to break clustering. This will entail a number
of big patch files and any wholesale reformatting once this begins will
be a PITA.
+1 on right after code freeze, since forward progress won't be impacted
in the interim.
On 3/10/12 4:12 AM, Grant Ingersoll wrote:
> +1 on doing it now.
>
> Another approach we might also consider is simply deferring all reformatting to release time. In other words, all committers should try to keep things in line when committing, but then at release time we just do a bulk reformat. I think it's reasonable to ask people to upgrade their patches after a release anyway. Then, we could turn off the Jenkins formatting checks, as we know the code will be pretty for every release and hopefully it won't drift too much in the interim.
>
> My bikeshedding proposal,
> -Grant
>
> On Mar 9, 2012, at 12:44 AM, Jake Mannix wrote:
>
>> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce<tc...@apache.org> wrote:
>>
>>> I could support this plan, but it might be hard to say something like "ok,
>>> this week we're lowering the threshold another 50 warnings - no more
>>> commits until we're passing again!". It's the sort of thing that is easy
>>> to keep deferring until you eventually forget you meant to do it.
>>>
>>> I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly and
>>> move on. Speaking of that, any feedback on MAHOUT-987?
>>
>> Historically I've been a strong opponent of big batch "fix a ton of
>> whitespace" commits, because it makes for ugly commit/blame history,
>> potentially invalidates patches, and similarly makes for merging other
>> branches (i.e. on GitHub) painful.
>>
>> I'm not sure if I have enough strength left in me to hold to that argument,
>> as my "broken windows theory" sense is deeply offended by the way I've
>> already gone way past "might" bitbucket the "unstable Jenkins build"
>> emails, and into "am totally ignoring it because it's noise"-land.
>>
>> So I guess I'm surprisingly enough a +1 on this ticket, at present. We
>> should hear back from more of a quorum on a commit that hits so many files
>> in different places.
>>
>> -jake
>>
>>
>>>
>>> -tom
>>>
>>>
>>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>>
>>>> +1 I'd like to see Jenkins become a reliable health indication and
>>>> setting the fb/pmd/cs bar too low does us no service unless we are prepared
>>>> to take those warnings seriously. Is it possible to raise the bar to where
>>>> we are "ok" again and then agree to lower it periodically to get us to
>>>> improve our hygiene index?
>>>>
>>>>
>>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>>
>>>>> Well we already have that in a sense - all the tests still run and we
>>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>>> complaints.
>>>>>
>>>>> My concern would be having 2 separate Jenkins tasks would make it even
>>>>> easier to ignore the non-test warnings.
>>>>>
>>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>>>> let us all test against the same rules as Jenkins in a straightforward
>>>>> way.
>>>>>
>>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>>> habit.
>>>>>
>>>>> -tom
>>>>>
>>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>>>> wrote:
>>>>>>
>>>>>>> About Jenkins:
>>>>>>> Will it be good to create separate maven profiles executed using
>>>>>>> separate
>>>>>>> Jenkins jobs for
>>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>>
>>>>>> Is it the intended distinction between normal and quality build? if
>>>>>> yes, +1, seems reasonable.
>>>>>>
>>>>>
> --------------------------------------------
> Grant Ingersoll
> http://www.lucidimagination.com
>
>
>
>
Re: ongoing jenkins failures
Posted by Grant Ingersoll <gs...@apache.org>.
+1 on doing it now.
Another approach we might also consider is simply deferring all reformatting to release time. In other words, all committers should try to keep things in line when committing, but then at release time we just do a bulk reformat. I think it's reasonable to ask people to upgrade their patches after a release anyway. Then, we could turn off the Jenkins formatting checks, as we know the code will be pretty for every release and hopefully it won't drift too much in the interim.
My bikeshedding proposal,
-Grant
On Mar 9, 2012, at 12:44 AM, Jake Mannix wrote:
> On Thu, Mar 8, 2012 at 9:10 PM, tom pierce <tc...@apache.org> wrote:
>
>> I could support this plan, but it might be hard to say something like "ok,
>> this week we're lowering the threshold another 50 warnings - no more
>> commits until we're passing again!". It's the sort of thing that is easy
>> to keep deferring until you eventually forget you meant to do it.
>>
>> I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly and
>> move on. Speaking of that, any feedback on MAHOUT-987?
>
>
> Historically I've been a strong opponent of big batch "fix a ton of
> whitespace" commits, because it makes for ugly commit/blame history,
> potentially invalidates patches, and similarly makes for merging other
> branches (i.e. on GitHub) painful.
>
> I'm not sure if I have enough strength left in me to hold to that argument,
> as my "broken windows theory" sense is deeply offended by the way I've
> already gone way past "might" bitbucket the "unstable Jenkins build"
> emails, and into "am totally ignoring it because it's noise"-land.
>
> So I guess I'm surprisingly enough a +1 on this ticket, at present. We
> should hear back from more of a quorum on a commit that hits so many files
> in different places.
>
> -jake
>
>
>>
>>
>> -tom
>>
>>
>> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>>
>>> +1 I'd like to see Jenkins become a reliable health indication and
>>> setting the fb/pmd/cs bar too low does us no service unless we are prepared
>>> to take those warnings seriously. Is it possible to raise the bar to where
>>> we are "ok" again and then agree to lower it periodically to get us to
>>> improve our hygiene index?
>>>
>>>
>>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>>
>>>> Well we already have that in a sense - all the tests still run and we
>>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>>> complaints.
>>>>
>>>> My concern would be having 2 separate Jenkins tasks would make it even
>>>> easier to ignore the non-test warnings.
>>>>
>>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>>> let us all test against the same rules as Jenkins in a straightforward
>>>> way.
>>>>
>>>> I'm only bringing this up because it bugs me that I'm starting to
>>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>>> habit.
>>>>
>>>> -tom
>>>>
>>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>>> wrote:
>>>>>
>>>>>> About Jenkins:
>>>>>> Will it be good to create separate maven profiles executed using
>>>>>> separate
>>>>>> Jenkins jobs for
>>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>>
>>>>> Is it the intended distinction between normal and quality build? if
>>>>> yes, +1, seems reasonable.
>>>>>
>>>>
>>>>
>>>
>>
--------------------------------------------
Grant Ingersoll
http://www.lucidimagination.com
Re: ongoing jenkins failures
Posted by Jake Mannix <ja...@gmail.com>.
On Thu, Mar 8, 2012 at 9:10 PM, tom pierce <tc...@apache.org> wrote:
> I could support this plan, but it might be hard to say something like "ok,
> this week we're lowering the threshold another 50 warnings - no more
> commits until we're passing again!". It's the sort of thing that is easy
> to keep deferring until you eventually forget you meant to do it.
>
> I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly and
> move on. Speaking of that, any feedback on MAHOUT-987?
Historically I've been a strong opponent of big batch "fix a ton of
whitespace" commits, because it makes for ugly commit/blame history,
potentially invalidates patches, and similarly makes for merging other
branches (i.e. on GitHub) painful.
I'm not sure if I have enough strength left in me to hold to that argument,
as my "broken windows theory" sense is deeply offended by the way I've
already gone way past "might" bitbucket the "unstable Jenkins build"
emails, and into "am totally ignoring it because it's noise"-land.
So I guess I'm surprisingly enough a +1 on this ticket, at present. We
should hear back from more of a quorum on a commit that hits so many files
in different places.
-jake
>
>
> -tom
>
>
> On 03/08/2012 11:25 AM, Jeff Eastman wrote:
>
>> +1 I'd like to see Jenkins become a reliable health indication and
>> setting the fb/pmd/cs bar too low does us no service unless we are prepared
>> to take those warnings seriously. Is it possible to raise the bar to where
>> we are "ok" again and then agree to lower it periodically to get us to
>> improve our hygiene index?
>>
>>
>> On 3/7/12 7:04 PM, Tom Pierce wrote:
>>
>>> Well we already have that in a sense - all the tests still run and we
>>> can see which fail even if findbugs/pmd/checkstyle have lots of
>>> complaints.
>>>
>>> My concern would be having 2 separate Jenkins tasks would make it even
>>> easier to ignore the non-test warnings.
>>>
>>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>>> counts go up, or drop those tasks from Jenkins entirely. This would
>>> let us all test against the same rules as Jenkins in a straightforward
>>> way.
>>>
>>> I'm only bringing this up because it bugs me that I'm starting to
>>> mentally bit-bucket "build is unstable" email, which is a terrible
>>> habit.
>>>
>>> -tom
>>>
>>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>>> wrote:
>>>
>>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>>> wrote:
>>>>
>>>>> About Jenkins:
>>>>> Will it be good to create separate maven profiles executed using
>>>>> separate
>>>>> Jenkins jobs for
>>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>>> b) quality build with findbugs/checkstyle/pmd
>>>>>
>>>> Is it the intended distinction between normal and quality build? if
>>>> yes, +1, seems reasonable.
>>>>
>>>
>>>
>>
>
Re: ongoing jenkins failures
Posted by tom pierce <tc...@apache.org>.
I could support this plan, but it might be hard to say something like
"ok, this week we're lowering the threshold another 50 warnings - no
more commits until we're passing again!". It's the sort of thing that
is easy to keep deferring until you eventually forget you meant to do it.
I guess I'm a "just rip the band-aid off" kind of guy; suffer quickly
and move on. Speaking of that, any feedback on MAHOUT-987?
-tom
On 03/08/2012 11:25 AM, Jeff Eastman wrote:
> +1 I'd like to see Jenkins become a reliable health indication and
> setting the fb/pmd/cs bar too low does us no service unless we are
> prepared to take those warnings seriously. Is it possible to raise the
> bar to where we are "ok" again and then agree to lower it periodically
> to get us to improve our hygiene index?
>
>
> On 3/7/12 7:04 PM, Tom Pierce wrote:
>> Well we already have that in a sense - all the tests still run and we
>> can see which fail even if findbugs/pmd/checkstyle have lots of
>> complaints.
>>
>> My concern would be having 2 separate Jenkins tasks would make it even
>> easier to ignore the non-test warnings.
>>
>> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
>> counts go up, or drop those tasks from Jenkins entirely. This would
>> let us all test against the same rules as Jenkins in a straightforward
>> way.
>>
>> I'm only bringing this up because it bugs me that I'm starting to
>> mentally bit-bucket "build is unstable" email, which is a terrible
>> habit.
>>
>> -tom
>>
>> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com>
>> wrote:
>>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com>
>>> wrote:
>>>> About Jenkins:
>>>> Will it be good to create separate maven profiles executed using
>>>> separate
>>>> Jenkins jobs for
>>>> a) normal build without findbugs/checkstyle/pmd etc
>>>> b) quality build with findbugs/checkstyle/pmd
>>> Is it the intended distinction between normal and quality build? if
>>> yes, +1, seems reasonable.
>>
>
Re: ongoing jenkins failures
Posted by Jeff Eastman <jd...@windwardsolutions.com>.
+1 I'd like to see Jenkins become a reliable health indication and
setting the fb/pmd/cs bar too low does us no service unless we are
prepared to take those warnings seriously. Is it possible to raise the
bar to where we are "ok" again and then agree to lower it periodically
to get us to improve our hygiene index?
On 3/7/12 7:04 PM, Tom Pierce wrote:
> Well we already have that in a sense - all the tests still run and we
> can see which fail even if findbugs/pmd/checkstyle have lots of
> complaints.
>
> My concern would be having 2 separate Jenkins tasks would make it even
> easier to ignore the non-test warnings.
>
> I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
> counts go up, or drop those tasks from Jenkins entirely. This would
> let us all test against the same rules as Jenkins in a straightforward
> way.
>
> I'm only bringing this up because it bugs me that I'm starting to
> mentally bit-bucket "build is unstable" email, which is a terrible
> habit.
>
> -tom
>
> On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov<dl...@gmail.com> wrote:
>> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan<pr...@xebia.com> wrote:
>>> About Jenkins:
>>> Will it be good to create separate maven profiles executed using separate
>>> Jenkins jobs for
>>> a) normal build without findbugs/checkstyle/pmd etc
>>> b) quality build with findbugs/checkstyle/pmd
>> Is it the intended distinction between normal and quality build? if
>> yes, +1, seems reasonable.
>
Re: ongoing jenkins failures
Posted by Tom Pierce <tc...@cloudera.com>.
Well we already have that in a sense - all the tests still run and we
can see which fail even if findbugs/pmd/checkstyle have lots of
complaints.
My concern would be having 2 separate Jenkins tasks would make it even
easier to ignore the non-test warnings.
I'd much rather make "mvn test" fail when findbugs/pmd/checkstyle
counts go up, or drop those tasks from Jenkins entirely. This would
let us all test against the same rules as Jenkins in a straightforward
way.
I'm only bringing this up because it bugs me that I'm starting to
mentally bit-bucket "build is unstable" email, which is a terrible
habit.
-tom
On Wed, Mar 7, 2012 at 3:59 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan <pr...@xebia.com> wrote:
>> About Jenkins:
>> Will it be good to create separate maven profiles executed using separate
>> Jenkins jobs for
>> a) normal build without findbugs/checkstyle/pmd etc
>> b) quality build with findbugs/checkstyle/pmd
>
> Is it the intended distinction between normal and quality build? if
> yes, +1, seems reasonable.
Re: ongoing jenkins failures
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
On Tue, Mar 6, 2012 at 10:04 PM, Paritosh Ranjan <pr...@xebia.com> wrote:
> About Jenkins:
> Will it be good to create separate maven profiles executed using separate
> Jenkins jobs for
> a) normal build without findbugs/checkstyle/pmd etc
> b) quality build with findbugs/checkstyle/pmd
Is it the intended distinction between normal and quality build? if
yes, +1, seems reasonable.
Re: ongoing jenkins failures
Posted by Paritosh Ranjan <pr...@xebia.com>.
About Jenkins:
Will it be good to create separate maven profiles executed using
separate Jenkins jobs for
a) normal build without findbugs/checkstyle/pmd etc
b) quality build with findbugs/checkstyle/pmd
So, the normal build will not fail due to formatting/findbugs etc issues
and quality build will keep a check on formatting etc. This will also
make the normal build very fast.
I can take up this task it if people agree.
About commits over formatting/static/final minor issues etc:
Since you are planning to do it for next few weeks, so, it might help in
reducing chaos if you pick up (some number of)packages/week or so and
inform others in advance( some info like package names and end time of
code change ).
Paritosh
On 07-03-2012 09:10, tom pierce wrote:
> Thanks for the tips; maybe we should have a precommit target that
> would run tests and checkstyle/pmd/findbugs and fail if counts go up?
> Seems simpler than trying to synchronize IntelliJ/Eclipse/Emacs/VI
> style rules.
>
> I guess what I was really trying to ask was whether there was a
> general commitment within the project to using Jenkins with the
> current style/checkbugs/etc rules.
>
> If there is, and folks will address upticks in error/warning counts as
> they occur, it is worth investing a bit of time to get the build back
> to stable. If not, we should either fix the Jenkins config to reflect
> what we're willing to maintain, or spare the ASF resources and disable
> it.
>
> I guess I'm willing to play janitor once - does anyone object to some
> omnibus commits over the next few weeks that deal with formatting and
> other minor final/static/etc. issues? This will inevitably screw up
> patches sitting around in JIRA, local changes, diffs against svn
> history, etc. in the usual small, but annoying, ways. I'm willing to
> fence off specific files and/or modules, if requested.
>
> Fair warning: I'll treat silence as consent!
>
> -tom
>
>
> On 03/05/2012 08:41 PM, Dmitriy Lyubimov wrote:
>> so bottom line, i just do autoformat and then post the patch to review
>> board and keep adjusting it manually until there are no "reds" there
>> as well.
>>
>> Also at some point i had a script that ran checkstyle and grep'd
>> warnings produced for files that are different from trunk (i.e.
>> current patch only) but it stopped working at some point for some
>> reason, perhaps something was done to maven config or something.
>>
>> Bottom line, if you find a good process that works for you to squish
>> these little critters, please don't hesitate to share. I ended up only
>> 50% there although i spent some time on trying to find an optimal
>> process with eclipse since this project seems to be sensitive about
>> these things.
>>
>> On Mon, Mar 5, 2012 at 5:34 PM, Dmitriy Lyubimov<dl...@gmail.com>
>> wrote:
>>> i've been using an eclipse style format settings which i tuned as
>>> close as i could to match intelliJ's that Sean patches produce.
>>>
>>> Some of those formattings (mostly wrapping preference rules) don't
>>> trigger checkstyle problems regardless of which tool is used but they
>>> do trigger reformatting if i reformat after somebody who formatted
>>> using Intellij and vice versa. Which really makes things difficult
>>> from the point of view of auto formatting.
>>>
>>> Some of the checkstyle problems inherently missed by my eclipse
>>> autoformatting. (such as trailing spaces in comments). But I suspect
>>> Intellij is probably not doing 100% either.
>>>
>>> Bottom line i haven't figured a really good way to address all
>>> formatting rule checks that Mahout's checkstyle settings employ. But i
>>> have a first approximation for eclipse styles config which i can
>>> share.
>>>
>>> Grant also was suggesting a codestyle for eclipse from Lucene which is
>>> probably not significantly better in that regard either.
>>>
>>> If you like intelliJ, you may bug Sean for his styles on that side of
>>> things. But if you run eclipse i'd really like to use one shared style
>>> so autoformat doesn't act differently on something that has already
>>> been formatted.
>>>
>>> -d
>>>
>>> On Mon, Mar 5, 2012 at 4:51 PM, tom pierce<tc...@apache.org> wrote:
>>>> I am starting to agree that Jenkins is calling the build "unstable"
>>>> on style
>>>> grounds, but my evidence is shaky: the style plot has a "caution
>>>> triangle"
>>>> and the checkstyle count has a storm cloud.
>>>>
>>>> I have made a couple of commits to clear up some style warnings
>>>> (which were
>>>> mostly of my own doing), and I was thinking about maybe trying to
>>>> knock off
>>>> a few small commits like this 1-2x/week.
>>>>
>>>> Before I start playing code-janitor I want to make sure I'm not
>>>> going to be
>>>> shoving against the tide. Are the style rules we're checking with
>>>> these
>>>> tools an accurate representation of the project standards? Are we
>>>> willing
>>>> to do things like set up a JIRA trigger to test patches (I think
>>>> Hadoop does
>>>> this), and generally pay attention to Jenkins failures? Is there a
>>>> maven
>>>> target that runs these checks?
>>>>
>>>> -tom
>>>>
>>>>
>>>>
>>>> On 03/05/2012 02:50 PM, Dmitriy Lyubimov wrote:
>>>>> For some reason i concluded it was related to big number PMD/style
>>>>> warnings. but i may be wrong and i don't remember why i concluded
>>>>> that. I certainly know the least about Jenkins and ci stuff.
>>>>>
>>>>> On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce<tc...@cloudera.com>
>>>>> wrote:
>>>>>> Hi folks,
>>>>>>
>>>>>> I spent a little time looking into our Jenkins failures. I am not
>>>>>> very familiar with Jenkins, so this is probably a little remedial -
>>>>>> pointers for other/better things to look at are appreciated!
>>>>>>
>>>>>> We've let this go for a long time:
>>>>>> * Last stable build (#1218), 3 mo 3 days ago
>>>>>>
>>>>>> Before that, it looks like we were having intermittent failures
>>>>>> (failures at 1196, 1208, 1217 and then 1219-).
>>>>>>
>>>>>> Our "workspace" disk usage spiked sometime around build 1370 - going
>>>>>> up over 800MB. Prior to this spike, workspace disk usage was
>>>>>> flat-lined at or very near 0 - going back to build 1218. Since the
>>>>>> spike, it looks like we've been oscillating between 800+0MB
>>>>>> workspace
>>>>>> usage. This is likely due to failures occurring at different points
>>>>>> (before or after scratch space is used?).
>>>>>>
>>>>>> Between the 1218/1219 Mahout Jenkins reports, we got a report about
>>>>>> Mahout-Examples (#37) failing, with a shell trace, including this
>>>>>> evidence of a misconfiguration on the test box:
>>>>>>
>>>>>> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
>>>>>> -DskipTests=true -U clean install
>>>>>> Error: JAVA_HOME is not defined correctly.
>>>>>> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
>>>>>> Build step 'Invoke top-level Maven targets' marked build as failure
>>>>>>
>>>>>> The last-good and first-bad builds are too old to have a full
>>>>>> Jenkins
>>>>>> report, but I think these are the intervening commits (between 1218
>>>>>> and 1219):
>>>>>>
>>>>>> http://svn.apache.org/viewvc?rev=1209476&view=rev
>>>>>> http://svn.apache.org/viewvc?rev=1209480&view=rev
>>>>>> http://svn.apache.org/viewvc?rev=1209577&view=rev
>>>>>> http://svn.apache.org/viewvc?rev=1209794&view=rev
>>>>>>
>>>>>> Those are pretty small changes except for the last - which added
>>>>>> a new
>>>>>> LDA implementation (MAHOUT-897).
>>>>>>
>>>>>> Looking through a full batch of Jenkins console output (30MB!)
>>>>>> did not
>>>>>> let me pin down the reason Jenkins deems the build unstable (and
>>>>>> certainly nothing that indicates the above changes are to blame). I
>>>>>> see lots of tests run (a few skipped, but no errors/failures),
>>>>>> lots of
>>>>>> checkstyle/findbugs problems, but nothing that screams "failure
>>>>>> here".
>>>>>> It does seem that the cause is related to the
>>>>>> checkstyle/pmd/findbugs
>>>>>> results, due to this line toward the end:
>>>>>>
>>>>>> Build step 'Report Violations' changed build result to UNSTABLE
>>>>>>
>>>>>> Does anyone know how to trace this back to a specific
>>>>>> condition/fault?
>>>>>>
>>>>>> -tom
>>>>
>
Re: ongoing jenkins failures
Posted by tom pierce <tc...@apache.org>.
Thanks for the tips; maybe we should have a precommit target that would
run tests and checkstyle/pmd/findbugs and fail if counts go up? Seems
simpler than trying to synchronize IntelliJ/Eclipse/Emacs/VI style rules.
I guess what I was really trying to ask was whether there was a general
commitment within the project to using Jenkins with the current
style/checkbugs/etc rules.
If there is, and folks will address upticks in error/warning counts as
they occur, it is worth investing a bit of time to get the build back to
stable. If not, we should either fix the Jenkins config to reflect what
we're willing to maintain, or spare the ASF resources and disable it.
I guess I'm willing to play janitor once - does anyone object to some
omnibus commits over the next few weeks that deal with formatting and
other minor final/static/etc. issues? This will inevitably screw up
patches sitting around in JIRA, local changes, diffs against svn
history, etc. in the usual small, but annoying, ways. I'm willing to
fence off specific files and/or modules, if requested.
Fair warning: I'll treat silence as consent!
-tom
On 03/05/2012 08:41 PM, Dmitriy Lyubimov wrote:
> so bottom line, i just do autoformat and then post the patch to review
> board and keep adjusting it manually until there are no "reds" there
> as well.
>
> Also at some point i had a script that ran checkstyle and grep'd
> warnings produced for files that are different from trunk (i.e.
> current patch only) but it stopped working at some point for some
> reason, perhaps something was done to maven config or something.
>
> Bottom line, if you find a good process that works for you to squish
> these little critters, please don't hesitate to share. I ended up only
> 50% there although i spent some time on trying to find an optimal
> process with eclipse since this project seems to be sensitive about
> these things.
>
> On Mon, Mar 5, 2012 at 5:34 PM, Dmitriy Lyubimov<dl...@gmail.com> wrote:
>> i've been using an eclipse style format settings which i tuned as
>> close as i could to match intelliJ's that Sean patches produce.
>>
>> Some of those formattings (mostly wrapping preference rules) don't
>> trigger checkstyle problems regardless of which tool is used but they
>> do trigger reformatting if i reformat after somebody who formatted
>> using Intellij and vice versa. Which really makes things difficult
>> from the point of view of auto formatting.
>>
>> Some of the checkstyle problems inherently missed by my eclipse
>> autoformatting. (such as trailing spaces in comments). But I suspect
>> Intellij is probably not doing 100% either.
>>
>> Bottom line i haven't figured a really good way to address all
>> formatting rule checks that Mahout's checkstyle settings employ. But i
>> have a first approximation for eclipse styles config which i can
>> share.
>>
>> Grant also was suggesting a codestyle for eclipse from Lucene which is
>> probably not significantly better in that regard either.
>>
>> If you like intelliJ, you may bug Sean for his styles on that side of
>> things. But if you run eclipse i'd really like to use one shared style
>> so autoformat doesn't act differently on something that has already
>> been formatted.
>>
>> -d
>>
>> On Mon, Mar 5, 2012 at 4:51 PM, tom pierce<tc...@apache.org> wrote:
>>> I am starting to agree that Jenkins is calling the build "unstable" on style
>>> grounds, but my evidence is shaky: the style plot has a "caution triangle"
>>> and the checkstyle count has a storm cloud.
>>>
>>> I have made a couple of commits to clear up some style warnings (which were
>>> mostly of my own doing), and I was thinking about maybe trying to knock off
>>> a few small commits like this 1-2x/week.
>>>
>>> Before I start playing code-janitor I want to make sure I'm not going to be
>>> shoving against the tide. Are the style rules we're checking with these
>>> tools an accurate representation of the project standards? Are we willing
>>> to do things like set up a JIRA trigger to test patches (I think Hadoop does
>>> this), and generally pay attention to Jenkins failures? Is there a maven
>>> target that runs these checks?
>>>
>>> -tom
>>>
>>>
>>>
>>> On 03/05/2012 02:50 PM, Dmitriy Lyubimov wrote:
>>>> For some reason i concluded it was related to big number PMD/style
>>>> warnings. but i may be wrong and i don't remember why i concluded
>>>> that. I certainly know the least about Jenkins and ci stuff.
>>>>
>>>> On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce<tc...@cloudera.com> wrote:
>>>>> Hi folks,
>>>>>
>>>>> I spent a little time looking into our Jenkins failures. I am not
>>>>> very familiar with Jenkins, so this is probably a little remedial -
>>>>> pointers for other/better things to look at are appreciated!
>>>>>
>>>>> We've let this go for a long time:
>>>>> * Last stable build (#1218), 3 mo 3 days ago
>>>>>
>>>>> Before that, it looks like we were having intermittent failures
>>>>> (failures at 1196, 1208, 1217 and then 1219-).
>>>>>
>>>>> Our "workspace" disk usage spiked sometime around build 1370 - going
>>>>> up over 800MB. Prior to this spike, workspace disk usage was
>>>>> flat-lined at or very near 0 - going back to build 1218. Since the
>>>>> spike, it looks like we've been oscillating between 800+0MB workspace
>>>>> usage. This is likely due to failures occurring at different points
>>>>> (before or after scratch space is used?).
>>>>>
>>>>> Between the 1218/1219 Mahout Jenkins reports, we got a report about
>>>>> Mahout-Examples (#37) failing, with a shell trace, including this
>>>>> evidence of a misconfiguration on the test box:
>>>>>
>>>>> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
>>>>> -DskipTests=true -U clean install
>>>>> Error: JAVA_HOME is not defined correctly.
>>>>> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
>>>>> Build step 'Invoke top-level Maven targets' marked build as failure
>>>>>
>>>>> The last-good and first-bad builds are too old to have a full Jenkins
>>>>> report, but I think these are the intervening commits (between 1218
>>>>> and 1219):
>>>>>
>>>>> http://svn.apache.org/viewvc?rev=1209476&view=rev
>>>>> http://svn.apache.org/viewvc?rev=1209480&view=rev
>>>>> http://svn.apache.org/viewvc?rev=1209577&view=rev
>>>>> http://svn.apache.org/viewvc?rev=1209794&view=rev
>>>>>
>>>>> Those are pretty small changes except for the last - which added a new
>>>>> LDA implementation (MAHOUT-897).
>>>>>
>>>>> Looking through a full batch of Jenkins console output (30MB!) did not
>>>>> let me pin down the reason Jenkins deems the build unstable (and
>>>>> certainly nothing that indicates the above changes are to blame). I
>>>>> see lots of tests run (a few skipped, but no errors/failures), lots of
>>>>> checkstyle/findbugs problems, but nothing that screams "failure here".
>>>>> It does seem that the cause is related to the checkstyle/pmd/findbugs
>>>>> results, due to this line toward the end:
>>>>>
>>>>> Build step 'Report Violations' changed build result to UNSTABLE
>>>>>
>>>>> Does anyone know how to trace this back to a specific condition/fault?
>>>>>
>>>>> -tom
>>>
Re: ongoing jenkins failures
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
so bottom line, i just do autoformat and then post the patch to review
board and keep adjusting it manually until there are no "reds" there
as well.
Also at some point i had a script that ran checkstyle and grep'd
warnings produced for files that are different from trunk (i.e.
current patch only) but it stopped working at some point for some
reason, perhaps something was done to maven config or something.
Bottom line, if you find a good process that works for you to squish
these little critters, please don't hesitate to share. I ended up only
50% there although i spent some time on trying to find an optimal
process with eclipse since this project seems to be sensitive about
these things.
On Mon, Mar 5, 2012 at 5:34 PM, Dmitriy Lyubimov <dl...@gmail.com> wrote:
> i've been using an eclipse style format settings which i tuned as
> close as i could to match intelliJ's that Sean patches produce.
>
> Some of those formattings (mostly wrapping preference rules) don't
> trigger checkstyle problems regardless of which tool is used but they
> do trigger reformatting if i reformat after somebody who formatted
> using Intellij and vice versa. Which really makes things difficult
> from the point of view of auto formatting.
>
> Some of the checkstyle problems inherently missed by my eclipse
> autoformatting. (such as trailing spaces in comments). But I suspect
> Intellij is probably not doing 100% either.
>
> Bottom line i haven't figured a really good way to address all
> formatting rule checks that Mahout's checkstyle settings employ. But i
> have a first approximation for eclipse styles config which i can
> share.
>
> Grant also was suggesting a codestyle for eclipse from Lucene which is
> probably not significantly better in that regard either.
>
> If you like intelliJ, you may bug Sean for his styles on that side of
> things. But if you run eclipse i'd really like to use one shared style
> so autoformat doesn't act differently on something that has already
> been formatted.
>
> -d
>
> On Mon, Mar 5, 2012 at 4:51 PM, tom pierce <tc...@apache.org> wrote:
>> I am starting to agree that Jenkins is calling the build "unstable" on style
>> grounds, but my evidence is shaky: the style plot has a "caution triangle"
>> and the checkstyle count has a storm cloud.
>>
>> I have made a couple of commits to clear up some style warnings (which were
>> mostly of my own doing), and I was thinking about maybe trying to knock off
>> a few small commits like this 1-2x/week.
>>
>> Before I start playing code-janitor I want to make sure I'm not going to be
>> shoving against the tide. Are the style rules we're checking with these
>> tools an accurate representation of the project standards? Are we willing
>> to do things like set up a JIRA trigger to test patches (I think Hadoop does
>> this), and generally pay attention to Jenkins failures? Is there a maven
>> target that runs these checks?
>>
>> -tom
>>
>>
>>
>> On 03/05/2012 02:50 PM, Dmitriy Lyubimov wrote:
>>>
>>> For some reason i concluded it was related to big number PMD/style
>>> warnings. but i may be wrong and i don't remember why i concluded
>>> that. I certainly know the least about Jenkins and ci stuff.
>>>
>>> On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce<tc...@cloudera.com> wrote:
>>>>
>>>> Hi folks,
>>>>
>>>> I spent a little time looking into our Jenkins failures. I am not
>>>> very familiar with Jenkins, so this is probably a little remedial -
>>>> pointers for other/better things to look at are appreciated!
>>>>
>>>> We've let this go for a long time:
>>>> * Last stable build (#1218), 3 mo 3 days ago
>>>>
>>>> Before that, it looks like we were having intermittent failures
>>>> (failures at 1196, 1208, 1217 and then 1219-).
>>>>
>>>> Our "workspace" disk usage spiked sometime around build 1370 - going
>>>> up over 800MB. Prior to this spike, workspace disk usage was
>>>> flat-lined at or very near 0 - going back to build 1218. Since the
>>>> spike, it looks like we've been oscillating between 800+0MB workspace
>>>> usage. This is likely due to failures occurring at different points
>>>> (before or after scratch space is used?).
>>>>
>>>> Between the 1218/1219 Mahout Jenkins reports, we got a report about
>>>> Mahout-Examples (#37) failing, with a shell trace, including this
>>>> evidence of a misconfiguration on the test box:
>>>>
>>>> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
>>>> -DskipTests=true -U clean install
>>>> Error: JAVA_HOME is not defined correctly.
>>>> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
>>>> Build step 'Invoke top-level Maven targets' marked build as failure
>>>>
>>>> The last-good and first-bad builds are too old to have a full Jenkins
>>>> report, but I think these are the intervening commits (between 1218
>>>> and 1219):
>>>>
>>>> http://svn.apache.org/viewvc?rev=1209476&view=rev
>>>> http://svn.apache.org/viewvc?rev=1209480&view=rev
>>>> http://svn.apache.org/viewvc?rev=1209577&view=rev
>>>> http://svn.apache.org/viewvc?rev=1209794&view=rev
>>>>
>>>> Those are pretty small changes except for the last - which added a new
>>>> LDA implementation (MAHOUT-897).
>>>>
>>>> Looking through a full batch of Jenkins console output (30MB!) did not
>>>> let me pin down the reason Jenkins deems the build unstable (and
>>>> certainly nothing that indicates the above changes are to blame). I
>>>> see lots of tests run (a few skipped, but no errors/failures), lots of
>>>> checkstyle/findbugs problems, but nothing that screams "failure here".
>>>> It does seem that the cause is related to the checkstyle/pmd/findbugs
>>>> results, due to this line toward the end:
>>>>
>>>> Build step 'Report Violations' changed build result to UNSTABLE
>>>>
>>>> Does anyone know how to trace this back to a specific condition/fault?
>>>>
>>>> -tom
>>
>>
Re: ongoing jenkins failures
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
i've been using an eclipse style format settings which i tuned as
close as i could to match intelliJ's that Sean patches produce.
Some of those formattings (mostly wrapping preference rules) don't
trigger checkstyle problems regardless of which tool is used but they
do trigger reformatting if i reformat after somebody who formatted
using Intellij and vice versa. Which really makes things difficult
from the point of view of auto formatting.
Some of the checkstyle problems inherently missed by my eclipse
autoformatting. (such as trailing spaces in comments). But I suspect
Intellij is probably not doing 100% either.
Bottom line i haven't figured a really good way to address all
formatting rule checks that Mahout's checkstyle settings employ. But i
have a first approximation for eclipse styles config which i can
share.
Grant also was suggesting a codestyle for eclipse from Lucene which is
probably not significantly better in that regard either.
If you like intelliJ, you may bug Sean for his styles on that side of
things. But if you run eclipse i'd really like to use one shared style
so autoformat doesn't act differently on something that has already
been formatted.
-d
On Mon, Mar 5, 2012 at 4:51 PM, tom pierce <tc...@apache.org> wrote:
> I am starting to agree that Jenkins is calling the build "unstable" on style
> grounds, but my evidence is shaky: the style plot has a "caution triangle"
> and the checkstyle count has a storm cloud.
>
> I have made a couple of commits to clear up some style warnings (which were
> mostly of my own doing), and I was thinking about maybe trying to knock off
> a few small commits like this 1-2x/week.
>
> Before I start playing code-janitor I want to make sure I'm not going to be
> shoving against the tide. Are the style rules we're checking with these
> tools an accurate representation of the project standards? Are we willing
> to do things like set up a JIRA trigger to test patches (I think Hadoop does
> this), and generally pay attention to Jenkins failures? Is there a maven
> target that runs these checks?
>
> -tom
>
>
>
> On 03/05/2012 02:50 PM, Dmitriy Lyubimov wrote:
>>
>> For some reason i concluded it was related to big number PMD/style
>> warnings. but i may be wrong and i don't remember why i concluded
>> that. I certainly know the least about Jenkins and ci stuff.
>>
>> On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce<tc...@cloudera.com> wrote:
>>>
>>> Hi folks,
>>>
>>> I spent a little time looking into our Jenkins failures. I am not
>>> very familiar with Jenkins, so this is probably a little remedial -
>>> pointers for other/better things to look at are appreciated!
>>>
>>> We've let this go for a long time:
>>> * Last stable build (#1218), 3 mo 3 days ago
>>>
>>> Before that, it looks like we were having intermittent failures
>>> (failures at 1196, 1208, 1217 and then 1219-).
>>>
>>> Our "workspace" disk usage spiked sometime around build 1370 - going
>>> up over 800MB. Prior to this spike, workspace disk usage was
>>> flat-lined at or very near 0 - going back to build 1218. Since the
>>> spike, it looks like we've been oscillating between 800+0MB workspace
>>> usage. This is likely due to failures occurring at different points
>>> (before or after scratch space is used?).
>>>
>>> Between the 1218/1219 Mahout Jenkins reports, we got a report about
>>> Mahout-Examples (#37) failing, with a shell trace, including this
>>> evidence of a misconfiguration on the test box:
>>>
>>> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
>>> -DskipTests=true -U clean install
>>> Error: JAVA_HOME is not defined correctly.
>>> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
>>> Build step 'Invoke top-level Maven targets' marked build as failure
>>>
>>> The last-good and first-bad builds are too old to have a full Jenkins
>>> report, but I think these are the intervening commits (between 1218
>>> and 1219):
>>>
>>> http://svn.apache.org/viewvc?rev=1209476&view=rev
>>> http://svn.apache.org/viewvc?rev=1209480&view=rev
>>> http://svn.apache.org/viewvc?rev=1209577&view=rev
>>> http://svn.apache.org/viewvc?rev=1209794&view=rev
>>>
>>> Those are pretty small changes except for the last - which added a new
>>> LDA implementation (MAHOUT-897).
>>>
>>> Looking through a full batch of Jenkins console output (30MB!) did not
>>> let me pin down the reason Jenkins deems the build unstable (and
>>> certainly nothing that indicates the above changes are to blame). I
>>> see lots of tests run (a few skipped, but no errors/failures), lots of
>>> checkstyle/findbugs problems, but nothing that screams "failure here".
>>> It does seem that the cause is related to the checkstyle/pmd/findbugs
>>> results, due to this line toward the end:
>>>
>>> Build step 'Report Violations' changed build result to UNSTABLE
>>>
>>> Does anyone know how to trace this back to a specific condition/fault?
>>>
>>> -tom
>
>
Re: ongoing jenkins failures
Posted by tom pierce <tc...@apache.org>.
I am starting to agree that Jenkins is calling the build "unstable" on
style grounds, but my evidence is shaky: the style plot has a "caution
triangle" and the checkstyle count has a storm cloud.
I have made a couple of commits to clear up some style warnings (which
were mostly of my own doing), and I was thinking about maybe trying to
knock off a few small commits like this 1-2x/week.
Before I start playing code-janitor I want to make sure I'm not going to
be shoving against the tide. Are the style rules we're checking with
these tools an accurate representation of the project standards? Are we
willing to do things like set up a JIRA trigger to test patches (I think
Hadoop does this), and generally pay attention to Jenkins failures? Is
there a maven target that runs these checks?
-tom
On 03/05/2012 02:50 PM, Dmitriy Lyubimov wrote:
> For some reason i concluded it was related to big number PMD/style
> warnings. but i may be wrong and i don't remember why i concluded
> that. I certainly know the least about Jenkins and ci stuff.
>
> On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce<tc...@cloudera.com> wrote:
>> Hi folks,
>>
>> I spent a little time looking into our Jenkins failures. I am not
>> very familiar with Jenkins, so this is probably a little remedial -
>> pointers for other/better things to look at are appreciated!
>>
>> We've let this go for a long time:
>> * Last stable build (#1218), 3 mo 3 days ago
>>
>> Before that, it looks like we were having intermittent failures
>> (failures at 1196, 1208, 1217 and then 1219-).
>>
>> Our "workspace" disk usage spiked sometime around build 1370 - going
>> up over 800MB. Prior to this spike, workspace disk usage was
>> flat-lined at or very near 0 - going back to build 1218. Since the
>> spike, it looks like we've been oscillating between 800+0MB workspace
>> usage. This is likely due to failures occurring at different points
>> (before or after scratch space is used?).
>>
>> Between the 1218/1219 Mahout Jenkins reports, we got a report about
>> Mahout-Examples (#37) failing, with a shell trace, including this
>> evidence of a misconfiguration on the test box:
>>
>> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
>> -DskipTests=true -U clean install
>> Error: JAVA_HOME is not defined correctly.
>> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
>> Build step 'Invoke top-level Maven targets' marked build as failure
>>
>> The last-good and first-bad builds are too old to have a full Jenkins
>> report, but I think these are the intervening commits (between 1218
>> and 1219):
>>
>> http://svn.apache.org/viewvc?rev=1209476&view=rev
>> http://svn.apache.org/viewvc?rev=1209480&view=rev
>> http://svn.apache.org/viewvc?rev=1209577&view=rev
>> http://svn.apache.org/viewvc?rev=1209794&view=rev
>>
>> Those are pretty small changes except for the last - which added a new
>> LDA implementation (MAHOUT-897).
>>
>> Looking through a full batch of Jenkins console output (30MB!) did not
>> let me pin down the reason Jenkins deems the build unstable (and
>> certainly nothing that indicates the above changes are to blame). I
>> see lots of tests run (a few skipped, but no errors/failures), lots of
>> checkstyle/findbugs problems, but nothing that screams "failure here".
>> It does seem that the cause is related to the checkstyle/pmd/findbugs
>> results, due to this line toward the end:
>>
>> Build step 'Report Violations' changed build result to UNSTABLE
>>
>> Does anyone know how to trace this back to a specific condition/fault?
>>
>> -tom
Re: ongoing jenkins failures
Posted by Dmitriy Lyubimov <dl...@gmail.com>.
For some reason i concluded it was related to big number PMD/style
warnings. but i may be wrong and i don't remember why i concluded
that. I certainly know the least about Jenkins and ci stuff.
On Mon, Mar 5, 2012 at 11:40 AM, Tom Pierce <tc...@cloudera.com> wrote:
> Hi folks,
>
> I spent a little time looking into our Jenkins failures. I am not
> very familiar with Jenkins, so this is probably a little remedial -
> pointers for other/better things to look at are appreciated!
>
> We've let this go for a long time:
> * Last stable build (#1218), 3 mo 3 days ago
>
> Before that, it looks like we were having intermittent failures
> (failures at 1196, 1208, 1217 and then 1219-).
>
> Our "workspace" disk usage spiked sometime around build 1370 - going
> up over 800MB. Prior to this spike, workspace disk usage was
> flat-lined at or very near 0 - going back to build 1218. Since the
> spike, it looks like we've been oscillating between 800+0MB workspace
> usage. This is likely due to failures occurring at different points
> (before or after scratch space is used?).
>
> Between the 1218/1219 Mahout Jenkins reports, we got a report about
> Mahout-Examples (#37) failing, with a shell trace, including this
> evidence of a misconfiguration on the test box:
>
> [trunk] $ /home/hudson/tools/maven/apache-maven-2.2.1/bin/mvn
> -DskipTests=true -U clean install
> Error: JAVA_HOME is not defined correctly.
> We cannot execute /home/hudson/tools/java/latest1.6/bin/java
> Build step 'Invoke top-level Maven targets' marked build as failure
>
> The last-good and first-bad builds are too old to have a full Jenkins
> report, but I think these are the intervening commits (between 1218
> and 1219):
>
> http://svn.apache.org/viewvc?rev=1209476&view=rev
> http://svn.apache.org/viewvc?rev=1209480&view=rev
> http://svn.apache.org/viewvc?rev=1209577&view=rev
> http://svn.apache.org/viewvc?rev=1209794&view=rev
>
> Those are pretty small changes except for the last - which added a new
> LDA implementation (MAHOUT-897).
>
> Looking through a full batch of Jenkins console output (30MB!) did not
> let me pin down the reason Jenkins deems the build unstable (and
> certainly nothing that indicates the above changes are to blame). I
> see lots of tests run (a few skipped, but no errors/failures), lots of
> checkstyle/findbugs problems, but nothing that screams "failure here".
> It does seem that the cause is related to the checkstyle/pmd/findbugs
> results, due to this line toward the end:
>
> Build step 'Report Violations' changed build result to UNSTABLE
>
> Does anyone know how to trace this back to a specific condition/fault?
>
> -tom