You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Steve Loughran <st...@hortonworks.com> on 2015/09/13 15:58:11 UTC
[DISCUSS] fixing jenkins; policing the build
Jenkins is pretty unhappy with the build ... its slowly been collecting patched (including one i put in) triggering test failures, and we've all been lax about fixing them. They're often pretty minor -as an example. trunk has been breaking on java 8 with javadoc errors.
I don't know what others think, but I personally think we need to be a lot more focused on keeping the build happy. This is alongside the Yertus work: that improves how we build; I'm more interested in the process of not breaking the build —and addressing it when it does.
If we were really ruthless, we'd be strict about reverting any patch that triggers a failure. And maybe even halt all other commits until jenkins is happy. That's pretty brutal, but it certainly gets people to care.
A less ruthless but stricter-than-today policy could be
1. Build failures are filed on JIRA @ critical or blocker. They need to take priority.
2. Patches that fix it get priority over everything else: don't be afraid to ping people to keep that build up.
3. We should recognise that trunk will be java8+ only, and even if don't (yet) switch the source to being java8, the jenkins scheduled builds should go to java8+. That way, we can't ignore java8+trunk failures, and don't have to worry about java7 & trunk.
4. Anyone who is a committer can get the login for builds.apache.org<http://builds.apache.org> to trigger rebuilds; It lets you do a quick verify that the full builds are happy.
Thoughts?
Re: [DISCUSS] fixing jenkins; policing the build
Posted by Ravi Prakash <ra...@ymail.com>.
With regard to the unit test failures, maybe it's time for another fix-it day?
+1
On Monday, September 14, 2015 1:50 PM, Colin P. McCabe <cm...@apache.org> wrote:
I think building on Jenkins with jdk8 (even with source version = 1.7)
would help prevent the jdk8 javadoc failures from creeping in.
With regard to the unit test failures, maybe it's time for another fix-it day?
cheers,
Colin
On Sun, Sep 13, 2015 at 6:58 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> Jenkins is pretty unhappy with the build ... its slowly been collecting patched (including one i put in) triggering test failures, and we've all been lax about fixing them. They're often pretty minor -as an example. trunk has been breaking on java 8 with javadoc errors.
>
> I don't know what others think, but I personally think we need to be a lot more focused on keeping the build happy. This is alongside the Yertus work: that improves how we build; I'm more interested in the process of not breaking the build —and addressing it when it does.
>
> If we were really ruthless, we'd be strict about reverting any patch that triggers a failure. And maybe even halt all other commits until jenkins is happy. That's pretty brutal, but it certainly gets people to care.
>
> A less ruthless but stricter-than-today policy could be
>
>
> 1. Build failures are filed on JIRA @ critical or blocker. They need to take priority.
> 2. Patches that fix it get priority over everything else: don't be afraid to ping people to keep that build up.
> 3. We should recognise that trunk will be java8+ only, and even if don't (yet) switch the source to being java8, the jenkins scheduled builds should go to java8+. That way, we can't ignore java8+trunk failures, and don't have to worry about java7 & trunk.
> 4. Anyone who is a committer can get the login for builds.apache.org<http://builds.apache.org> to trigger rebuilds; It lets you do a quick verify that the full builds are happy.
>
> Thoughts?
Re: [DISCUSS] fixing jenkins; policing the build
Posted by Chris Nauroth <cn...@hortonworks.com>.
I was just thinking the same thing. Some of those test fixes in
HADOOP-11984 are relevant for stabilizing the build regardless of the
parallelization work. I'll queue up a few smaller patches.
--Chris Nauroth
On 9/16/15, 2:20 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>
>> On 15 Sep 2015, at 16:49, Chris Nauroth <cn...@hortonworks.com>
>>wrote:
>>
>> It would be great if we could get to a point where a failure in the
>> nightly builds automatically creates a blocker JIRA against the branch's
>> corresponding version. Our tests are nowhere near stable enough for
>>that
>> to be realistic right now, so it would have to be a long-term
>>improvement.
>> We'd have to invest effort into stabilizing the tests first.
>>
>
>Automatic would be nice, but as there are so many causes of build
>breakages right now, we can't even think of that.
>
>> As a side effect, stabilizing the tests would make it feasible to enable
>> parallel execution of tests for faster pre-commit. This is tracked in
>> HADOOP-11984. I stabilized a few of the tests while working on that,
>>but
>> I had to put the effort on hold.
>
>wow ‹that's a serious piece of work. I'm impressed.
>
>Bringing an HDFS run down to 45 min from 2.5 hours would be a great
>achievement.
>
>How about applying some of the source test file patches right now, so
>that the actual patch to turn on parallel test runs would be less?
>
Re: [DISCUSS] fixing jenkins; policing the build
Posted by Steve Loughran <st...@hortonworks.com>.
> On 15 Sep 2015, at 16:49, Chris Nauroth <cn...@hortonworks.com> wrote:
>
> It would be great if we could get to a point where a failure in the
> nightly builds automatically creates a blocker JIRA against the branch's
> corresponding version. Our tests are nowhere near stable enough for that
> to be realistic right now, so it would have to be a long-term improvement.
> We'd have to invest effort into stabilizing the tests first.
>
Automatic would be nice, but as there are so many causes of build breakages right now, we can't even think of that.
> As a side effect, stabilizing the tests would make it feasible to enable
> parallel execution of tests for faster pre-commit. This is tracked in
> HADOOP-11984. I stabilized a few of the tests while working on that, but
> I had to put the effort on hold.
wow —that's a serious piece of work. I'm impressed.
Bringing an HDFS run down to 45 min from 2.5 hours would be a great achievement.
How about applying some of the source test file patches right now, so that the actual patch to turn on parallel test runs would be less?
Re: [DISCUSS] fixing jenkins; policing the build
Posted by Chris Nauroth <cn...@hortonworks.com>.
It would be great if we could get to a point where a failure in the
nightly builds automatically creates a blocker JIRA against the branch's
corresponding version. Our tests are nowhere near stable enough for that
to be realistic right now, so it would have to be a long-term improvement.
We'd have to invest effort into stabilizing the tests first.
As a side effect, stabilizing the tests would make it feasible to enable
parallel execution of tests for faster pre-commit. This is tracked in
HADOOP-11984. I stabilized a few of the tests while working on that, but
I had to put the effort on hold.
--Chris Nauroth
On 9/15/15, 1:51 AM, "Steve Loughran" <st...@hortonworks.com> wrote:
>
>> On 14 Sep 2015, at 21:50, Colin P. McCabe <cm...@apache.org> wrote:
>>
>> I think building on Jenkins with jdk8 (even with source version = 1.7)
>> would help prevent the jdk8 javadoc failures from creeping in.
>>
>
>we are doing Java 8 alongside Java 7, but nobody is worrying about the
>java 8 build failing. (not enough people are worrying about java 7,
>either)
>
>switching to Java 8 only (with that source flag), is simply a matter of
>stopping the java 7 build and having everyone agree to care about jenkins
>trunk builds.
>
>> With regard to the unit test failures, maybe it's time for another
>>fix-it day?
>>
>
>+1, though that was very much a patch backlog day.
>
>I want us all to care more about failing jenkins builds and react fast to
>failures
>
>
Re: [DISCUSS] fixing jenkins; policing the build
Posted by Steve Loughran <st...@hortonworks.com>.
> On 14 Sep 2015, at 21:50, Colin P. McCabe <cm...@apache.org> wrote:
>
> I think building on Jenkins with jdk8 (even with source version = 1.7)
> would help prevent the jdk8 javadoc failures from creeping in.
>
we are doing Java 8 alongside Java 7, but nobody is worrying about the java 8 build failing. (not enough people are worrying about java 7, either)
switching to Java 8 only (with that source flag), is simply a matter of stopping the java 7 build and having everyone agree to care about jenkins trunk builds.
> With regard to the unit test failures, maybe it's time for another fix-it day?
>
+1, though that was very much a patch backlog day.
I want us all to care more about failing jenkins builds and react fast to failures
Re: [DISCUSS] fixing jenkins; policing the build
Posted by "Colin P. McCabe" <cm...@apache.org>.
I think building on Jenkins with jdk8 (even with source version = 1.7)
would help prevent the jdk8 javadoc failures from creeping in.
With regard to the unit test failures, maybe it's time for another fix-it day?
cheers,
Colin
On Sun, Sep 13, 2015 at 6:58 AM, Steve Loughran <st...@hortonworks.com> wrote:
>
> Jenkins is pretty unhappy with the build ... its slowly been collecting patched (including one i put in) triggering test failures, and we've all been lax about fixing them. They're often pretty minor -as an example. trunk has been breaking on java 8 with javadoc errors.
>
> I don't know what others think, but I personally think we need to be a lot more focused on keeping the build happy. This is alongside the Yertus work: that improves how we build; I'm more interested in the process of not breaking the build —and addressing it when it does.
>
> If we were really ruthless, we'd be strict about reverting any patch that triggers a failure. And maybe even halt all other commits until jenkins is happy. That's pretty brutal, but it certainly gets people to care.
>
> A less ruthless but stricter-than-today policy could be
>
>
> 1. Build failures are filed on JIRA @ critical or blocker. They need to take priority.
> 2. Patches that fix it get priority over everything else: don't be afraid to ping people to keep that build up.
> 3. We should recognise that trunk will be java8+ only, and even if don't (yet) switch the source to being java8, the jenkins scheduled builds should go to java8+. That way, we can't ignore java8+trunk failures, and don't have to worry about java7 & trunk.
> 4. Anyone who is a committer can get the login for builds.apache.org<http://builds.apache.org> to trigger rebuilds; It lets you do a quick verify that the full builds are happy.
>
> Thoughts?