You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Tom White <to...@cloudera.com> on 2012/04/16 22:51:03 UTC
Supporting cross-project Jenkins builds
Currently Jenkins QA builds don't support cross-project patches, since
they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
tree, and reject any patches that are not strictly confined to that
tree. For changes that span projects (e.g. moving code from HDFS or MR
to Common, or build system changes) developers have to run test-patch
and tests manually, which is cumbersome.
I'd like to propose that we change the Common Jenkins build so that it
runs from the top-level
(http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
MapReduce builds would be unchanged. This would have two implications:
i) all patches would have to be rooted at the top-level, and ii) the
unit tests for all projects would be run for Common changes.
I think i) is reasonable (and most patches are like this now) - and
it's easy to regenerate a patch that is rooted at the wrong level. For
ii), running all tests for a change in Common is probably a good thing
to do in general.
Thoughts?
Tom
Re: Supporting cross-project Jenkins builds
Posted by Tom White <to...@cloudera.com>.
I have a patch at https://issues.apache.org/jira/browse/HADOOP-8308
which takes the approach that Alejandro outlined.
Cheers,
Tom
On Wed, Apr 18, 2012 at 10:11 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Giri,
>
> On Tue, Apr 17, 2012 at 8:36 PM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> Alejandro,
>>
>> On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> Giri,
>>>
>>> I agree that running ALL tests all time takes a lot of time
>>> (personally I'd prefer we do this at the penalty of longer runs).
>>>
>>> Still we have a problem to solve, we need to find a solution on
>>> test-patch working for ALL maven modules, currently changes outside of
>>> common/hdfs/mapred or cross-projects test-patch does not work.
>>>
>>> So, how about the following approach:
>>>
>>> * All patches must be at trunk/ level
>>> * All patches do a full clean TARBALL creation without running testcases
>>> * From the patch file we find out the maven modules and for those
>>> modules we do javac-warns/javadoc-warns/findbugs/testcases
>>
>> I like this approach of doing a clean tarball.
>> and doing the other checks ( javac warnings, javadoc warnings, findbug
>> warnings and release audit.)
>> for that specific module.
>>
>
> Great, the idea of the doing a clean tarball it to ensure that nothing
> in the build/assembly is broken and that no API change breaks other
> modules.
>
>>>
>>> This would speed up test-patch runs and together with a nightly
>>> jenkins jobs running ALL testcases would give a complete coverage.
>>>
>>
>> test-patch and nightly jenkins jobs running ALL testcase?
>> could you pls explain this?
>
> you want to verify there is no regression because of functional
> changes in all modules, doing this once a day seems reasonable and
> will help identify the culprit early on (that is why I said in my
> original email my preference would be to run everything every time)
>
> thxs
>
> ---
> Alejandro
Re: Supporting cross-project Jenkins builds
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
On Tue, Apr 17, 2012 at 8:36 PM, Giridharan Kesavan
<gk...@hortonworks.com> wrote:
> Alejandro,
>
> On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> Giri,
>>
>> I agree that running ALL tests all time takes a lot of time
>> (personally I'd prefer we do this at the penalty of longer runs).
>>
>> Still we have a problem to solve, we need to find a solution on
>> test-patch working for ALL maven modules, currently changes outside of
>> common/hdfs/mapred or cross-projects test-patch does not work.
>>
>> So, how about the following approach:
>>
>> * All patches must be at trunk/ level
>> * All patches do a full clean TARBALL creation without running testcases
>> * From the patch file we find out the maven modules and for those
>> modules we do javac-warns/javadoc-warns/findbugs/testcases
>
> I like this approach of doing a clean tarball.
> and doing the other checks ( javac warnings, javadoc warnings, findbug
> warnings and release audit.)
> for that specific module.
>
Great, the idea of the doing a clean tarball it to ensure that nothing
in the build/assembly is broken and that no API change breaks other
modules.
>>
>> This would speed up test-patch runs and together with a nightly
>> jenkins jobs running ALL testcases would give a complete coverage.
>>
>
> test-patch and nightly jenkins jobs running ALL testcase?
> could you pls explain this?
you want to verify there is no regression because of functional
changes in all modules, doing this once a day seems reasonable and
will help identify the culprit early on (that is why I said in my
original email my preference would be to run everything every time)
thxs
---
Alejandro
Re: Supporting cross-project Jenkins builds
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
Alejandro,
On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Giri,
>
> I agree that running ALL tests all time takes a lot of time
> (personally I'd prefer we do this at the penalty of longer runs).
>
> Still we have a problem to solve, we need to find a solution on
> test-patch working for ALL maven modules, currently changes outside of
> common/hdfs/mapred or cross-projects test-patch does not work.
>
> So, how about the following approach:
>
> * All patches must be at trunk/ level
> * All patches do a full clean TARBALL creation without running testcases
> * From the patch file we find out the maven modules and for those
> modules we do javac-warns/javadoc-warns/findbugs/testcases
I like this approach of doing a clean tarball.
and doing the other checks ( javac warnings, javadoc warnings, findbug
warnings and release audit.)
for that specific module.
>
> This would speed up test-patch runs and together with a nightly
> jenkins jobs running ALL testcases would give a complete coverage.
>
test-patch and nightly jenkins jobs running ALL testcase?
could you pls explain this?
> Does this seem reasonable?
>
> Thxs.
>
> Alejandro
>
> On Tue, Apr 17, 2012 at 3:31 PM, Tom White <to...@cloudera.com> wrote:
>> Giri,
>>
>> I think Aaron was talking about not running all test cases for changes
>> to any project (e.g. HDFS and MapReduce). My proposal was to run all
>> the tests for any Common change. An HDFS change would only run HDFS
>> tests, and any MapReduce change would only run MapReduce tests.
>>
>> Another thing I didn't mention was that currently Jenkins doesn't run
>> tests or apply patches for any changes in hadoop-tools, which would be
>> fixed by the change I'm suggesting.
>>
>> Tom
>>
>> On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
>> <gk...@hortonworks.com> wrote:
>>> I agree with Aaron. Its going increase the test patch build timings
>>> significantly which may not be very helpful
>>>
>>> Im -1 on this.
>>>
>>> -Giri
>>>
>>>
>>>
>>> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>>>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>>
>>>>> * all testcases should always be run (else a change in hdfs could
>>>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>>>
>>>>
>>>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>>>> dependent Hadoop sub-projects for every patch, but it will also
>>>> dramatically increase the time test-patch takes to run for any given patch.
>>>> In my experience, the vast majority of patches stand little chance of
>>>> breaking the dependent sub-projects, making this largely unnecessary and
>>>> thus a waste of time and Jenkins build slave resources.
>>>>
>>>> --
>>>> Aaron T. Myers
>>>> Software Engineer, Cloudera
>
>
>
> --
> Alejandro
Re: Supporting cross-project Jenkins builds
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> * All patches must be at trunk/ level
> * All patches do a full clean TARBALL creation without running testcases
> * From the patch file we find out the maven modules and for those
> modules we do javac-warns/javadoc-warns/findbugs/testcases
>
+1, I like this proposal a lot. Seems to do a good job of balancing the
need to check for dependent sub-project breakage with the desire to not
unnecessarily inflate the run time of test-patch.
--
Aaron T. Myers
Software Engineer, Cloudera
Re: Supporting cross-project Jenkins builds
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Giri,
I agree that running ALL tests all time takes a lot of time
(personally I'd prefer we do this at the penalty of longer runs).
Still we have a problem to solve, we need to find a solution on
test-patch working for ALL maven modules, currently changes outside of
common/hdfs/mapred or cross-projects test-patch does not work.
So, how about the following approach:
* All patches must be at trunk/ level
* All patches do a full clean TARBALL creation without running testcases
* From the patch file we find out the maven modules and for those
modules we do javac-warns/javadoc-warns/findbugs/testcases
This would speed up test-patch runs and together with a nightly
jenkins jobs running ALL testcases would give a complete coverage.
Does this seem reasonable?
Thxs.
Alejandro
On Tue, Apr 17, 2012 at 3:31 PM, Tom White <to...@cloudera.com> wrote:
> Giri,
>
> I think Aaron was talking about not running all test cases for changes
> to any project (e.g. HDFS and MapReduce). My proposal was to run all
> the tests for any Common change. An HDFS change would only run HDFS
> tests, and any MapReduce change would only run MapReduce tests.
>
> Another thing I didn't mention was that currently Jenkins doesn't run
> tests or apply patches for any changes in hadoop-tools, which would be
> fixed by the change I'm suggesting.
>
> Tom
>
> On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
> <gk...@hortonworks.com> wrote:
>> I agree with Aaron. Its going increase the test patch build timings
>> significantly which may not be very helpful
>>
>> Im -1 on this.
>>
>> -Giri
>>
>>
>>
>> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>>
>>>> * all testcases should always be run (else a change in hdfs could
>>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>>
>>>
>>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>>> dependent Hadoop sub-projects for every patch, but it will also
>>> dramatically increase the time test-patch takes to run for any given patch.
>>> In my experience, the vast majority of patches stand little chance of
>>> breaking the dependent sub-projects, making this largely unnecessary and
>>> thus a waste of time and Jenkins build slave resources.
>>>
>>> --
>>> Aaron T. Myers
>>> Software Engineer, Cloudera
--
Alejandro
Re: Supporting cross-project Jenkins builds
Posted by Tom White <to...@cloudera.com>.
Giri,
I think Aaron was talking about not running all test cases for changes
to any project (e.g. HDFS and MapReduce). My proposal was to run all
the tests for any Common change. An HDFS change would only run HDFS
tests, and any MapReduce change would only run MapReduce tests.
Another thing I didn't mention was that currently Jenkins doesn't run
tests or apply patches for any changes in hadoop-tools, which would be
fixed by the change I'm suggesting.
Tom
On Tue, Apr 17, 2012 at 3:17 PM, Giridharan Kesavan
<gk...@hortonworks.com> wrote:
> I agree with Aaron. Its going increase the test patch build timings
> significantly which may not be very helpful
>
> Im -1 on this.
>
> -Giri
>
>
>
> On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <at...@cloudera.com> wrote:
>> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>>
>>> * all testcases should always be run (else a change in hdfs could
>>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>>
>>
>> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
>> dependent Hadoop sub-projects for every patch, but it will also
>> dramatically increase the time test-patch takes to run for any given patch.
>> In my experience, the vast majority of patches stand little chance of
>> breaking the dependent sub-projects, making this largely unnecessary and
>> thus a waste of time and Jenkins build slave resources.
>>
>> --
>> Aaron T. Myers
>> Software Engineer, Cloudera
Re: Supporting cross-project Jenkins builds
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
PLEASEEEEE!!!!
On Fri, Apr 27, 2012 at 4:48 PM, Tom White <to...@cloudera.com> wrote:
> Just to be clear, the HADOOP-8308 patch is selective about which tests
> it runs, so it doesn't increase build run times. Unless there are any
> objections, I'd like to commit the patch and switch Jenkins to run
> from the top-level directory early next week.
>
> Cheers,
> Tom
>
> On Fri, Apr 27, 2012 at 2:21 AM, Radim Kolar <hs...@filez.com> wrote:
> > Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
> >
> >> I agree with Aaron. Its going increase the test patch build timings
> >> significantly which may not be very helpful
> >
> > There are 9 build machines dedicated to hadoop. I would like that
> automatic
> > checking can finally test my patches like
> > MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
> > HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
> > HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>
>
--
Alejandro
Re: Supporting cross-project Jenkins builds
Posted by Tom White <to...@cloudera.com>.
Just to be clear, the HADOOP-8308 patch is selective about which tests
it runs, so it doesn't increase build run times. Unless there are any
objections, I'd like to commit the patch and switch Jenkins to run
from the top-level directory early next week.
Cheers,
Tom
On Fri, Apr 27, 2012 at 2:21 AM, Radim Kolar <hs...@filez.com> wrote:
> Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
>
>> I agree with Aaron. Its going increase the test patch build timings
>> significantly which may not be very helpful
>
> There are 9 build machines dedicated to hadoop. I would like that automatic
> checking can finally test my patches like
> MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
> HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
> HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>
Re: Supporting cross-project Jenkins builds
Posted by Radim Kolar <hs...@filez.com>.
Dne 18.4.2012 0:17, Giridharan Kesavan napsal(a):
> I agree with Aaron. Its going increase the test patch build timings
> significantly which may not be very helpful
There are 9 build machines dedicated to hadoop. I would like that
automatic checking can finally test my patches like
MAPREDUCE-4116 <https://issues.apache.org/jira/browse/MAPREDUCE-4116>,
HADOOP-8317 <https://issues.apache.org/jira/browse/HADOOP-8317>,
HADOOP-8268. <https://issues.apache.org/jira/browse/HADOOP-8268>
Re: Supporting cross-project Jenkins builds
Posted by Giridharan Kesavan <gk...@hortonworks.com>.
I agree with Aaron. Its going increase the test patch build timings
significantly which may not be very helpful
Im -1 on this.
-Giri
On Mon, Apr 16, 2012 at 2:22 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> * all testcases should always be run (else a change in hdfs could
>> affect yarn/tools but not be detected, or one in yarn affect tools)
>>
>
> I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
> dependent Hadoop sub-projects for every patch, but it will also
> dramatically increase the time test-patch takes to run for any given patch.
> In my experience, the vast majority of patches stand little chance of
> breaking the dependent sub-projects, making this largely unnecessary and
> thus a waste of time and Jenkins build slave resources.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
Re: Supporting cross-project Jenkins builds
Posted by "Aaron T. Myers" <at...@cloudera.com>.
On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> * all testcases should always be run (else a change in hdfs could
> affect yarn/tools but not be detected, or one in yarn affect tools)
>
I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
dependent Hadoop sub-projects for every patch, but it will also
dramatically increase the time test-patch takes to run for any given patch.
In my experience, the vast majority of patches stand little chance of
breaking the dependent sub-projects, making this largely unnecessary and
thus a waste of time and Jenkins build slave resources.
--
Aaron T. Myers
Software Engineer, Cloudera
Re: Supporting cross-project Jenkins builds
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
+1,
I'd still go a bit further in the simplification of what testpatch does.
* patches must at root level (not even try to be smart)
* all testcases should always be run (else a change in hdfs could
affect yarn/tools but not be detected, or one in yarn affect tools)
Thxs.
Alejandro
On Mon, Apr 16, 2012 at 1:55 PM, Aaron T. Myers <at...@cloudera.com> wrote:
> +1
>
> This JIRA has been sitting patch available for a few weeks:
> https://issues.apache.org/jira/browse/HADOOP-7416
>
> I don't think it's quite ready for prime time (it doesn't implement all the
> features you described) but it'd be a good starting point if someone wanted
> to take it over the finish line.
>
> --
> Aaron T. Myers
> Software Engineer, Cloudera
>
>
>
> On Mon, Apr 16, 2012 at 1:51 PM, Tom White <to...@cloudera.com> wrote:
>
>> Currently Jenkins QA builds don't support cross-project patches, since
>> they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
>> tree, and reject any patches that are not strictly confined to that
>> tree. For changes that span projects (e.g. moving code from HDFS or MR
>> to Common, or build system changes) developers have to run test-patch
>> and tests manually, which is cumbersome.
>>
>> I'd like to propose that we change the Common Jenkins build so that it
>> runs from the top-level
>> (http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
>> MapReduce builds would be unchanged. This would have two implications:
>> i) all patches would have to be rooted at the top-level, and ii) the
>> unit tests for all projects would be run for Common changes.
>>
>> I think i) is reasonable (and most patches are like this now) - and
>> it's easy to regenerate a patch that is rooted at the wrong level. For
>> ii), running all tests for a change in Common is probably a good thing
>> to do in general.
>>
>> Thoughts?
>>
>> Tom
>>
--
Alejandro
Re: Supporting cross-project Jenkins builds
Posted by "Aaron T. Myers" <at...@cloudera.com>.
+1
This JIRA has been sitting patch available for a few weeks:
https://issues.apache.org/jira/browse/HADOOP-7416
I don't think it's quite ready for prime time (it doesn't implement all the
features you described) but it'd be a good starting point if someone wanted
to take it over the finish line.
--
Aaron T. Myers
Software Engineer, Cloudera
On Mon, Apr 16, 2012 at 1:51 PM, Tom White <to...@cloudera.com> wrote:
> Currently Jenkins QA builds don't support cross-project patches, since
> they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
> tree, and reject any patches that are not strictly confined to that
> tree. For changes that span projects (e.g. moving code from HDFS or MR
> to Common, or build system changes) developers have to run test-patch
> and tests manually, which is cumbersome.
>
> I'd like to propose that we change the Common Jenkins build so that it
> runs from the top-level
> (http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
> MapReduce builds would be unchanged. This would have two implications:
> i) all patches would have to be rooted at the top-level, and ii) the
> unit tests for all projects would be run for Common changes.
>
> I think i) is reasonable (and most patches are like this now) - and
> it's easy to regenerate a patch that is rooted at the wrong level. For
> ii), running all tests for a change in Common is probably a good thing
> to do in general.
>
> Thoughts?
>
> Tom
>