Re: follow up Hadoop mavenization work

[Tom White <to...@cloudera.com>]

HADOOP-6671 has now received two +1s (one from Eric Yang and one from
me), so I would like to commit it on Tuesday at 16:00 GMT
(http://s.apache.org/6nx). I'll also update the Jenkins jobs for
running test-patch and performing the nightly build.

For developers this change will mean that you need to use Maven to
build Hadoop Common. The build instructions are listed in the
BUILDING.txt file in the patch, as well as at
http://wiki.apache.org/hadoop/HowToContribute and
http://s.apache.org/wb.

Note that HDFS and MapReduce will still use Ant for building, but
follow on JIRAs HDFS-2096 and MAPREDUCE-2607 will introduce Maven to
those builds in the near future. (In terms of staging, it makes sense
for MAPREDUCE-2607 to go in after MAPREDUCE-279, since the MR2 work
uses Maven to build its new modules, so the Mavenization of MapReduce
should build on that work.)

Thanks,
Tom

On Thu, Jul 28, 2011 at 4:41 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Following up with Hadoop Common mavenization (HADOOP-6671) I've just posted
> a patch for HDFS mavenization (HDFS-2096)
>
> The HADOOP-6671 patch integrates all feedback received in the JIRA and, IMO,
> it is ready for prime time.
>
> In order not break HDFS and MAPRED which are still Ant based, there are 2
> patches HDFS-2196 & MAPREDUCE-2741that make some corrections in the ivy
> configuration to work correctly with the Hadoop common JAR (build/published
> by Mavenized build).
>
> HDFS-2096 is not 100% ready, some testcases are failing and native code
> testing is not wired, but everything else (compile, test, package, tar,
> binary, jdiff, etc is wired).
>
> * https://issues.apache.org/jira/browse/HADOOP-6671
> * https://issues.apache.org/jira/browse/HDFS-2196
> * https://issues.apache.org/jira/browse/MAPREDUCE-2741
> * https://issues.apache.org/jira/browse/HDFS-2096
>
> I know these are big changes and we'll have some hiccups, but the benefits
> are big (running testcases is faster, it easily works from IDEs, Maven build
> system can easily be understood by anybody that knows Maven).
>
> Keeping the patches current is time-consuming, because of this, it would be
> great if we can get in the ones ready (HADOOP-6671, HDFS-2196,
> MAPREDUCE-2741) so we can focus on the rest of the Mavenization work.
>
> Thanks.
>
> Alejandro
>

Re: follow up Hadoop mavenization work

[Tom White <to...@cloudera.com>]

HADOOP-6671 is now in trunk. After you do an update you will need to
use Maven 3 to build Common. There are instructions at

https://svn.apache.org/repos/asf/hadoop/common/trunk/hadoop-common/BUILDING.txt

HDFS and MapReduce still use Ant, and the instructions for
cross-project builds are in
http://wiki.apache.org/hadoop/HowToContribute.

Please send any questions/ideas for improvements, etc to the dev list.

Thanks,
Tom

On Tue, Aug 2, 2011 at 9:16 AM, Tom White <to...@cloudera.com> wrote:
> On Sun, Jul 31, 2011 at 9:57 PM, Tom White <to...@cloudera.com> wrote:
>> HADOOP-6671 has now received two +1s (one from Eric Yang and one from
>> me), so I would like to commit it on Tuesday at 16:00 GMT
>> (http://s.apache.org/6nx). I'll also update the Jenkins jobs for
>> running test-patch and performing the nightly build.
>
> I'm going to go ahead and commit the patch now.
>
> Tom
>
>>
>> For developers this change will mean that you need to use Maven to
>> build Hadoop Common. The build instructions are listed in the
>> BUILDING.txt file in the patch, as well as at
>> http://wiki.apache.org/hadoop/HowToContribute and
>> http://s.apache.org/wb.
>>
>> Note that HDFS and MapReduce will still use Ant for building, but
>> follow on JIRAs HDFS-2096 and MAPREDUCE-2607 will introduce Maven to
>> those builds in the near future. (In terms of staging, it makes sense
>> for MAPREDUCE-2607 to go in after MAPREDUCE-279, since the MR2 work
>> uses Maven to build its new modules, so the Mavenization of MapReduce
>> should build on that work.)
>>
>> Thanks,
>> Tom
>>
>> On Thu, Jul 28, 2011 at 4:41 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> Following up with Hadoop Common mavenization (HADOOP-6671) I've just posted
>>> a patch for HDFS mavenization (HDFS-2096)
>>>
>>> The HADOOP-6671 patch integrates all feedback received in the JIRA and, IMO,
>>> it is ready for prime time.
>>>
>>> In order not break HDFS and MAPRED which are still Ant based, there are 2
>>> patches HDFS-2196 & MAPREDUCE-2741that make some corrections in the ivy
>>> configuration to work correctly with the Hadoop common JAR (build/published
>>> by Mavenized build).
>>>
>>> HDFS-2096 is not 100% ready, some testcases are failing and native code
>>> testing is not wired, but everything else (compile, test, package, tar,
>>> binary, jdiff, etc is wired).
>>>
>>> * https://issues.apache.org/jira/browse/HADOOP-6671
>>> * https://issues.apache.org/jira/browse/HDFS-2196
>>> * https://issues.apache.org/jira/browse/MAPREDUCE-2741
>>> * https://issues.apache.org/jira/browse/HDFS-2096
>>>
>>> I know these are big changes and we'll have some hiccups, but the benefits
>>> are big (running testcases is faster, it easily works from IDEs, Maven build
>>> system can easily be understood by anybody that knows Maven).
>>>
>>> Keeping the patches current is time-consuming, because of this, it would be
>>> great if we can get in the ones ready (HADOOP-6671, HDFS-2196,
>>> MAPREDUCE-2741) so we can focus on the rest of the Mavenization work.
>>>
>>> Thanks.
>>>
>>> Alejandro
>>>
>>
>

Re: follow up Hadoop mavenization work

[Tom White <to...@cloudera.com>]

On Sun, Jul 31, 2011 at 9:57 PM, Tom White <to...@cloudera.com> wrote:
> HADOOP-6671 has now received two +1s (one from Eric Yang and one from
> me), so I would like to commit it on Tuesday at 16:00 GMT
> (http://s.apache.org/6nx). I'll also update the Jenkins jobs for
> running test-patch and performing the nightly build.

I'm going to go ahead and commit the patch now.

Tom

>
> For developers this change will mean that you need to use Maven to
> build Hadoop Common. The build instructions are listed in the
> BUILDING.txt file in the patch, as well as at
> http://wiki.apache.org/hadoop/HowToContribute and
> http://s.apache.org/wb.
>
> Note that HDFS and MapReduce will still use Ant for building, but
> follow on JIRAs HDFS-2096 and MAPREDUCE-2607 will introduce Maven to
> those builds in the near future. (In terms of staging, it makes sense
> for MAPREDUCE-2607 to go in after MAPREDUCE-279, since the MR2 work
> uses Maven to build its new modules, so the Mavenization of MapReduce
> should build on that work.)
>
> Thanks,
> Tom
>
> On Thu, Jul 28, 2011 at 4:41 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> Following up with Hadoop Common mavenization (HADOOP-6671) I've just posted
>> a patch for HDFS mavenization (HDFS-2096)
>>
>> The HADOOP-6671 patch integrates all feedback received in the JIRA and, IMO,
>> it is ready for prime time.
>>
>> In order not break HDFS and MAPRED which are still Ant based, there are 2
>> patches HDFS-2196 & MAPREDUCE-2741that make some corrections in the ivy
>> configuration to work correctly with the Hadoop common JAR (build/published
>> by Mavenized build).
>>
>> HDFS-2096 is not 100% ready, some testcases are failing and native code
>> testing is not wired, but everything else (compile, test, package, tar,
>> binary, jdiff, etc is wired).
>>
>> * https://issues.apache.org/jira/browse/HADOOP-6671
>> * https://issues.apache.org/jira/browse/HDFS-2196
>> * https://issues.apache.org/jira/browse/MAPREDUCE-2741
>> * https://issues.apache.org/jira/browse/HDFS-2096
>>
>> I know these are big changes and we'll have some hiccups, but the benefits
>> are big (running testcases is faster, it easily works from IDEs, Maven build
>> system can easily be understood by anybody that knows Maven).
>>
>> Keeping the patches current is time-consuming, because of this, it would be
>> great if we can get in the ones ready (HADOOP-6671, HDFS-2196,
>> MAPREDUCE-2741) so we can focus on the rest of the Mavenization work.
>>
>> Thanks.
>>
>> Alejandro
>>
>