Building hadoop-common, hadoop-hdfs, hadoop-mapreduce from source, without using the hadoop Ivy / Maven repositories

[Andrei Dragomir <ad...@adobe.com>]

I have some issues with compiling hadoop-common, hdfs and mapreduce from source. 

Right now, the hdfs and mapreduce builds are broken, because of MAPREDUCE-1803 and HDFS-1166. However, even if this build would work, it wouldn't be good for me. What I want to do is use ONLY the hadoop source code for compiling it. 

Up until now, we were using some recipes written by the Cloudera guys, which basically did the following: 

1. build hadoop-common
2. copy hadoop-common/build/*.jar to hdfs/lib and mapreduce/lib. This step does not work anymore, because the hadoop dependency is taken from ivy which automagically downloads it from somewhere. I'd like to be able to shortcut this and just use the libraries that I just built in the first step. 
3. compile hadoop-hdfs
4. copy hadoop-hdfs/build/*.jar to hadoop-mapreduce/lib
5. compile hadoop-mapreduce
6. copy everything in hadoop-hdfs/build and mapreduce/build to hadoop-common build
7. execute ant tar in hadoop-common

My question is: how does one create a hadoop common, hdfs and mapreduce build completely from source (using ivy / maven repos ONLY for third party libraries, NOT the hadoop core libraries for hdfs and mapreduce). 

Thank you, 
  Andrei Dragomir.

Re: Building hadoop-common, hadoop-hdfs, hadoop-mapreduce from source, without using the hadoop Ivy / Maven repositories

[Konstantin Boudnik <co...@yahoo-inc.com>]

Check http://wiki.apache.org/hadoop/HowToContribute 
look for "Changes that span projects" section

On Thu, May 20, 2010 at 07:28AM, Andrei Dragomir wrote:
> I have some issues with compiling hadoop-common, hdfs and mapreduce from source. 
> 
> Right now, the hdfs and mapreduce builds are broken, because of MAPREDUCE-1803 and HDFS-1166. However, even if this build would work, it wouldn't be good for me. What I want to do is use ONLY the hadoop source code for compiling it. 
> 
> Up until now, we were using some recipes written by the Cloudera guys, which basically did the following: 
> 
> 1. build hadoop-common
> 2. copy hadoop-common/build/*.jar to hdfs/lib and mapreduce/lib. This step does not work anymore, because the hadoop dependency is taken from ivy which automagically downloads it from somewhere. I'd like to be able to shortcut this and just use the libraries that I just built in the first step. 
> 3. compile hadoop-hdfs
> 4. copy hadoop-hdfs/build/*.jar to hadoop-mapreduce/lib
> 5. compile hadoop-mapreduce
> 6. copy everything in hadoop-hdfs/build and mapreduce/build to hadoop-common build
> 7. execute ant tar in hadoop-common
> 
> My question is: how does one create a hadoop common, hdfs and mapreduce build completely from source (using ivy / maven repos ONLY for third party libraries, NOT the hadoop core libraries for hdfs and mapreduce). 
> 
> Thank you, 
>   Andrei Dragomir.