You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Jim Falgout <ji...@pervasive.com> on 2011/07/01 21:00:19 UTC
trouble building MR-279 branch
Hi,
I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
Thanks,
Jim.
Re: trouble building MR-279 branch
Posted by Josh Wills <jw...@cloudera.com>.
Hey Anupam,
I think Jim is asking about building non-MR application masters using the
RM/NM. I think Arun has promised to post a design doc soon. In the meantime,
I'm just reading through the code and fixing little issues I come across
(e.g., https://issues.apache.org/jira/browse/MAPREDUCE-2630 and
https://issues.apache.org/jira/browse/MAPREDUCE-2633).
Jim, what sort of app are you looking to build? I'd be happy to have some
company wading through the code.
Best,
Josh
On Fri, Jul 1, 2011 at 3:02 PM, Anupam Seth <an...@yahoo-inc.com> wrote:
> Hi Jim,
>
> Not sure if this helps at all, but not sure if you could get an idea by
> looking at the included examples:
>
>
> http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/examples/org/apache/hadoop/examples
>
> that can be run as:
>
> hadoop jar $HADOOP_MAPRED_HOME/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar
> wordcount input output
>
> Regards,
> Anupam
>
> -----Original Message-----
> From: Jim Falgout [mailto:jim.falgout@pervasive.com]
> Sent: Friday, July 01, 2011 4:23 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: RE: trouble building MR-279 branch
>
> Thanks Anupam! I had skipped the step to get protobuf into the maven
> repository. Also, I had Apache Forrest version 0.9 installed. That caused
> the builds to fail on generating some docs. Installed version 0.8 and it
> worked smoothly.
>
> On another note: do you know of any design docs for building an application
> executor using the new resource manager? Just wondering how to get started?
>
> -----Original Message-----
> From: Anupam Seth [mailto:anupams@yahoo-inc.com]
> Sent: Friday, July 01, 2011 2:10 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> ==========
> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from
> http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1
> and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf
> 2.4.0a) ./google/protobuf/descriptor.h:1152: error:
> `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_'
> is private Replace descriptor.cc with
> http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them:
> -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must
> also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean
> install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:jim.falgout@pervasive.com]
> Sent: Friday, July 01, 2011 2:00 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to
> build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a
> "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>
>
RE: trouble building MR-279 branch
Posted by Anupam Seth <an...@yahoo-inc.com>.
Hi Jim,
Not sure if this helps at all, but not sure if you could get an idea by looking at the included examples:
http://svn.apache.org/viewvc/hadoop/common/branches/MR-279/mapreduce/src/examples/org/apache/hadoop/examples
that can be run as:
hadoop jar $HADOOP_MAPRED_HOME/hadoop-mapred-examples-0.22.0-SNAPSHOT.jar wordcount input output
Regards,
Anupam
-----Original Message-----
From: Jim Falgout [mailto:jim.falgout@pervasive.com]
Sent: Friday, July 01, 2011 4:23 PM
To: mapreduce-dev@hadoop.apache.org
Subject: RE: trouble building MR-279 branch
Thanks Anupam! I had skipped the step to get protobuf into the maven repository. Also, I had Apache Forrest version 0.9 installed. That caused the builds to fail on generating some docs. Installed version 0.8 and it worked smoothly.
On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
-----Original Message-----
From: Anupam Seth [mailto:anupams@yahoo-inc.com]
Sent: Friday, July 01, 2011 2:10 PM
To: mapreduce-dev@hadoop.apache.org
Subject: RE: trouble building MR-279 branch
Hi Jim,
Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
Build YARN
==========
These are in addition to what you would normally do.
Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)
Installing protoc requires gcc 4.1.x or higher (RHEL5).
If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
Setup environment
export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib
Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
Build common
cd MR-279/common
ant veryclean mvn-install tar
Build hdfs
cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar
Build yarn
cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal
-----Original Message-----
From: Jim Falgout [mailto:jim.falgout@pervasive.com]
Sent: Friday, July 01, 2011 2:00 PM
To: mapreduce-dev@hadoop.apache.org
Subject: trouble building MR-279 branch
Hi,
I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
Thanks,
Jim.
Re: trouble building MR-279 branch
Posted by Jim Falgout <ji...@pervasive.com>.
Thanks Arun, that's what I'm looking for. I hear you are working on a dev guide. I'm open to working with an early copy and providing feedback. Thanks.
----- Original Message -----
From: Arun C Murthy <ac...@hortonworks.com>
To: mapreduce-dev@hadoop.apache.org <ma...@hadoop.apache.org>
Cc: Sharad Agarwal <sh...@yahoo-inc.com>
Sent: Sun Jul 03 14:16:12 2011
Subject: Re: trouble building MR-279 branch
Jim,
On Jul 1, 2011, at 2:23 PM, Jim Falgout wrote:
> On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
>
Jim, we are in the process of doing a better job with docs etc.
Sharad just gave a talk on writing a new AM at the Hadoop Core Contributor Meeting on 7/1...
Sharad - could you upload your slides?
thanks,
Arun
> -----Original Message-----
> From: Anupam Seth [mailto:anupams@yahoo-inc.com]
> Sent: Friday, July 01, 2011 2:10 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> ==========
> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:jim.falgout@pervasive.com]
> Sent: Friday, July 01, 2011 2:00 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>
Re: trouble building MR-279 branch
Posted by Arun C Murthy <ac...@hortonworks.com>.
Jim,
On Jul 1, 2011, at 2:23 PM, Jim Falgout wrote:
> On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
>
Jim, we are in the process of doing a better job with docs etc.
Sharad just gave a talk on writing a new AM at the Hadoop Core Contributor Meeting on 7/1...
Sharad - could you upload your slides?
thanks,
Arun
> -----Original Message-----
> From: Anupam Seth [mailto:anupams@yahoo-inc.com]
> Sent: Friday, July 01, 2011 2:10 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: RE: trouble building MR-279 branch
>
> Hi Jim,
>
> Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
>
> Build YARN
> ==========
> These are in addition to what you would normally do.
>
> Install protobuf
> use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
>
> install the protoc executable (configure, make, make install)
> install the maven artifact (cd java; mvn install)
>
> Installing protoc requires gcc 4.1.x or higher (RHEL5).
>
> If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
>
> Setup environment
>
> export MAVEN_OPTS=-Xmx512m
> export LD_LIBRARY_PATH=/usr/local/lib
>
> Checkout the MR-279 branch
> svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
>
> Build common
>
> cd MR-279/common
> ant veryclean mvn-install tar
>
> Build hdfs
>
> cd ../hdfs
> ant veryclean mvn-install -Dresolvers=internal tar
>
> Build yarn
>
> cd ../mapreduce
> mvn clean install assembly:assembly
> # If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
> ant veryclean jar jar-test tar -Dresolvers=internal
>
> -----Original Message-----
> From: Jim Falgout [mailto:jim.falgout@pervasive.com]
> Sent: Friday, July 01, 2011 2:00 PM
> To: mapreduce-dev@hadoop.apache.org
> Subject: trouble building MR-279 branch
>
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
>
> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
>
> Thanks,
> Jim.
>
>
RE: trouble building MR-279 branch
Posted by Jim Falgout <ji...@pervasive.com>.
Thanks Anupam! I had skipped the step to get protobuf into the maven repository. Also, I had Apache Forrest version 0.9 installed. That caused the builds to fail on generating some docs. Installed version 0.8 and it worked smoothly.
On another note: do you know of any design docs for building an application executor using the new resource manager? Just wondering how to get started?
-----Original Message-----
From: Anupam Seth [mailto:anupams@yahoo-inc.com]
Sent: Friday, July 01, 2011 2:10 PM
To: mapreduce-dev@hadoop.apache.org
Subject: RE: trouble building MR-279 branch
Hi Jim,
Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
Build YARN
==========
These are in addition to what you would normally do.
Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)
Installing protoc requires gcc 4.1.x or higher (RHEL5).
If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
Setup environment
export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib
Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
Build common
cd MR-279/common
ant veryclean mvn-install tar
Build hdfs
cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar
Build yarn
cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true # NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option # (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal
-----Original Message-----
From: Jim Falgout [mailto:jim.falgout@pervasive.com]
Sent: Friday, July 01, 2011 2:00 PM
To: mapreduce-dev@hadoop.apache.org
Subject: trouble building MR-279 branch
Hi,
I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
Thanks,
Jim.
RE: trouble building MR-279 branch
Posted by Anupam Seth <an...@yahoo-inc.com>.
Hi Jim,
Following are some notes (thanks to Jeff Naisbitt & Bobby Evans).
Build YARN
==========
These are in addition to what you would normally do.
Install protobuf
use Protobuf 2.4.0a (Download from http://code.google.com/p/protobuf/downloads/list) I actually used 2.4.1 and it worked
install the protoc executable (configure, make, make install)
install the maven artifact (cd java; mvn install)
Installing protoc requires gcc 4.1.x or higher (RHEL5).
If the make step fails with (Valid until a fix is released for protobuf 2.4.0a) ./google/protobuf/descriptor.h:1152: error: `google::protobuf::internal::Mutex*google::protobuf::DescriptorPool::mutex_' is private Replace descriptor.cc with http://protobuf.googlecode.com/svn-history/r380/trunk/src/google/protobuf/descriptor.cc
Setup environment
export MAVEN_OPTS=-Xmx512m
export LD_LIBRARY_PATH=/usr/local/lib
Checkout the MR-279 branch
svn checkout http://svn.apache.org/repos/asf/hadoop/common/branches/MR-279
Build common
cd MR-279/common
ant veryclean mvn-install tar
Build hdfs
cd ../hdfs
ant veryclean mvn-install -Dresolvers=internal tar
Build yarn
cd ../mapreduce
mvn clean install assembly:assembly
# If tests are failing, add this option to skip them: -Dmaven.test.skip.exec=true
# NOTE: in order to build on a Mac, you must also include the <code>-P-cbuild</code> option
# (e.g. <code>mvn clean install assembly:assembly -P-cbuild -Dmaven.test.skip.exec=true</code>)
ant veryclean jar jar-test tar -Dresolvers=internal
-----Original Message-----
From: Jim Falgout [mailto:jim.falgout@pervasive.com]
Sent: Friday, July 01, 2011 2:00 PM
To: mapreduce-dev@hadoop.apache.org
Subject: trouble building MR-279 branch
Hi,
I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
Thanks,
Jim.
Re: trouble building MR-279 branch
Posted by Luke Lu <ll...@vicaya.com>.
On Fri, Jul 1, 2011 at 12:00 PM, Jim Falgout <ji...@pervasive.com> wrote:
> Hi,
>
> I've checked out the MR-279 branch and am having trouble getting it to build. Any docs/blogs/notes around that may give some hints?
https://svn.apache.org/repos/asf/hadoop/common/branches/MR-279/mapreduce/INSTALL
> The build is unable to resolve the yarn-server even though I ran a "mvn-install" in the yarn source area to install it in the local repository.
Tell us at which step you're having problems.
__Luke