You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by "Wilkes, Chris" <cw...@gmail.com> on 2010/01/04 21:05:18 UTC
How can I compile mapreduce to use my own version of hadoop-core 0.21-SNAPSHOT?
I'm trying to build the 0.21.0-SNAPSHOT packages of common, hdfs, and
mapreduce and am doing so with "ant binary" (I'm more used to maven so
I'm not very good at debugging this), I've also done a mvn-install to
put the jars into the local maven repo.
"ant binary" in seems to pull down the hdfs-core.jar file from the
Hudson server:
java -cp ./build/hadoop-mapred-0.21.0-SNAPSHOT/lib/hadoop-
core-0.21.0-SNAPSHOT.jar org.apache.hadoop.util.VersionInfo
Hadoop 0.21.0-SNAPSHOT
Subversion http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21
-r 893667
Compiled by hudson on Thu Dec 24 09:34:21 UTC 2009
From source with checksum 337d7f1a109e0ac3c7d2874300ce9b78
and not my one. Also it pulls down the 0.22-SNAPSHOT one as well.
Perhaps I shouldn't be concerned about this but I'm always getting
this error on the tasktracker startup:
2010-01-04 11:55:33,090 ERROR org.apache.hadoop.mapred.TaskTracker:
Can not start task tracker because java.lang.NoSuchMethodError:
org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/
InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Ljava/lang/
Object;
at
org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:601)
at
org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1108)
at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:
2985)
I've fixed that in the past but I can't remember what I did, searching
through mailing lists only points to different jars in the classpath.
There's only one copy of the hadoop*jar files on the classpath. Also
going to the :50030/jobtracker.jsp page gives back a 500 error saying
that it can't find the org.apache.hadoop.mapred.ClusterStatus class
which is in hadoop-mapred-0.21.0-SNAPSHOT.jar
Re: How can I compile mapreduce to use my own version of hadoop-core
0.21-SNAPSHOT?
Posted by Kay Kay <ka...@gmail.com>.
On 1/4/10 12:05 PM, Wilkes, Chris wrote:
> I'm trying to build the 0.21.0-SNAPSHOT packages of common, hdfs, and
> mapreduce and am doing so with "ant binary" (I'm more used to maven so
> I'm not very good at debugging this), I've also done a mvn-install to
> put the jars into the local maven repo.
>
> "ant binary" in seems to pull down the hdfs-core.jar file from the
> Hudson server:
> java -cp
> ./build/hadoop-mapred-0.21.0-SNAPSHOT/lib/hadoop-core-0.21.0-SNAPSHOT.jar
> org.apache.hadoop.util.VersionInfo
> Hadoop 0.21.0-SNAPSHOT
> Subversion
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r
> 893667
> Compiled by hudson on Thu Dec 24 09:34:21 UTC 2009
> From source with checksum 337d7f1a109e0ac3c7d2874300ce9b78
> and not my one. Also it pulls down the 0.22-SNAPSHOT one as well.
>
> Perhaps I shouldn't be concerned about this but I'm always getting
> this error on the tasktracker startup:
>
> 2010-01-04 11:55:33,090 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.lang.NoSuchMethodError:
> org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Ljava/lang/Object;
>
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:601)
> at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1108)
> at
> org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2985)
>
> I've fixed that in the past but I can't remember what I did, searching
> through mailing lists only points to different jars in the classpath.
> There's only one copy of the hadoop*jar files on the classpath. Also
> going to the :50030/jobtracker.jsp page gives back a 500 error saying
> that it can't find the org.apache.hadoop.mapred.ClusterStatus class
> which is in hadoop-mapred-0.21.0-SNAPSHOT.jar
The root cause of it seems to be a .pom misconfiguration across hdfs and
mapreduce . See MAPREDUCE-1352 for more details. Once the build is green
and the jars are republished with the .pom files - this should go away.
Also - a good number of 0.21 mapreduce builds have been failing, that
might be related to HDFS-869 as well.
As the first step -we need to get mapred-0.21 to be green before we can
add more patches to that.
Re: How can I compile mapreduce to use my own version of hadoop-core
0.21-SNAPSHOT?
Posted by Kay Kay <ka...@gmail.com>.
On 1/4/10 12:05 PM, Wilkes, Chris wrote:
> I'm trying to build the 0.21.0-SNAPSHOT packages of common, hdfs, and
> mapreduce and am doing so with "ant binary" (I'm more used to maven so
> I'm not very good at debugging this), I've also done a mvn-install to
> put the jars into the local maven repo.
>
> "ant binary" in seems to pull down the hdfs-core.jar file from the
> Hudson server:
> java -cp
> ./build/hadoop-mapred-0.21.0-SNAPSHOT/lib/hadoop-core-0.21.0-SNAPSHOT.jar
> org.apache.hadoop.util.VersionInfo
> Hadoop 0.21.0-SNAPSHOT
> Subversion
> http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.21 -r
> 893667
> Compiled by hudson on Thu Dec 24 09:34:21 UTC 2009
> From source with checksum 337d7f1a109e0ac3c7d2874300ce9b78
> and not my one. Also it pulls down the 0.22-SNAPSHOT one as well.
>
> Perhaps I shouldn't be concerned about this but I'm always getting
> this error on the tasktracker startup:
>
> 2010-01-04 11:55:33,090 ERROR org.apache.hadoop.mapred.TaskTracker:
> Can not start task tracker because java.lang.NoSuchMethodError:
> org.apache.hadoop.ipc.RPC.waitForProxy(Ljava/lang/Class;JLjava/net/InetSocketAddress;Lorg/apache/hadoop/conf/Configuration;)Ljava/lang/Object;
>
> at
> org.apache.hadoop.mapred.TaskTracker.initialize(TaskTracker.java:601)
> at
> org.apache.hadoop.mapred.TaskTracker.<init>(TaskTracker.java:1108)
> at
> org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:2985)
>
> I've fixed that in the past but I can't remember what I did, searching
> through mailing lists only points to different jars in the classpath.
> There's only one copy of the hadoop*jar files on the classpath. Also
> going to the :50030/jobtracker.jsp page gives back a 500 error saying
> that it can't find the org.apache.hadoop.mapred.ClusterStatus class
> which is in hadoop-mapred-0.21.0-SNAPSHOT.jar
The root cause of it seems to be a .pom misconfiguration across hdfs and
mapreduce . See MAPREDUCE-1352 for more details. Once the build is green
and the jars are republished with the .pom files - this should go away.
Also - a good number of 0.21 mapreduce builds have been failing, that
might be related to HDFS-869 as well.
As the first step -we need to get mapred-0.21 to be green before we can
add more patches to that.