You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Nishant Neeraj <ni...@gmail.com> on 2012/10/26 07:14:55 UTC
Pig 0.10.0 and Hadoop 2.0.2: String JobControl.addJob(Job)
I am new to Hadoop and Pig.
I am trying to get Pig 0.10.0 [1] and Hadoop 2.0.2 [2] working together. I
get
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
error, see complete stack trace in footnote [3]. How can I fix this?
Here is what I am doing:
grunt> cd hdfs:///data/
grunt> cat test.data
2012-10-26 10:38:15,110 [main] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
0 Wed Oct 24 14:35:21 IST 2012 name
1 Wed Oct 24 14:35:26 IST 2012 name
2 Wed Oct 24 14:35:31 IST 2012 name
3 Wed Oct 24 14:35:36 IST 2012 name
4 Wed Oct 24 14:35:41 IST 2012 name
5 Wed Oct 24 14:35:46 IST 2012 name
6 Wed Oct 24 14:35:51 IST 2012 name
7 Wed Oct 24 14:35:56 IST 2012 name
8 Wed Oct 24 14:36:01 IST 2012 name
9 Wed Oct 24 14:36:06 IST 2012 name
grunt> data = LOAD 'test.data' ;
2012-10-26 10:38:29,389 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use
mapreduce.jobtracker.maxtasks.perjob
[ -- snip --]
grunt> name = FILTER data BY $0 MATCHES '.*name.*' ;
grunt> DUMP name
[ -- snip -- ]
2012-10-26 10:39:23,299 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 2998: Unhandled internal error.
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
----
[1]
Apache Pig version 0.10.0 (r1328203)
compiled Apr 19 2012, 22:54:12
[2]
Hadoop 2.0.2-alpha
Subversion
https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.0.2-alpha/hadoop-common-project/hadoop-common-r
1392682
Compiled by hortonmu on Tue Oct 2 00:44:10 UTC 2012
>From source with checksum efbdb59af73bfc103f1945d65dbf3071
[3]
Pig Stack Trace
---------------
ERROR 2998: Unhandled internal error.
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
java.lang.NoSuchMethodError:
org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:261)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
at org.apache.pig.PigServer.storeEx(PigServer.java:952)
at org.apache.pig.PigServer.store(PigServer.java:919)
at org.apache.pig.PigServer.openIterator(PigServer.java:832)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:490)
at org.apache.pig.Main.main(Main.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Re: Pig 0.10.0 and Hadoop 2.0.2: String JobControl.addJob(Job)
Posted by Nishant Neeraj <ni...@gmail.com>.
Hi Cheolsoo,
Yep, that fixed the problem. Thank you very much for quick response.
- Nishant
Re: Pig 0.10.0 and Hadoop 2.0.2: String JobControl.addJob(Job)
Posted by Cheolsoo Park <ch...@cloudera.com>.
Hi Nishant,
That's because the default pig jar file are compiled against hadoop-1.0.x,
but you're running it against hadoop-2.0.x. You need to re-build the pig
jar file.
Please do the following: (I am assuming that you're the installed hadoop
and pig-withouthadoop.jar.)
1) cd <pig home directory> (assuming that you have source code as well as
binaries.)
2) mv pig-0.10.0-withouthadoop.jar pig-0.10.0-withouthadoop.jar.bk
3) ant clean jar-withouthadoop -Dhadoopversion=23
You will see a new pig-0.10.0-withouthadoop.jar is built in the current
directory. Now you will no longer the error.
Thanks,
Cheolsoo
On Thu, Oct 25, 2012 at 10:14 PM, Nishant Neeraj <
nishant.has.a.question@gmail.com> wrote:
> I am new to Hadoop and Pig.
> I am trying to get Pig 0.10.0 [1] and Hadoop 2.0.2 [2] working together. I
> get
>
> java.lang.NoSuchMethodError:
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
> error, see complete stack trace in footnote [3]. How can I fix this?
>
> Here is what I am doing:
>
> grunt> cd hdfs:///data/
> grunt> cat test.data
> 2012-10-26 10:38:15,110 [main] WARN
> org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 0 Wed Oct 24 14:35:21 IST 2012 name
> 1 Wed Oct 24 14:35:26 IST 2012 name
> 2 Wed Oct 24 14:35:31 IST 2012 name
> 3 Wed Oct 24 14:35:36 IST 2012 name
> 4 Wed Oct 24 14:35:41 IST 2012 name
> 5 Wed Oct 24 14:35:46 IST 2012 name
> 6 Wed Oct 24 14:35:51 IST 2012 name
> 7 Wed Oct 24 14:35:56 IST 2012 name
> 8 Wed Oct 24 14:36:01 IST 2012 name
> 9 Wed Oct 24 14:36:06 IST 2012 name
>
> grunt> data = LOAD 'test.data' ;
> 2012-10-26 10:38:29,389 [main] WARN org.apache.hadoop.conf.Configuration -
> mapred.jobtracker.maxtasks.per.job is deprecated. Instead, use
> mapreduce.jobtracker.maxtasks.perjob
> [ -- snip --]
>
> grunt> name = FILTER data BY $0 MATCHES '.*name.*' ;
> grunt> DUMP name
> [ -- snip -- ]
> 2012-10-26 10:39:23,299 [main] ERROR org.apache.pig.tools.grunt.Grunt -
> ERROR 2998: Unhandled internal error.
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
>
> ----
> [1]
> Apache Pig version 0.10.0 (r1328203)
> compiled Apr 19 2012, 22:54:12
>
> [2]
> Hadoop 2.0.2-alpha
> Subversion
>
> https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2.0.2-alpha/hadoop-common-project/hadoop-common-r
> 1392682
> Compiled by hortonmu on Tue Oct 2 00:44:10 UTC 2012
> From source with checksum efbdb59af73bfc103f1945d65dbf3071
>
> [3]
> Pig Stack Trace
> ---------------
> ERROR 2998: Unhandled internal error.
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
>
> java.lang.NoSuchMethodError:
>
> org.apache.hadoop.mapred.jobcontrol.JobControl.addJob(Lorg/apache/hadoop/mapred/jobcontrol/Job;)Ljava/lang/String;
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler.compile(JobControlCompiler.java:261)
> at
>
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:180)
> at org.apache.pig.PigServer.launchPlan(PigServer.java:1270)
> at
> org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1255)
> at org.apache.pig.PigServer.storeEx(PigServer.java:952)
> at org.apache.pig.PigServer.store(PigServer.java:919)
> at org.apache.pig.PigServer.openIterator(PigServer.java:832)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:682)
> at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:189)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
> at org.apache.pig.Main.run(Main.java:490)
> at org.apache.pig.Main.main(Main.java:111)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
>