You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Uttam Kumar <uk...@gmail.com> on 2013/11/26 20:56:02 UTC
PIG-12 generating Error msg while running with Hadoop-23
Hi,
I am fairly new to Hadoop MR2 environment. I have Hadoop0.23.1 (single
node) on Linux box and can successfully execute YARN test job, but when I
execute PIG (Ver. 12) to access and process files at HDFS, it generate
error msg as follows. As it was mentioned somewhere to compile the PIG
with Hadoop23, so I ran following command and still getting error msg:
ant clean jar-withouthadoop -Dhadoopversion=23
Could you please advise what I am missing here !!
Note: PIG in local mode works without any issue.
================
*grunt> A = load 'NYSE_dividends' as (exch, symb, dt, div);*
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
*grunt> dump A ;*
2013-11-26 13:20:31,805 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:31,806 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:31,841 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-11-26 13:20:31,913 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune,
DuplicateForEachColumnRewrite, GroupByConstParallelSetter,
ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2013-11-26 13:20:32,159 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,266 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2013-11-26 13:20:32,319 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,498 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2013-11-26 13:20:32,506 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1088179330840878220.jar
2013-11-26 13:20:35,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1088179330840878220.jar created
2013-11-26 13:20:35,473 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jar is deprecated. Instead, use mapreduce.job.jar
2013-11-26 13:20:35,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-11-26 13:20:35,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2013-11-26 13:20:35,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-11-26 13:20:35,514 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2013-11-26 13:20:35,973 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area
null/hadoop/.staging/job_1385489868206_0001
*2013-11-26 13:20:35,975 [JobControl] ERROR
org.apache.pig.backend.hadoop23.PigJobControl - Error while trying to run
jobs.java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException*
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
*Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)*
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_1385489868206_0001
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases A
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: A[1,4] C: R:
2013-11-26 13:20:35,979 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2013-11-26 13:20:35,983 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
stop immediately on failure.
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_1385489868206_0001 has failed! Stop running all dependent jobs
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2013-11-26 13:20:36,104 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Exception thrown by remote
end.
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: Trying
to get information for an absent application application_1385489868206_0001
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:150)
at $Proxy7.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClientImpl.java:116)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:343)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:142)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:288)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:365)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:483)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:627)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:625)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:625)
at
org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:679)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:673)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:428)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
2013-11-26 13:20:36,105 [main] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Trying to get information for an absent application
application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR: Trying to get
information for an absent application application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.23.1 0.12.1-SNAPSHOT hadoop 2013-11-26 13:20:32 2013-11-26
13:20:36 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1385489868206_0001 A MAP_ONLY Message: Unexpected System
Error Occured: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029,
Input(s):
Failed to read data from
"hdfs://sasserver05:54310/user/hadoop/NYSE_dividends"
Output(s):
Failed to produce result in
"hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1385489868206_0001
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2013-11-26 13:20:36,112 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias A. Backend error : Trying to
get information for an absent application application_1385489868206_0001
Details at logfile: /home/hadoop/pig_1385489876211.log
grunt>
=============
--
Regards,
Uttam