You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Juan Martin Pampliega <jp...@gmail.com> on 2013/11/26 20:07:03 UTC
Re: Support for Hadoop 2.2
Hi Claudio,
It would be nice to know which were the settings that you had to tune to
get this. I am having a similar issue with some jobs that I am running.
Thanks,
Juan.
On Wed, Oct 30, 2013 at 7:40 PM, Claudio Romo Otto <
claudio.romo@namictech.com> wrote:
> Jarcec, finally I got solved this problem by learning more on hadoop 2
> (lot of reading), and then tuning some settings to let the work move from
> the SCHEDULED state. With this said, the last problem was only concerning
> on hadoop.
>
> Thanks for your support!
>
> El 30/10/13 18:03, Jarek Jarcec Cecho escribió:
>
> Hi Claudio,
>> it's hard to guess from the limited information. I would suggest to take
>> a look into logs to see what is happening.
>>
>> One guess though - You've mentioned that the task was "running" for 30
>> minutes, but it still seems to be in SCHEDULED time - are your node
>> managers correctly running?
>>
>> Jarcec
>>
>> On Fri, Oct 25, 2013 at 04:10:12PM -0300, Claudio Romo Otto wrote:
>>
>>> You got it!
>>>
>>> The solution was to compile with -Dhadoopversion=23 option. After
>>> your message I tried another test removing Cassandra from the chain
>>> and Pig sent successfully the job to hadoop.
>>>
>>> BUT! the problem changed, now the Map task remains forever stuck on
>>> Hadoop (30 minutes waiting, no other jobs running):
>>>
>>> Task
>>>
>>> Progress
>>>
>>> State
>>>
>>> Start Time
>>>
>>> Finish Time
>>>
>>> Elapsed Time
>>> task_1382631533263_0012_m_000000 <http://topgps-test-3.
>>> dnsalias.com:8088/proxy/application_1382631533263_
>>> 0012/mapreduce/task/task_1382631533263_0012_m_000000>
>>>
>>> SCHEDULED Fri, 25 Oct 2013 18:18:32 GMT N/A 0sec
>>>
>>>
>>>
>>> Attempt
>>>
>>> Progress
>>>
>>> State
>>>
>>> Node
>>>
>>> Logs
>>>
>>> Started
>>>
>>> Finished
>>>
>>> Elapsed
>>>
>>> Note
>>> attempt_1382631533263_0012_m_000000_0 0,00 STARTING N/A
>>> N/A N/A
>>> N/A 0sec
>>>
>>>
>>> Don't know if this is a Hadoop problem or Pig, what do you think?
>>>
>>>
>>> El 25/10/13 13:11, Jarek Jarcec Cecho escribió:
>>>
>>>> It seems that Pig was correctly compiled against Hadoop 23, but the
>>>> Cassandra piece was not, check out the where the exception is coming from:
>>>>
>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>
>>>> So, I would say that you also need to get Hadoop 2 compatible Cassandra
>>>> connector first.
>>>>
>>>> Jarcec
>>>>
>>>> On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
>>>>
>>>>> After change from hadoop20 to hadoop23 the warning dissapeared but I
>>>>> got the same exception (Found interface
>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>>>>>
>>>>> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
>>>>> compiled by me, no other product nor configuration, just two
>>>>> servers, one master with ResourceManager and NameNode, one slave
>>>>> with DataNode and NodeManager.
>>>>>
>>>>> I can't understand why over this fresh cluster Pig 0.12 fails. Here
>>>>> is the new trace:
>>>>>
>>>>> 2013-10-24 16:10:52,351 [JobControl] ERROR
>>>>> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
>>>>> to run jobs.
>>>>> java.lang.RuntimeException: java.lang.reflect.
>>>>> InvocationTargetException
>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>> PigJobControl.java:130)
>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.run(
>>>>> PigJobControl.java:191)
>>>>> at java.lang.Thread.run(Thread.java:724)
>>>>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
>>>>> MapReduceLauncher$1.run(MapReduceLauncher.java:257)
>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>> NativeMethodAccessorImpl.java:57)
>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>> PigJobControl.java:128)
>>>>> ... 3 more
>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
>>>>> JobSubmitter.java:491)
>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
>>>>> JobSubmitter.java:508)
>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
>>>>> JobSubmitter.java:392)
>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>>>>> UserGroupInformation.java:1491)
>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>> at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
>>>>> submit(ControlledJob.java:335)
>>>>> ... 8 more
>>>>>
>>>>>
>>>>> El 24/10/13 21:33, Prashant Kommireddi escribió:
>>>>>
>>>>>> Yes it does. You need to recompile Pig for hadoop 2
>>>>>>
>>>>>> ant clean jar-withouthadoop -Dhadoopversion=23
>>>>>>
>>>>>>
>>>>>> On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
>>>>>> claudio.romo@namictech.com> wrote:
>>>>>>
>>>>>> Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I
>>>>>>> get an
>>>>>>> error even with simple operations like
>>>>>>>
>>>>>>> data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
>>>>>>> dump data;
>>>>>>>
>>>>>>> I only got a warning first and then and exception:
>>>>>>>
>>>>>>> 2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**
>>>>>>> hadoop20.PigJobControl
>>>>>>> - falling back to default JobControl (not using hadoop 0.20 ?)
>>>>>>> java.lang.**NoSuchFieldException: runnerState
>>>>>>> at java.lang.Class.**getDeclaredField(Class.java:**1938)
>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.<**
>>>>>>> clinit>(PigJobControl.java:51)
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.shims.**
>>>>>>> HadoopShims.newJobControl(**HadoopShims.java:97)
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>> mapReduceLayer.**
>>>>>>> JobControlCompiler.compile(**JobControlCompiler.java:285)
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>> mapReduceLayer.**
>>>>>>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:177)
>>>>>>> at org.apache.pig.PigServer.**launchPlan(PigServer.java:**1264)
>>>>>>> at org.apache.pig.PigServer.**executeCompiledLogicalPlan(**
>>>>>>> PigServer.java:1249)
>>>>>>> at org.apache.pig.PigServer.**storeEx(PigServer.java:931)
>>>>>>> at org.apache.pig.PigServer.**store(PigServer.java:898)
>>>>>>> at org.apache.pig.PigServer.**openIterator(PigServer.java:**
>>>>>>> 811)
>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>> GruntParser.java:696)
>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>> PigScriptParser.parse(**
>>>>>>> PigScriptParser.java:320)
>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>>>>>> GruntParser.java:194)
>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>>>>>> GruntParser.java:170)
>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>> Method)
>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>
>>>>>>> ------------------------------**--------
>>>>>>>
>>>>>>> Backend error message during job submission
>>>>>>> ------------------------------**-------------
>>>>>>> Unexpected System Error Occured: java.lang.**
>>>>>>> IncompatibleClassChangeError:
>>>>>>> Found interface org.apache.hadoop.mapreduce.**JobContext, but class
>>>>>>> was
>>>>>>> expected
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>> mapReduceLayer.**PigOutputFormat.**setupUdfEnvAndStores(**
>>>>>>> PigOutputFormat.java:225)
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>> mapReduceLayer.**PigOutputFormat.**checkOutputSpecs(**
>>>>>>> PigOutputFormat.java:186)
>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.checkSpecs(**
>>>>>>> JobSubmitter.java:456)
>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.**
>>>>>>> submitJobInternal(
>>>>>>> **JobSubmitter.java:342)
>>>>>>> at org.apache.hadoop.mapreduce.**Job$10.run(Job.java:1268)
>>>>>>> at org.apache.hadoop.mapreduce.**Job$10.run(Job.java:1265)
>>>>>>> at java.security.**AccessController.doPrivileged(**Native
>>>>>>> Method)
>>>>>>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>>>>>>> at org.apache.hadoop.security.**
>>>>>>> UserGroupInformation.doAs(**
>>>>>>> UserGroupInformation.java:**1491)
>>>>>>> at org.apache.hadoop.mapreduce.**Job.submit(Job.java:1265)
>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>> lib.jobcontrol.ControlledJob.**
>>>>>>> submit(ControlledJob.java:335)
>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>> lib.jobcontrol.JobControl.run(**
>>>>>>> JobControl.java:240)
>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.run(**
>>>>>>> PigJobControl.java:121)
>>>>>>> at java.lang.Thread.run(Thread.**java:724)
>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>> mapReduceLayer.**MapReduceLauncher$1.run(**
>>>>>>> MapReduceLauncher.java:257)
>>>>>>>
>>>>>>> Pig Stack Trace
>>>>>>> ---------------
>>>>>>> ERROR 1066: Unable to open iterator for alias data
>>>>>>>
>>>>>>> org.apache.pig.impl.**logicalLayer.**FrontendException: ERROR 1066:
>>>>>>> Unable to open iterator for alias data
>>>>>>> at org.apache.pig.PigServer.**
>>>>>>> openIterator(PigServer.java:**836)
>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>> GruntParser.java:696)
>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>> PigScriptParser.parse(**PigScriptParser.java:320)
>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>> GruntParser.parseStopOnError(**
>>>>>>> GruntParser.java:194)
>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>> GruntParser.parseStopOnError(**
>>>>>>> GruntParser.java:170)
>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>> Method)
>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>> Caused by: java.io.IOException: Job terminated with anomalous status
>>>>>>> FAILED
>>>>>>> at org.apache.pig.PigServer.**
>>>>>>> openIterator(PigServer.java:**828)
>>>>>>> ... 12 more
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: Support for Hadoop 2.2
Posted by Uttam Kumar <uk...@gmail.com>.
Hi Folks,
I am fairly new to Hadoop MR2 environment. I have Hadoop0.23.1 (single
node) on Linux box and can successfully execute YARN test job, but when I
execute PIG (Ver. 12) to access and process files at HDFS, generate error
msg as follows. I recompiled PIG with Hadoop23, but still no luck.
ant clean jar-withouthadoop -Dhadoopversion=23
Can someone please help and advise what I am missing here !!
Note: PIG in local mode works without any issue.
================
*grunt> A = load 'NYSE_dividends' as (exch, symb, dt, div);*
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.
Configuration - fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,151 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:23,323 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
*grunt> dump A ;*
2013-11-26 13:20:31,805 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:31,806 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:31,841 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2013-11-26 13:20:31,913 [main] INFO
org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer -
{RULES_ENABLED=[AddForEach, ColumnMapKeyPrune,
DuplicateForEachColumnRewrite, GroupByConstParallelSetter,
ImplicitSplitInserter, LimitOptimizer, LoadTypeCastInserter, MergeFilter,
MergeForEach, NewPartitionFilterOptimizer, PartitionFilterOptimizer,
PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter],
RULES_DISABLED=[FilterLogicExpressionSimplifier]}
2013-11-26 13:20:32,159 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,160 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,266 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler -
File concatenation threshold: 100 optimistic? false
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,304 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,318 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2013-11-26 13:20:32,319 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
fs.default.name is deprecated. Instead, use fs.defaultFS
2013-11-26 13:20:32,322 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2013-11-26 13:20:32,498 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are added
to the job
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use
mapreduce.reduce.markreset.buffer.percent
2013-11-26 13:20:32,504 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.output.compress is deprecated. Instead, use
mapreduce.output.fileoutputformat.compress
2013-11-26 13:20:32,506 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- creating jar file Job1088179330840878220.jar
2013-11-26 13:20:35,473 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- jar file Job1088179330840878220.jar created
2013-11-26 13:20:35,473 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.jar is deprecated. Instead, use mapreduce.job.jar
2013-11-26 13:20:35,488 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2013-11-26 13:20:35,492 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Map only job, skipping reducer estimation
2013-11-26 13:20:35,513 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2013-11-26 13:20:35,514 [main] WARN org.apache.hadoop.conf.Configuration -
mapred.job.tracker.http.address is deprecated. Instead, use
mapreduce.jobtracker.http.address
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths
to process : 1
2013-11-26 13:20:35,954 [JobControl] INFO
org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input
paths to process : 1
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
2013-11-26 13:20:35,971 [JobControl] WARN
org.apache.hadoop.io.compress.snappy.LoadSnappy - Snappy native library not
loaded
2013-11-26 13:20:35,973 [JobControl] INFO
org.apache.hadoop.mapreduce.JobSubmitter - Cleaning up the staging area
null/hadoop/.staging/job_1385489868206_0001
*2013-11-26 13:20:35,975 [JobControl] ERROR
org.apache.pig.backend.hadoop23.PigJobControl - Error while trying to run
jobs.java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException*
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
*Caused by: java.lang.reflect.InvocationTargetException at
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)*
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- HadoopJobId: job_1385489868206_0001
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Processing aliases A
2013-11-26 13:20:35,976 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- detailed locations: M: A[1,4] C: R:
2013-11-26 13:20:35,979 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2013-11-26 13:20:35,983 [main] WARN
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to
stop immediately on failure.
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- job job_1385489868206_0001 has failed! Stop running all dependent jobs
2013-11-26 13:20:35,983 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2013-11-26 13:20:36,104 [main] WARN
org.apache.hadoop.mapred.ClientServiceDelegate - Exception thrown by remote
end.
RemoteTrace:
at LocalTrace:
org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl: Trying
to get information for an absent application application_1385489868206_0001
at
org.apache.hadoop.yarn.ipc.ProtoOverHadoopRpcEngine$Invoker.invoke(ProtoOverHadoopRpcEngine.java:150)
at $Proxy7.getApplicationReport(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getApplicationReport(ClientRMProtocolPBClientImpl.java:116)
at
org.apache.hadoop.mapred.ResourceMgrDelegate.getApplicationReport(ResourceMgrDelegate.java:343)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getProxy(ClientServiceDelegate.java:142)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:288)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:365)
at
org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:483)
at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:186)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:627)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:625)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at
org.apache.hadoop.mapred.JobClient.getJobUsingCluster(JobClient.java:625)
at
org.apache.hadoop.mapred.JobClient.getTaskReports(JobClient.java:679)
at
org.apache.hadoop.mapred.JobClient.getMapTaskReports(JobClient.java:673)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.Launcher.getStats(Launcher.java:150)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:428)
at org.apache.pig.PigServer.launchPlan(PigServer.java:1322)
at
org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1307)
at org.apache.pig.PigServer.storeEx(PigServer.java:978)
at org.apache.pig.PigServer.store(PigServer.java:942)
at org.apache.pig.PigServer.openIterator(PigServer.java:855)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:200)
2013-11-26 13:20:36,105 [main] ERROR
org.apache.hadoop.security.UserGroupInformation -
PriviledgedActionException as:hadoop (auth:SIMPLE)
cause:org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl:
Trying to get information for an absent application
application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.SimplePigStats - ERROR: Trying to get
information for an absent application application_1385489868206_0001
2013-11-26 13:20:36,105 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.tools.pigstats.SimplePigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt
Features
0.23.1 0.12.1-SNAPSHOT hadoop 2013-11-26 13:20:32 2013-11-26
13:20:36 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
job_1385489868206_0001 A MAP_ONLY Message: Unexpected System
Error Occured: java.lang.RuntimeException:
java.lang.reflect.InvocationTargetException
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:130)
at
org.apache.pig.backend.hadoop23.PigJobControl.run(PigJobControl.java:191)
at java.lang.Thread.run(Thread.java:619)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher$1.run(MapReduceLauncher.java:270)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.pig.backend.hadoop23.PigJobControl.submit(PigJobControl.java:128)
... 3 more
Caused by: java.lang.NoSuchMethodError:
org.apache.hadoop.fs.FileSystem.getDefaultBlockSize(Lorg/apache/hadoop/fs/Path;)J
at
org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getDefaultBlockSize(HadoopShims.java:108)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:277)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:451)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:468)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:360)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1221)
at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1218)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1218)
at
org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.submit(ControlledJob.java:336)
... 8 more
hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029,
Input(s):
Failed to read data from
"hdfs://sasserver05:54310/user/hadoop/NYSE_dividends"
Output(s):
Failed to produce result in
"hdfs://sasserver05:54310/tmp/temp601167191/tmp-1605231029"
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
job_1385489868206_0001
2013-11-26 13:20:36,107 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2013-11-26 13:20:36,112 [main] ERROR org.apache.pig.tools.grunt.Grunt -
ERROR 1066: Unable to open iterator for alias A. Backend error : Trying to
get information for an absent application application_1385489868206_0001
Details at logfile: /home/hadoop/pig_1385489876211.log
grunt>
=============
On Tue, Nov 26, 2013 at 3:32 PM, Claudio Romo Otto <
claudio.romo@namictech.com> wrote:
> Hi Juan,
>
> In a nutshell, you must pay attention to memory settings inside
> mapred-site.xml, yarn-site.xml, hadoop-env.sh and yarn-env.sh, so you have
> to design a memory distribution strategy according to your performance
> requirements. In this way you will have, among other things, enough memory
> for the Scheduler.
>
> Remember to reserve at least 600 - 800 mb for the operative system to
> avoid OOM errors.
>
> Best regards
> El 26/11/13 16:07, Juan Martin Pampliega escribió:
>
>> Hi Claudio,
>>
>> It would be nice to know which were the settings that you had to tune to
>> get this. I am having a similar issue with some jobs that I am running.
>> Thanks,
>> Juan.
>>
>>
>> On Wed, Oct 30, 2013 at 7:40 PM, Claudio Romo Otto <
>> claudio.romo@namictech.com> wrote:
>>
>> Jarcec, finally I got solved this problem by learning more on hadoop 2
>>> (lot of reading), and then tuning some settings to let the work move from
>>> the SCHEDULED state. With this said, the last problem was only concerning
>>> on hadoop.
>>>
>>> Thanks for your support!
>>>
>>> El 30/10/13 18:03, Jarek Jarcec Cecho escribió:
>>>
>>> Hi Claudio,
>>>
>>>> it's hard to guess from the limited information. I would suggest to take
>>>> a look into logs to see what is happening.
>>>>
>>>> One guess though - You've mentioned that the task was "running" for 30
>>>> minutes, but it still seems to be in SCHEDULED time - are your node
>>>> managers correctly running?
>>>>
>>>> Jarcec
>>>>
>>>> On Fri, Oct 25, 2013 at 04:10:12PM -0300, Claudio Romo Otto wrote:
>>>>
>>>> You got it!
>>>>>
>>>>> The solution was to compile with -Dhadoopversion=23 option. After
>>>>> your message I tried another test removing Cassandra from the chain
>>>>> and Pig sent successfully the job to hadoop.
>>>>>
>>>>> BUT! the problem changed, now the Map task remains forever stuck on
>>>>> Hadoop (30 minutes waiting, no other jobs running):
>>>>>
>>>>> Task
>>>>>
>>>>> Progress
>>>>>
>>>>> State
>>>>>
>>>>> Start Time
>>>>>
>>>>> Finish Time
>>>>>
>>>>> Elapsed Time
>>>>> task_1382631533263_0012_m_000000 <http://topgps-test-3.
>>>>> dnsalias.com:8088/proxy/application_1382631533263_
>>>>> 0012/mapreduce/task/task_1382631533263_0012_m_000000>
>>>>>
>>>>> SCHEDULED Fri, 25 Oct 2013 18:18:32 GMT N/A 0sec
>>>>>
>>>>>
>>>>>
>>>>> Attempt
>>>>>
>>>>> Progress
>>>>>
>>>>> State
>>>>>
>>>>> Node
>>>>>
>>>>> Logs
>>>>>
>>>>> Started
>>>>>
>>>>> Finished
>>>>>
>>>>> Elapsed
>>>>>
>>>>> Note
>>>>> attempt_1382631533263_0012_m_000000_0 0,00 STARTING N/A
>>>>> N/A N/A
>>>>> N/A 0sec
>>>>>
>>>>>
>>>>> Don't know if this is a Hadoop problem or Pig, what do you think?
>>>>>
>>>>>
>>>>> El 25/10/13 13:11, Jarek Jarcec Cecho escribió:
>>>>>
>>>>> It seems that Pig was correctly compiled against Hadoop 23, but the
>>>>>> Cassandra piece was not, check out the where the exception is coming
>>>>>> from:
>>>>>>
>>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>>
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>>>
>>>>>>> So, I would say that you also need to get Hadoop 2 compatible
>>>>>> Cassandra
>>>>>> connector first.
>>>>>>
>>>>>> Jarcec
>>>>>>
>>>>>> On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
>>>>>>
>>>>>> After change from hadoop20 to hadoop23 the warning dissapeared but I
>>>>>>> got the same exception (Found interface
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>>>>>>>
>>>>>>> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
>>>>>>> compiled by me, no other product nor configuration, just two
>>>>>>> servers, one master with ResourceManager and NameNode, one slave
>>>>>>> with DataNode and NodeManager.
>>>>>>>
>>>>>>> I can't understand why over this fresh cluster Pig 0.12 fails. Here
>>>>>>> is the new trace:
>>>>>>>
>>>>>>> 2013-10-24 16:10:52,351 [JobControl] ERROR
>>>>>>> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
>>>>>>> to run jobs.
>>>>>>> java.lang.RuntimeException: java.lang.reflect.
>>>>>>> InvocationTargetException
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>>> PigJobControl.java:130)
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.run(
>>>>>>> PigJobControl.java:191)
>>>>>>> at java.lang.Thread.run(Thread.java:724)
>>>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>>>> mapReduceLayer.
>>>>>>> MapReduceLauncher$1.run(MapReduceLauncher.java:257)
>>>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>> NativeMethodAccessorImpl.java:57)
>>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>>> PigJobControl.java:128)
>>>>>>> ... 3 more
>>>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>>>> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
>>>>>>> JobSubmitter.java:491)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
>>>>>>> JobSubmitter.java:508)
>>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
>>>>>>> JobSubmitter.java:392)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>>>>>>> UserGroupInformation.java:1491)
>>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>>> at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
>>>>>>> submit(ControlledJob.java:335)
>>>>>>> ... 8 more
>>>>>>>
>>>>>>>
>>>>>>> El 24/10/13 21:33, Prashant Kommireddi escribió:
>>>>>>>
>>>>>>> Yes it does. You need to recompile Pig for hadoop 2
>>>>>>>>
>>>>>>>> ant clean jar-withouthadoop -Dhadoopversion=23
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
>>>>>>>> claudio.romo@namictech.com> wrote:
>>>>>>>>
>>>>>>>> Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I
>>>>>>>>
>>>>>>>>> get an
>>>>>>>>> error even with simple operations like
>>>>>>>>>
>>>>>>>>> data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
>>>>>>>>> dump data;
>>>>>>>>>
>>>>>>>>> I only got a warning first and then and exception:
>>>>>>>>>
>>>>>>>>> 2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**
>>>>>>>>> hadoop20.PigJobControl
>>>>>>>>> - falling back to default JobControl (not using hadoop 0.20 ?)
>>>>>>>>> java.lang.**NoSuchFieldException: runnerState
>>>>>>>>> at java.lang.Class.**getDeclaredField(Class.java:**1938)
>>>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.<**
>>>>>>>>> clinit>(PigJobControl.java:51)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.shims.**
>>>>>>>>> HadoopShims.newJobControl(**HadoopShims.java:97)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**
>>>>>>>>> JobControlCompiler.compile(**JobControlCompiler.java:285)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**
>>>>>>>>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:177)
>>>>>>>>> at org.apache.pig.PigServer.**launchPlan(PigServer.java:**
>>>>>>>>> 1264)
>>>>>>>>> at org.apache.pig.PigServer.**executeCompiledLogicalPlan(**
>>>>>>>>> PigServer.java:1249)
>>>>>>>>> at org.apache.pig.PigServer.**storeEx(PigServer.java:931)
>>>>>>>>> at org.apache.pig.PigServer.**store(PigServer.java:898)
>>>>>>>>> at org.apache.pig.PigServer.**openIterator(PigServer.java:**
>>>>>>>>> 811)
>>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>>>> GruntParser.java:696)
>>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>>> PigScriptParser.parse(**
>>>>>>>>> PigScriptParser.java:320)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:194)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:170)
>>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>>>> Method)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>>>
>>>>>>>>> ------------------------------**--------
>>>>>>>>>
>>>>>>>>> Backend error message during job submission
>>>>>>>>> ------------------------------**-------------
>>>>>>>>> Unexpected System Error Occured: java.lang.**
>>>>>>>>> IncompatibleClassChangeError:
>>>>>>>>> Found interface org.apache.hadoop.mapreduce.**JobContext, but
>>>>>>>>> class
>>>>>>>>> was
>>>>>>>>> expected
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**PigOutputFormat.**setupUdfEnvAndStores(**
>>>>>>>>> PigOutputFormat.java:225)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**PigOutputFormat.**checkOutputSpecs(**
>>>>>>>>> PigOutputFormat.java:186)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> JobSubmitter.checkSpecs(**
>>>>>>>>> JobSubmitter.java:456)
>>>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.**
>>>>>>>>> submitJobInternal(
>>>>>>>>> **JobSubmitter.java:342)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job$10.run(Job.java:1268)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job$10.run(Job.java:1265)
>>>>>>>>> at java.security.**AccessController.doPrivileged(
>>>>>>>>> **Native
>>>>>>>>> Method)
>>>>>>>>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>>>>>>>>> at org.apache.hadoop.security.**
>>>>>>>>> UserGroupInformation.doAs(**
>>>>>>>>> UserGroupInformation.java:**1491)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> Job.submit(Job.java:1265)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> lib.jobcontrol.ControlledJob.**
>>>>>>>>> submit(ControlledJob.java:335)
>>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>>> lib.jobcontrol.JobControl.run(**
>>>>>>>>> JobControl.java:240)
>>>>>>>>> at org.apache.pig.backend.**
>>>>>>>>> hadoop20.PigJobControl.run(**
>>>>>>>>> PigJobControl.java:121)
>>>>>>>>> at java.lang.Thread.run(Thread.**java:724)
>>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>>> mapReduceLayer.**MapReduceLauncher$1.run(**
>>>>>>>>> MapReduceLauncher.java:257)
>>>>>>>>>
>>>>>>>>> Pig Stack Trace
>>>>>>>>> ---------------
>>>>>>>>> ERROR 1066: Unable to open iterator for alias data
>>>>>>>>>
>>>>>>>>> org.apache.pig.impl.**logicalLayer.**FrontendException: ERROR
>>>>>>>>> 1066:
>>>>>>>>> Unable to open iterator for alias data
>>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>>> openIterator(PigServer.java:**836)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.processDump(**
>>>>>>>>> GruntParser.java:696)
>>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>>> PigScriptParser.parse(**PigScriptParser.java:320)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:194)
>>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>>> GruntParser.java:170)
>>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**
>>>>>>>>> invoke0(Native
>>>>>>>>> Method)
>>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.*
>>>>>>>>> *invoke(**
>>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>>> Caused by: java.io.IOException: Job terminated with anomalous
>>>>>>>>> status
>>>>>>>>> FAILED
>>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>>> openIterator(PigServer.java:**828)
>>>>>>>>> ... 12 more
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>
--
Regards,
Uttam Kumar
Re: Support for Hadoop 2.2
Posted by Claudio Romo Otto <cl...@namictech.com>.
Hi Juan,
In a nutshell, you must pay attention to memory settings inside
mapred-site.xml, yarn-site.xml, hadoop-env.sh and yarn-env.sh, so you
have to design a memory distribution strategy according to your
performance requirements. In this way you will have, among other things,
enough memory for the Scheduler.
Remember to reserve at least 600 - 800 mb for the operative system to
avoid OOM errors.
Best regards
El 26/11/13 16:07, Juan Martin Pampliega escribió:
> Hi Claudio,
>
> It would be nice to know which were the settings that you had to tune to
> get this. I am having a similar issue with some jobs that I am running.
> Thanks,
> Juan.
>
>
> On Wed, Oct 30, 2013 at 7:40 PM, Claudio Romo Otto <
> claudio.romo@namictech.com> wrote:
>
>> Jarcec, finally I got solved this problem by learning more on hadoop 2
>> (lot of reading), and then tuning some settings to let the work move from
>> the SCHEDULED state. With this said, the last problem was only concerning
>> on hadoop.
>>
>> Thanks for your support!
>>
>> El 30/10/13 18:03, Jarek Jarcec Cecho escribió:
>>
>> Hi Claudio,
>>> it's hard to guess from the limited information. I would suggest to take
>>> a look into logs to see what is happening.
>>>
>>> One guess though - You've mentioned that the task was "running" for 30
>>> minutes, but it still seems to be in SCHEDULED time - are your node
>>> managers correctly running?
>>>
>>> Jarcec
>>>
>>> On Fri, Oct 25, 2013 at 04:10:12PM -0300, Claudio Romo Otto wrote:
>>>
>>>> You got it!
>>>>
>>>> The solution was to compile with -Dhadoopversion=23 option. After
>>>> your message I tried another test removing Cassandra from the chain
>>>> and Pig sent successfully the job to hadoop.
>>>>
>>>> BUT! the problem changed, now the Map task remains forever stuck on
>>>> Hadoop (30 minutes waiting, no other jobs running):
>>>>
>>>> Task
>>>>
>>>> Progress
>>>>
>>>> State
>>>>
>>>> Start Time
>>>>
>>>> Finish Time
>>>>
>>>> Elapsed Time
>>>> task_1382631533263_0012_m_000000 <http://topgps-test-3.
>>>> dnsalias.com:8088/proxy/application_1382631533263_
>>>> 0012/mapreduce/task/task_1382631533263_0012_m_000000>
>>>>
>>>> SCHEDULED Fri, 25 Oct 2013 18:18:32 GMT N/A 0sec
>>>>
>>>>
>>>>
>>>> Attempt
>>>>
>>>> Progress
>>>>
>>>> State
>>>>
>>>> Node
>>>>
>>>> Logs
>>>>
>>>> Started
>>>>
>>>> Finished
>>>>
>>>> Elapsed
>>>>
>>>> Note
>>>> attempt_1382631533263_0012_m_000000_0 0,00 STARTING N/A
>>>> N/A N/A
>>>> N/A 0sec
>>>>
>>>>
>>>> Don't know if this is a Hadoop problem or Pig, what do you think?
>>>>
>>>>
>>>> El 25/10/13 13:11, Jarek Jarcec Cecho escribió:
>>>>
>>>>> It seems that Pig was correctly compiled against Hadoop 23, but the
>>>>> Cassandra piece was not, check out the where the exception is coming from:
>>>>>
>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>>
>>>>> So, I would say that you also need to get Hadoop 2 compatible Cassandra
>>>>> connector first.
>>>>>
>>>>> Jarcec
>>>>>
>>>>> On Thu, Oct 24, 2013 at 10:34:49PM -0300, Claudio Romo Otto wrote:
>>>>>
>>>>>> After change from hadoop20 to hadoop23 the warning dissapeared but I
>>>>>> got the same exception (Found interface
>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected)
>>>>>>
>>>>>> I have tried over a fresh install: hadoop 2.2.0 and pig 0.12.1
>>>>>> compiled by me, no other product nor configuration, just two
>>>>>> servers, one master with ResourceManager and NameNode, one slave
>>>>>> with DataNode and NodeManager.
>>>>>>
>>>>>> I can't understand why over this fresh cluster Pig 0.12 fails. Here
>>>>>> is the new trace:
>>>>>>
>>>>>> 2013-10-24 16:10:52,351 [JobControl] ERROR
>>>>>> org.apache.pig.backend.hadoop23.PigJobControl - Error while trying
>>>>>> to run jobs.
>>>>>> java.lang.RuntimeException: java.lang.reflect.
>>>>>> InvocationTargetException
>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>> PigJobControl.java:130)
>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.run(
>>>>>> PigJobControl.java:191)
>>>>>> at java.lang.Thread.run(Thread.java:724)
>>>>>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.
>>>>>> MapReduceLauncher$1.run(MapReduceLauncher.java:257)
>>>>>> Caused by: java.lang.reflect.InvocationTargetException
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>> NativeMethodAccessorImpl.java:57)
>>>>>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>> DelegatingMethodAccessorImpl.java:43)
>>>>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>>>> at org.apache.pig.backend.hadoop23.PigJobControl.submit(
>>>>>> PigJobControl.java:128)
>>>>>> ... 3 more
>>>>>> Caused by: java.lang.IncompatibleClassChangeError: Found interface
>>>>>> org.apache.hadoop.mapreduce.JobContext, but class was expected
>>>>>> at org.apache.cassandra.hadoop.AbstractColumnFamilyInputForma
>>>>>> t.getSplits(AbstractColumnFamilyInputFormat.java:113)
>>>>>> at org.apache.pig.backend.hadoop.executionengine.
>>>>>> mapReduceLayer.PigInputFormat.getSplits(PigInputFormat.java:274)
>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(
>>>>>> JobSubmitter.java:491)
>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(
>>>>>> JobSubmitter.java:508)
>>>>>> at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(
>>>>>> JobSubmitter.java:392)
>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1268)
>>>>>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1265)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:415)
>>>>>> at org.apache.hadoop.security.UserGroupInformation.doAs(
>>>>>> UserGroupInformation.java:1491)
>>>>>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1265)
>>>>>> at org.apache.hadoop.mapreduce.lib.jobcontrol.ControlledJob.
>>>>>> submit(ControlledJob.java:335)
>>>>>> ... 8 more
>>>>>>
>>>>>>
>>>>>> El 24/10/13 21:33, Prashant Kommireddi escribió:
>>>>>>
>>>>>>> Yes it does. You need to recompile Pig for hadoop 2
>>>>>>>
>>>>>>> ant clean jar-withouthadoop -Dhadoopversion=23
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Oct 24, 2013 at 5:37 AM, Claudio Romo Otto <
>>>>>>> claudio.romo@namictech.com> wrote:
>>>>>>>
>>>>>>> Does Pig support Hadoop 2.2? When I try Pig 0.12 and Hadoop 2.2 I
>>>>>>>> get an
>>>>>>>> error even with simple operations like
>>>>>>>>
>>>>>>>> data = LOAD 'cql://keyspace1/testcf?' USING CqlStorage();
>>>>>>>> dump data;
>>>>>>>>
>>>>>>>> I only got a warning first and then and exception:
>>>>>>>>
>>>>>>>> 2013-10-24 09:35:19,300 [main] WARN org.apache.pig.backend.**
>>>>>>>> hadoop20.PigJobControl
>>>>>>>> - falling back to default JobControl (not using hadoop 0.20 ?)
>>>>>>>> java.lang.**NoSuchFieldException: runnerState
>>>>>>>> at java.lang.Class.**getDeclaredField(Class.java:**1938)
>>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.<**
>>>>>>>> clinit>(PigJobControl.java:51)
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.shims.**
>>>>>>>> HadoopShims.newJobControl(**HadoopShims.java:97)
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>> mapReduceLayer.**
>>>>>>>> JobControlCompiler.compile(**JobControlCompiler.java:285)
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>> mapReduceLayer.**
>>>>>>>> MapReduceLauncher.launchPig(**MapReduceLauncher.java:177)
>>>>>>>> at org.apache.pig.PigServer.**launchPlan(PigServer.java:**1264)
>>>>>>>> at org.apache.pig.PigServer.**executeCompiledLogicalPlan(**
>>>>>>>> PigServer.java:1249)
>>>>>>>> at org.apache.pig.PigServer.**storeEx(PigServer.java:931)
>>>>>>>> at org.apache.pig.PigServer.**store(PigServer.java:898)
>>>>>>>> at org.apache.pig.PigServer.**openIterator(PigServer.java:**
>>>>>>>> 811)
>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>>> GruntParser.java:696)
>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>> PigScriptParser.parse(**
>>>>>>>> PigScriptParser.java:320)
>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>>>>>>> GruntParser.java:194)
>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.parseStopOnError(**
>>>>>>>> GruntParser.java:170)
>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>>> Method)
>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>>
>>>>>>>> ------------------------------**--------
>>>>>>>>
>>>>>>>> Backend error message during job submission
>>>>>>>> ------------------------------**-------------
>>>>>>>> Unexpected System Error Occured: java.lang.**
>>>>>>>> IncompatibleClassChangeError:
>>>>>>>> Found interface org.apache.hadoop.mapreduce.**JobContext, but class
>>>>>>>> was
>>>>>>>> expected
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>> mapReduceLayer.**PigOutputFormat.**setupUdfEnvAndStores(**
>>>>>>>> PigOutputFormat.java:225)
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>> mapReduceLayer.**PigOutputFormat.**checkOutputSpecs(**
>>>>>>>> PigOutputFormat.java:186)
>>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.checkSpecs(**
>>>>>>>> JobSubmitter.java:456)
>>>>>>>> at org.apache.hadoop.mapreduce.**JobSubmitter.**
>>>>>>>> submitJobInternal(
>>>>>>>> **JobSubmitter.java:342)
>>>>>>>> at org.apache.hadoop.mapreduce.**Job$10.run(Job.java:1268)
>>>>>>>> at org.apache.hadoop.mapreduce.**Job$10.run(Job.java:1265)
>>>>>>>> at java.security.**AccessController.doPrivileged(**Native
>>>>>>>> Method)
>>>>>>>> at javax.security.auth.Subject.**doAs(Subject.java:415)
>>>>>>>> at org.apache.hadoop.security.**
>>>>>>>> UserGroupInformation.doAs(**
>>>>>>>> UserGroupInformation.java:**1491)
>>>>>>>> at org.apache.hadoop.mapreduce.**Job.submit(Job.java:1265)
>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>> lib.jobcontrol.ControlledJob.**
>>>>>>>> submit(ControlledJob.java:335)
>>>>>>>> at org.apache.hadoop.mapreduce.**
>>>>>>>> lib.jobcontrol.JobControl.run(**
>>>>>>>> JobControl.java:240)
>>>>>>>> at org.apache.pig.backend.**hadoop20.PigJobControl.run(**
>>>>>>>> PigJobControl.java:121)
>>>>>>>> at java.lang.Thread.run(Thread.**java:724)
>>>>>>>> at org.apache.pig.backend.hadoop.**executionengine.**
>>>>>>>> mapReduceLayer.**MapReduceLauncher$1.run(**
>>>>>>>> MapReduceLauncher.java:257)
>>>>>>>>
>>>>>>>> Pig Stack Trace
>>>>>>>> ---------------
>>>>>>>> ERROR 1066: Unable to open iterator for alias data
>>>>>>>>
>>>>>>>> org.apache.pig.impl.**logicalLayer.**FrontendException: ERROR 1066:
>>>>>>>> Unable to open iterator for alias data
>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>> openIterator(PigServer.java:**836)
>>>>>>>> at org.apache.pig.tools.grunt.**GruntParser.processDump(**
>>>>>>>> GruntParser.java:696)
>>>>>>>> at org.apache.pig.tools.**pigscript.parser.**
>>>>>>>> PigScriptParser.parse(**PigScriptParser.java:320)
>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>> GruntParser.java:194)
>>>>>>>> at org.apache.pig.tools.grunt.**
>>>>>>>> GruntParser.parseStopOnError(**
>>>>>>>> GruntParser.java:170)
>>>>>>>> at org.apache.pig.tools.grunt.**Grunt.run(Grunt.java:69)
>>>>>>>> at org.apache.pig.Main.run(Main.**java:538)
>>>>>>>> at org.apache.pig.Main.main(Main.**java:157)
>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke0(Native
>>>>>>>> Method)
>>>>>>>> at sun.reflect.**NativeMethodAccessorImpl.**invoke(**
>>>>>>>> NativeMethodAccessorImpl.java:**57)
>>>>>>>> at sun.reflect.**DelegatingMethodAccessorImpl.**invoke(**
>>>>>>>> DelegatingMethodAccessorImpl.**java:43)
>>>>>>>> at java.lang.reflect.Method.**invoke(Method.java:606)
>>>>>>>> at org.apache.hadoop.util.RunJar.**main(RunJar.java:212)
>>>>>>>> Caused by: java.io.IOException: Job terminated with anomalous status
>>>>>>>> FAILED
>>>>>>>> at org.apache.pig.PigServer.**
>>>>>>>> openIterator(PigServer.java:**828)
>>>>>>>> ... 12 more
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>