You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Erik Onnen <eo...@gmail.com> on 2011/06/20 20:07:34 UTC
Couldn't retrieve job
Hello all,
I'm trying to run pig 0.8.1 jobs on top of HBase with a custom
LoadFunc. This has worked in the past for us but for some reason it's
now not working and I can't quite tell why. With verbose logging all I
get is the following:
grunt> REGISTER /home/erik/cargo-1.0.jar;
grunt> DEVICES = LOAD 'all_devices' USING cargo.AllDevices AS
(deviceID:chararray);
grunt> DUMP DEVICES;
2011-06-20 13:51:20,045 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
script: UNKNOWN
2011-06-20 13:51:20,045 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
pig.usenewlogicalplan is set to true. New logical plan will be used.
2011-06-20 13:51:20,197 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
(Name: DEVICES:
Store(hdfs://master-0:7080/tmp/temp-1092292460/tmp-1962219688:org.apache.pig.impl.io.InterStorage)
- scope-5 Operator Key: scope-5)
2011-06-20 13:51:20,207 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
- File concatenation threshold: 100 optimistic? false
2011-06-20 13:51:20,226 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2011-06-20 13:51:20,226 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2011-06-20 13:51:20,241 [main] INFO
org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
added to the job
2011-06-20 13:51:20,255 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- mapred.job.reduce.markreset.buffer.percent is not set, set to
default 0.3
2011-06-20 13:51:33,902 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2011-06-20 13:51:33,934 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map-reduce job(s) waiting for submission.
2011-06-20 13:51:34,436 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2011-06-20 13:51:34,584 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2011-06-20 13:51:34,584 [main] ERROR
org.apache.pig.tools.pigstats.PigStatsUtil - 0 map reduce job(s)
failed!
2011-06-20 13:51:34,586 [main] INFO
org.apache.pig.tools.pigstats.PigStats - Script Statistics:
HadoopVersion PigVersion UserId StartedAt FinishedAt Features
0.20.3-SNAPSHOT 0.8.1 erik 2011-06-20 13:51:20 2011-06-20 13:51:34 UNKNOWN
Failed!
Failed Jobs:
JobId Alias Feature Message Outputs
Input(s):
Output(s):
Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0
Job DAG:
null
2011-06-20 13:51:34,586 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Success!
2011-06-20 13:51:34,588 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 1066: Unable to open iterator for alias DEVICES
2011-06-20 13:51:34,588 [main] ERROR org.apache.pig.tools.grunt.Grunt
- org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
Unable to open iterator for alias DEVICES
at org.apache.pig.PigServer.openIterator(PigServer.java:765)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:615)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:455)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Couldn't retrieve job.
at org.apache.pig.PigServer.store(PigServer.java:829)
at org.apache.pig.PigServer.openIterator(PigServer.java:739)
... 7 more
I can see the job's temp directory but it's empty which I presume is
why the iterator cannot be opened. I've tried with several LoadFunc
implementations and none of them seem to be working. Any pointers on
what I'm doing wrong, logs are pretty sparse and Googling hasn't
yielded anything that seems related to my issue.
Re: Couldn't retrieve job
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Sounds like the setup step failed for the generated MR job.
If you go to the hadoop JT web ui and find your job, you should be
able to see something like "Job Setup: Failed" near the top of the
page. Click on that for the setup log, that might tell you more.
What does cargo.AllDevices loader do? Is it just an extension of hbasestorage?
D
On Mon, Jun 20, 2011 at 11:07 AM, Erik Onnen <eo...@gmail.com> wrote:
> Hello all,
>
> I'm trying to run pig 0.8.1 jobs on top of HBase with a custom
> LoadFunc. This has worked in the past for us but for some reason it's
> now not working and I can't quite tell why. With verbose logging all I
> get is the following:
>
> grunt> REGISTER /home/erik/cargo-1.0.jar;
> grunt> DEVICES = LOAD 'all_devices' USING cargo.AllDevices AS
> (deviceID:chararray);
> grunt> DUMP DEVICES;
> 2011-06-20 13:51:20,045 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig features used in the
> script: UNKNOWN
> 2011-06-20 13:51:20,045 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> pig.usenewlogicalplan is set to true. New logical plan will be used.
> 2011-06-20 13:51:20,197 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> (Name: DEVICES:
> Store(hdfs://master-0:7080/tmp/temp-1092292460/tmp-1962219688:org.apache.pig.impl.io.InterStorage)
> - scope-5 Operator Key: scope-5)
> 2011-06-20 13:51:20,207 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler
> - File concatenation threshold: 100 optimistic? false
> 2011-06-20 13:51:20,226 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2011-06-20 13:51:20,226 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2011-06-20 13:51:20,241 [main] INFO
> org.apache.pig.tools.pigstats.ScriptState - Pig script settings are
> added to the job
> 2011-06-20 13:51:20,255 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - mapred.job.reduce.markreset.buffer.percent is not set, set to
> default 0.3
> 2011-06-20 13:51:33,902 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2011-06-20 13:51:33,934 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map-reduce job(s) waiting for submission.
> 2011-06-20 13:51:34,436 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2011-06-20 13:51:34,584 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2011-06-20 13:51:34,584 [main] ERROR
> org.apache.pig.tools.pigstats.PigStatsUtil - 0 map reduce job(s)
> failed!
> 2011-06-20 13:51:34,586 [main] INFO
> org.apache.pig.tools.pigstats.PigStats - Script Statistics:
>
> HadoopVersion PigVersion UserId StartedAt FinishedAt Features
> 0.20.3-SNAPSHOT 0.8.1 erik 2011-06-20 13:51:20 2011-06-20 13:51:34 UNKNOWN
>
> Failed!
>
> Failed Jobs:
> JobId Alias Feature Message Outputs
>
> Input(s):
>
> Output(s):
>
> Counters:
> Total records written : 0
> Total bytes written : 0
> Spillable Memory Manager spill count : 0
> Total bags proactively spilled: 0
> Total records proactively spilled: 0
>
> Job DAG:
> null
>
>
> 2011-06-20 13:51:34,586 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Success!
> 2011-06-20 13:51:34,588 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias DEVICES
> 2011-06-20 13:51:34,588 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> Unable to open iterator for alias DEVICES
> at org.apache.pig.PigServer.openIterator(PigServer.java:765)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:615)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> at org.apache.pig.Main.run(Main.java:455)
> at org.apache.pig.Main.main(Main.java:107)
> Caused by: java.io.IOException: Couldn't retrieve job.
> at org.apache.pig.PigServer.store(PigServer.java:829)
> at org.apache.pig.PigServer.openIterator(PigServer.java:739)
> ... 7 more
>
> I can see the job's temp directory but it's empty which I presume is
> why the iterator cannot be opened. I've tried with several LoadFunc
> implementations and none of them seem to be working. Any pointers on
> what I'm doing wrong, logs are pretty sparse and Googling hasn't
> yielded anything that seems related to my issue.
>