You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by "Bae, Jae Hyeon" <me...@gmail.com> on 2010/03/16 07:02:01 UTC
Error on LoadFunc implementation
Hi everyone.
I implemented LoadFunc and Slicer interface and I tried to use it with
the following pig script:
a = LOAD '/data/part-00000.anc.gz' USING MyStorage() AS (source:chararray);
b = FILTER a BY (source == 'http://www.sonyfinance.co.jp/');
DUMP b;
MapReduce job is dead immediately with the following error stack trace:
java.lang.ArrayIndexOutOfBoundsException: 0
at org.apache.hadoop.mapred.JobInProgress.getTaskInProgress(JobInProgress.java:2523)
at org.apache.hadoop.mapred.taskdetails_jsp._jspService(taskdetails_jsp.java:113)
at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:324)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
What should I do to solve this problem? I am using hadoop-0.20.1 and pig-0.6.0.
Re: Error on LoadFunc implementation
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
I found a solution...
Implementation of MySlice.getLocations() had a bug :)
Sorry for disturbing you...
2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
> I tested MyStorage with local-installation of hadoop, it worked successfully!!!
>
> OMG, what's wrong with my implementation? could you guess?
>
> 2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
>> I am sorry that the following error messages are generated from taskdetails.jsp.
>>
>> I tried to investigate what caused mapreduce job to fail with reading
>> pig source codes, but I couldn't...
>>
>> Please let me know how I can fix it...
>>
>> I attached console output of pig execution:
>>
>> 2010-03-16 15:15:59,092 [main] INFO org.apache.pig.Main - Logging
>> error messages to: /home1/irteam/work/jaybae/pig/pig_1268720159091.log
>> 2010-03-16 15:16:00,931 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting to hadoop file system at: hdfs://h7b611:9000/
>> 2010-03-16 15:16:01,348 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
>> Connecting to map-reduce job tracker at: hdfs://h7b612:9001/
>> 2010-03-16 15:16:03,487 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column
>> pruned for a
>> 2010-03-16 15:16:03,487 [main] INFO
>> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
>> pruned for a
>> 2010-03-16 15:16:05,765 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size before optimization: 1
>> 2010-03-16 15:16:05,765 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
>> - MR plan size after optimization: 1
>> 2010-03-16 15:16:08,856 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
>> - Setting up single store job
>> 2010-03-16 15:16:10,046 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Cannot get jobid for this job
>> 2010-03-16 15:16:11,746 [Thread-9] WARN
>> org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
>> parsing the arguments. Applications should implement Tool for the
>> same.
>> 2010-03-16 15:16:20,002 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 0% complete
>> 2010-03-16 15:16:26,138 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 100% complete
>> 2010-03-16 15:16:26,427 [main] ERROR
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - 1 map reduce job(s) failed!
>> 2010-03-16 15:16:26,437 [main] ERROR
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Failed to produce result in:
>> "hdfs://h7b611:9000/tmp/temp-1085687982/tmp1226074556"
>> 2010-03-16 15:16:26,439 [main] INFO
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - Failed!
>> 2010-03-16 15:16:27,271 [main] ERROR org.apache.pig.tools.grunt.Grunt
>> - ERROR 1066: Unable to open iterator for alias b
>> Details at logfile: pig_1268720159091.log
>>
>>
>> Here is the log file
>>
>> Pig Stack Trace
>> ---------------
>> ERROR 1066: Unable to open iterator for alias b
>>
>> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
>> to open iterator for alias b
>> at org.apache.pig.PigServer.openIterator(PigServer.java:482)
>> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
>> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
>> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
>> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
>> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
>> at org.apache.pig.Main.main(Main.java:386)
>> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
>> at org.apache.pig.PigServer.openIterator(PigServer.java:476)
>> ... 6 more
>> ================================================================================
>>
>>
>> 2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
>>> Hi everyone.
>>>
>>> I implemented LoadFunc and Slicer interface and I tried to use it with
>>> the following pig script:
>>>
>>> a = LOAD '/data/part-00000.anc.gz' USING MyStorage() AS (source:chararray);
>>> b = FILTER a BY (source == 'http://www.sonyfinance.co.jp/');
>>> DUMP b;
>>>
>>> MapReduce job is dead immediately with the following error stack trace:
>>>
>>> java.lang.ArrayIndexOutOfBoundsException: 0
>>> at org.apache.hadoop.mapred.JobInProgress.getTaskInProgress(JobInProgress.java:2523)
>>> at org.apache.hadoop.mapred.taskdetails_jsp._jspService(taskdetails_jsp.java:113)
>>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>> at org.mortbay.jetty.Server.handle(Server.java:324)
>>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>>> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>>
>>> What should I do to solve this problem? I am using hadoop-0.20.1 and pig-0.6.0.
>>>
>>
>
Re: Error on LoadFunc implementation
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
I tested MyStorage with local-installation of hadoop, it worked successfully!!!
OMG, what's wrong with my implementation? could you guess?
2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
> I am sorry that the following error messages are generated from taskdetails.jsp.
>
> I tried to investigate what caused mapreduce job to fail with reading
> pig source codes, but I couldn't...
>
> Please let me know how I can fix it...
>
> I attached console output of pig execution:
>
> 2010-03-16 15:15:59,092 [main] INFO org.apache.pig.Main - Logging
> error messages to: /home1/irteam/work/jaybae/pig/pig_1268720159091.log
> 2010-03-16 15:16:00,931 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting to hadoop file system at: hdfs://h7b611:9000/
> 2010-03-16 15:16:01,348 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
> Connecting to map-reduce job tracker at: hdfs://h7b612:9001/
> 2010-03-16 15:16:03,487 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column
> pruned for a
> 2010-03-16 15:16:03,487 [main] INFO
> org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
> pruned for a
> 2010-03-16 15:16:05,765 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size before optimization: 1
> 2010-03-16 15:16:05,765 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
> - MR plan size after optimization: 1
> 2010-03-16 15:16:08,856 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
> - Setting up single store job
> 2010-03-16 15:16:10,046 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Cannot get jobid for this job
> 2010-03-16 15:16:11,746 [Thread-9] WARN
> org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
> parsing the arguments. Applications should implement Tool for the
> same.
> 2010-03-16 15:16:20,002 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 0% complete
> 2010-03-16 15:16:26,138 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> 2010-03-16 15:16:26,427 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 1 map reduce job(s) failed!
> 2010-03-16 15:16:26,437 [main] ERROR
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed to produce result in:
> "hdfs://h7b611:9000/tmp/temp-1085687982/tmp1226074556"
> 2010-03-16 15:16:26,439 [main] INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - Failed!
> 2010-03-16 15:16:27,271 [main] ERROR org.apache.pig.tools.grunt.Grunt
> - ERROR 1066: Unable to open iterator for alias b
> Details at logfile: pig_1268720159091.log
>
>
> Here is the log file
>
> Pig Stack Trace
> ---------------
> ERROR 1066: Unable to open iterator for alias b
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> to open iterator for alias b
> at org.apache.pig.PigServer.openIterator(PigServer.java:482)
> at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
> at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
> at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
> at org.apache.pig.Main.main(Main.java:386)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> at org.apache.pig.PigServer.openIterator(PigServer.java:476)
> ... 6 more
> ================================================================================
>
>
> 2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
>> Hi everyone.
>>
>> I implemented LoadFunc and Slicer interface and I tried to use it with
>> the following pig script:
>>
>> a = LOAD '/data/part-00000.anc.gz' USING MyStorage() AS (source:chararray);
>> b = FILTER a BY (source == 'http://www.sonyfinance.co.jp/');
>> DUMP b;
>>
>> MapReduce job is dead immediately with the following error stack trace:
>>
>> java.lang.ArrayIndexOutOfBoundsException: 0
>> at org.apache.hadoop.mapred.JobInProgress.getTaskInProgress(JobInProgress.java:2523)
>> at org.apache.hadoop.mapred.taskdetails_jsp._jspService(taskdetails_jsp.java:113)
>> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
>> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
>> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
>> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
>> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
>> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>> at org.mortbay.jetty.Server.handle(Server.java:324)
>> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
>> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
>> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
>> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
>> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>>
>> What should I do to solve this problem? I am using hadoop-0.20.1 and pig-0.6.0.
>>
>
Re: Error on LoadFunc implementation
Posted by "Bae, Jae Hyeon" <me...@gmail.com>.
I am sorry that the following error messages are generated from taskdetails.jsp.
I tried to investigate what caused mapreduce job to fail with reading
pig source codes, but I couldn't...
Please let me know how I can fix it...
I attached console output of pig execution:
2010-03-16 15:15:59,092 [main] INFO org.apache.pig.Main - Logging
error messages to: /home1/irteam/work/jaybae/pig/pig_1268720159091.log
2010-03-16 15:16:00,931 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to hadoop file system at: hdfs://h7b611:9000/
2010-03-16 15:16:01,348 [main] INFO
org.apache.pig.backend.hadoop.executionengine.HExecutionEngine -
Connecting to map-reduce job tracker at: hdfs://h7b612:9001/
2010-03-16 15:16:03,487 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No column
pruned for a
2010-03-16 15:16:03,487 [main] INFO
org.apache.pig.impl.logicalLayer.optimizer.PruneColumns - No map keys
pruned for a
2010-03-16 15:16:05,765 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size before optimization: 1
2010-03-16 15:16:05,765 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer
- MR plan size after optimization: 1
2010-03-16 15:16:08,856 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Setting up single store job
2010-03-16 15:16:10,046 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Cannot get jobid for this job
2010-03-16 15:16:11,746 [Thread-9] WARN
org.apache.hadoop.mapred.JobClient - Use GenericOptionsParser for
parsing the arguments. Applications should implement Tool for the
same.
2010-03-16 15:16:20,002 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 0% complete
2010-03-16 15:16:26,138 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
2010-03-16 15:16:26,427 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 1 map reduce job(s) failed!
2010-03-16 15:16:26,437 [main] ERROR
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed to produce result in:
"hdfs://h7b611:9000/tmp/temp-1085687982/tmp1226074556"
2010-03-16 15:16:26,439 [main] INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- Failed!
2010-03-16 15:16:27,271 [main] ERROR org.apache.pig.tools.grunt.Grunt
- ERROR 1066: Unable to open iterator for alias b
Details at logfile: pig_1268720159091.log
Here is the log file
Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias b
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
to open iterator for alias b
at org.apache.pig.PigServer.openIterator(PigServer.java:482)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:539)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:241)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:168)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:144)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:89)
at org.apache.pig.Main.main(Main.java:386)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:476)
... 6 more
================================================================================
2010/3/16 Bae, Jae Hyeon <me...@gmail.com>:
> Hi everyone.
>
> I implemented LoadFunc and Slicer interface and I tried to use it with
> the following pig script:
>
> a = LOAD '/data/part-00000.anc.gz' USING MyStorage() AS (source:chararray);
> b = FILTER a BY (source == 'http://www.sonyfinance.co.jp/');
> DUMP b;
>
> MapReduce job is dead immediately with the following error stack trace:
>
> java.lang.ArrayIndexOutOfBoundsException: 0
> at org.apache.hadoop.mapred.JobInProgress.getTaskInProgress(JobInProgress.java:2523)
> at org.apache.hadoop.mapred.taskdetails_jsp._jspService(taskdetails_jsp.java:113)
> at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:97)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
> at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:363)
> at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
> at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
> at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
> at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:417)
> at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
> at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
> at org.mortbay.jetty.Server.handle(Server.java:324)
> at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
> at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:864)
> at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533)
> at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207)
> at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403)
> at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
> at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:522)
>
> What should I do to solve this problem? I am using hadoop-0.20.1 and pig-0.6.0.
>