You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Dimitris Iliopoulos <di...@gmail.com> on 2011/04/01 23:16:50 UTC
Unable to run Pig in MapReduce mode
Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things in local
mode as well as run fs -* commands from within Grunt running in MapReduce
mode. I can't seem to be able to execute any Pig Latin statement in
MapReduce mode. The fact that I can browse HDFS and access files leads me to
believe that the setup is correct (also verified by the correct namenode and
jobtracker addresses connected to when firing off any pig command). I don't
know why though I can't run anything else. A simple command like the
following:
data = LOAD 'test.csv' USING PigStorage(',');
DUMP data;
Produces the following stack trace:
Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias data
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias data
at org.apache.pig.PigServer.openIterator(PigServer.java:754)
at
org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
at
org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
at org.apache.pig.Main.run(Main.java:465)
at org.apache.pig.Main.main(Main.java:107)
Caused by: java.io.IOException: Job terminated with anomalous status FAILED
at org.apache.pig.PigServer.openIterator(PigServer.java:744)
... 7 more
Any other more elaborate collection of Pig statements will produce this:
Pig Stack Trace
---------------
ERROR 2244: Job failed, hadoop does not return any error message
org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job
failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
at
org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
at org.apache.pig.Main.run(Main.java:510)
at org.apache.pig.Main.main(Main.java:107)
Where any input file is flagged as unreadable by the following error message
during execution:
Input(s):
Failed to read data from "test.csv"
All files are accessible in MapReduce mode via cat, ls etc. I might be
missing something very obvious here since this is the first time I've used
Pig so any help would be greatly appreciated.
- Dimitris
Re: Unable to run Pig in MapReduce mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
This kind of sounds like https://issues.apache.org/jira/browse/PIG-1955 but
I don't understand why it doesn't happen for anyone else..
On Mon, Apr 4, 2011 at 2:14 PM, Dimitris Iliopoulos <
dimitris.iliopoulos@gmail.com> wrote:
> So in terms of my Pig setup as I said I am using 0.8.0 and have the
> following environment variables set:
>
> export PIG_INSTALL=/home/UNIXHOME/diliopoulos/Hadoop/lib/pig-0.8.0
> export PIG_CLASSPATH=$HADOOP_INSTALL/conf/
> export PATH=$PATH:$PIG_INSTALL/bin
>
> I have build Pig locally and have tried running the tutorial examples but I
> am running into the same error messages when I run them on hadoop versus
> locally. I have tried to move the above environment variables into
> hadoop_env.sh but the errors persist. Have also tried explicitly passing
> the
> classpath by using "hadoop -cp" but still no luck. I'm pretty sure I am
> missing something really obvious here but I can't seem to find it.
>
> - Dimitris
>
> On Fri, Apr 1, 2011 at 6:51 PM, <di...@gmail.com> wrote:
>
> > Running the 0.8.0 release with Hadoop 0.20.2 and the rest are pretty
> > standard.
> > Sent via BlackBerry from T-Mobile
> >
> > -----Original Message-----
> > From: Dmitriy Ryaboy <dv...@gmail.com>
> > Date: Fri, 1 Apr 2011 18:26:34
> > To: <us...@pig.apache.org>
> > Reply-To: user@pig.apache.org
> > Subject: Re: Unable to run Pig in MapReduce mode
> >
> > Are you running a released version of Pig or something from svn?
> > Anything interesting about your log setup or classpath we should know
> > about?
> >
> > D
> >
> > On Fri, Apr 1, 2011 at 6:19 PM, Dimitris Iliopoulos <
> > dimitris.iliopoulos@gmail.com> wrote:
> >
> > > There is a "Fail" link and the error trace that I get from there is as
> > > follows:
> > >
> > > java.io.IOException: Deserialization error:
> > > org.apache.commons.logging.impl.Log4JLogger;
> > > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > > deserialization
> > >
> > > at
> > >
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
> > > at
> > >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getStores(PigOutputFormat.java:217)
> > > at
> > >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:244)
> > > at
> > >
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:274)
> > > at org.apache.hadoop.mapred.Task.initialize(Task.java:415)
> > > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
> > > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > > Caused by: java.io.InvalidClassException:
> > > org.apache.commons.logging.impl.Log4JLogger;
> > > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > > deserialization
> > > at
> > > java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:713)
> > > at
> > >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
> > > at
> > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > > at
> > >
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> > > at
> > > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> > > at
> > >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> > > at
> > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > > at
> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> > > at java.util.LinkedList.readObject(LinkedList.java:964)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > > at
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > > at java.lang.reflect.Method.invoke(Method.java:597)
> > > at
> > > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> > > at
> > > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> > > at
> > >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> > > at
> > > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > > at
> > java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> > > at
> > >
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
> > > ... 6 more
> > > Caused by: java.io.InvalidClassException:
> > > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > > deserialization
> > > at
> > > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:587)
> > > at
> > > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
> > > at
> > > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
> > > at
> > >
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
> > > ... 23 more
> > >
> > >
> > > On Fri, Apr 1, 2011 at 6:13 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> > >
> > > > Is there a line that says "Job Setup: Failed" at the top of the job
> > page?
> > > > Failed (or "Successful") should be a link, click through that to get
> > the
> > > > setup log.
> > > >
> > > > D
> > > >
> > > > On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
> > > > dimitris.iliopoulos@gmail.com> wrote:
> > > >
> > > > > Dmitriy,
> > > > >
> > > > > The jobtracker does indeed mark the job as failed but the problem
> is
> > > that
> > > > > there are no log files. All map and reduce tasks are flagged as
> > killed
> > > > and
> > > > > upon inspection produce no logs. The only logs are the ones I
> > appended
> > > > > before produced by Pig.
> > > > >
> > > > > - Dimitris
> > > > >
> > > > > On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dvryaboy@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Dimitris,
> > > > > > You should be able to look at the jobs submitted to your
> jobtracker
> > > and
> > > > > > find
> > > > > > out what happened to the pig job (the message " Job terminated
> with
> > > > > > anomalous status FAILED" leads me to believe there is some job
> > setup
> > > > > > problem). Do you see any pig job attempts on your jobtracker UI?
> > > > > >
> > > > > > D
> > > > > >
> > > > > > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > > > > > dimitris.iliopoulos@gmail.com> wrote:
> > > > > >
> > > > > > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run
> things
> > > in
> > > > > > local
> > > > > > > mode as well as run fs -* commands from within Grunt running in
> > > > > MapReduce
> > > > > > > mode. I can't seem to be able to execute any Pig Latin
> statement
> > in
> > > > > > > MapReduce mode. The fact that I can browse HDFS and access
> files
> > > > leads
> > > > > me
> > > > > > > to
> > > > > > > believe that the setup is correct (also verified by the correct
> > > > > namenode
> > > > > > > and
> > > > > > > jobtracker addresses connected to when firing off any pig
> > command).
> > > I
> > > > > > don't
> > > > > > > know why though I can't run anything else. A simple command
> like
> > > the
> > > > > > > following:
> > > > > > >
> > > > > > > data = LOAD 'test.csv' USING PigStorage(',');
> > > > > > > DUMP data;
> > > > > > >
> > > > > > > Produces the following stack trace:
> > > > > > >
> > > > > > > Pig Stack Trace
> > > > > > > ---------------
> > > > > > > ERROR 1066: Unable to open iterator for alias data
> > > > > > >
> > > > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> > > > Unable
> > > > > to
> > > > > > > open iterator for alias data
> > > > > > > at
> > org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > > > > > at org.apache.pig.Main.run(Main.java:465)
> > > > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > > > Caused by: java.io.IOException: Job terminated with anomalous
> > > status
> > > > > > FAILED
> > > > > > > at
> > org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > > > > > ... 7 more
> > > > > > >
> > > > > > > Any other more elaborate collection of Pig statements will
> > produce
> > > > > this:
> > > > > > >
> > > > > > > Pig Stack Trace
> > > > > > > ---------------
> > > > > > > ERROR 2244: Job failed, hadoop does not return any error
> message
> > > > > > >
> > > > > > > org.apache.pig.backend.executionengine.ExecException: ERROR
> 2244:
> > > Job
> > > > > > > failed, hadoop does not return any error message
> > > > > > > at
> > > > > > >
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > > > > > at
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > > > > > at org.apache.pig.Main.run(Main.java:510)
> > > > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > > >
> > > > > > > Where any input file is flagged as unreadable by the following
> > > error
> > > > > > > message
> > > > > > > during execution:
> > > > > > >
> > > > > > > Input(s):
> > > > > > > Failed to read data from "test.csv"
> > > > > > >
> > > > > > > All files are accessible in MapReduce mode via cat, ls etc. I
> > might
> > > > be
> > > > > > > missing something very obvious here since this is the first
> time
> > > I've
> > > > > > used
> > > > > > > Pig so any help would be greatly appreciated.
> > > > > > >
> > > > > > > - Dimitris
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dimitris Iliopoulos <di...@gmail.com>.
So in terms of my Pig setup as I said I am using 0.8.0 and have the
following environment variables set:
export PIG_INSTALL=/home/UNIXHOME/diliopoulos/Hadoop/lib/pig-0.8.0
export PIG_CLASSPATH=$HADOOP_INSTALL/conf/
export PATH=$PATH:$PIG_INSTALL/bin
I have build Pig locally and have tried running the tutorial examples but I
am running into the same error messages when I run them on hadoop versus
locally. I have tried to move the above environment variables into
hadoop_env.sh but the errors persist. Have also tried explicitly passing the
classpath by using "hadoop -cp" but still no luck. I'm pretty sure I am
missing something really obvious here but I can't seem to find it.
- Dimitris
On Fri, Apr 1, 2011 at 6:51 PM, <di...@gmail.com> wrote:
> Running the 0.8.0 release with Hadoop 0.20.2 and the rest are pretty
> standard.
> Sent via BlackBerry from T-Mobile
>
> -----Original Message-----
> From: Dmitriy Ryaboy <dv...@gmail.com>
> Date: Fri, 1 Apr 2011 18:26:34
> To: <us...@pig.apache.org>
> Reply-To: user@pig.apache.org
> Subject: Re: Unable to run Pig in MapReduce mode
>
> Are you running a released version of Pig or something from svn?
> Anything interesting about your log setup or classpath we should know
> about?
>
> D
>
> On Fri, Apr 1, 2011 at 6:19 PM, Dimitris Iliopoulos <
> dimitris.iliopoulos@gmail.com> wrote:
>
> > There is a "Fail" link and the error trace that I get from there is as
> > follows:
> >
> > java.io.IOException: Deserialization error:
> > org.apache.commons.logging.impl.Log4JLogger;
> > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > deserialization
> >
> > at
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getStores(PigOutputFormat.java:217)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:244)
> > at
> >
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:274)
> > at org.apache.hadoop.mapred.Task.initialize(Task.java:415)
> > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
> > at org.apache.hadoop.mapred.Child.main(Child.java:170)
> > Caused by: java.io.InvalidClassException:
> > org.apache.commons.logging.impl.Log4JLogger;
> > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > deserialization
> > at
> > java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:713)
> > at
> > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
> > at
> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > at
> > java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> > at
> > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> > at
> > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> > at
> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> > at java.util.LinkedList.readObject(LinkedList.java:964)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> > java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> > at
> > java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> > at
> > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> > at
> > java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> > at
> java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> > at
> >
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
> > ... 6 more
> > Caused by: java.io.InvalidClassException:
> > org.apache.commons.logging.impl.Log4JLogger; class invalid for
> > deserialization
> > at
> > java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:587)
> > at
> > java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
> > at
> > java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
> > at
> > java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
> > ... 23 more
> >
> >
> > On Fri, Apr 1, 2011 at 6:13 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
> >
> > > Is there a line that says "Job Setup: Failed" at the top of the job
> page?
> > > Failed (or "Successful") should be a link, click through that to get
> the
> > > setup log.
> > >
> > > D
> > >
> > > On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
> > > dimitris.iliopoulos@gmail.com> wrote:
> > >
> > > > Dmitriy,
> > > >
> > > > The jobtracker does indeed mark the job as failed but the problem is
> > that
> > > > there are no log files. All map and reduce tasks are flagged as
> killed
> > > and
> > > > upon inspection produce no logs. The only logs are the ones I
> appended
> > > > before produced by Pig.
> > > >
> > > > - Dimitris
> > > >
> > > > On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > > wrote:
> > > >
> > > > > Dimitris,
> > > > > You should be able to look at the jobs submitted to your jobtracker
> > and
> > > > > find
> > > > > out what happened to the pig job (the message " Job terminated with
> > > > > anomalous status FAILED" leads me to believe there is some job
> setup
> > > > > problem). Do you see any pig job attempts on your jobtracker UI?
> > > > >
> > > > > D
> > > > >
> > > > > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > > > > dimitris.iliopoulos@gmail.com> wrote:
> > > > >
> > > > > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things
> > in
> > > > > local
> > > > > > mode as well as run fs -* commands from within Grunt running in
> > > > MapReduce
> > > > > > mode. I can't seem to be able to execute any Pig Latin statement
> in
> > > > > > MapReduce mode. The fact that I can browse HDFS and access files
> > > leads
> > > > me
> > > > > > to
> > > > > > believe that the setup is correct (also verified by the correct
> > > > namenode
> > > > > > and
> > > > > > jobtracker addresses connected to when firing off any pig
> command).
> > I
> > > > > don't
> > > > > > know why though I can't run anything else. A simple command like
> > the
> > > > > > following:
> > > > > >
> > > > > > data = LOAD 'test.csv' USING PigStorage(',');
> > > > > > DUMP data;
> > > > > >
> > > > > > Produces the following stack trace:
> > > > > >
> > > > > > Pig Stack Trace
> > > > > > ---------------
> > > > > > ERROR 1066: Unable to open iterator for alias data
> > > > > >
> > > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> > > Unable
> > > > to
> > > > > > open iterator for alias data
> > > > > > at
> org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > > > > at
> > > > > >
> > > >
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > > > > at org.apache.pig.Main.run(Main.java:465)
> > > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > > Caused by: java.io.IOException: Job terminated with anomalous
> > status
> > > > > FAILED
> > > > > > at
> org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > > > > ... 7 more
> > > > > >
> > > > > > Any other more elaborate collection of Pig statements will
> produce
> > > > this:
> > > > > >
> > > > > > Pig Stack Trace
> > > > > > ---------------
> > > > > > ERROR 2244: Job failed, hadoop does not return any error message
> > > > > >
> > > > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2244:
> > Job
> > > > > > failed, hadoop does not return any error message
> > > > > > at
> > > > > >
> > > >
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > > > > at org.apache.pig.Main.run(Main.java:510)
> > > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > >
> > > > > > Where any input file is flagged as unreadable by the following
> > error
> > > > > > message
> > > > > > during execution:
> > > > > >
> > > > > > Input(s):
> > > > > > Failed to read data from "test.csv"
> > > > > >
> > > > > > All files are accessible in MapReduce mode via cat, ls etc. I
> might
> > > be
> > > > > > missing something very obvious here since this is the first time
> > I've
> > > > > used
> > > > > > Pig so any help would be greatly appreciated.
> > > > > >
> > > > > > - Dimitris
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
Re: Unable to run Pig in MapReduce mode
Posted by di...@gmail.com.
Running the 0.8.0 release with Hadoop 0.20.2 and the rest are pretty standard.
Sent via BlackBerry from T-Mobile
-----Original Message-----
From: Dmitriy Ryaboy <dv...@gmail.com>
Date: Fri, 1 Apr 2011 18:26:34
To: <us...@pig.apache.org>
Reply-To: user@pig.apache.org
Subject: Re: Unable to run Pig in MapReduce mode
Are you running a released version of Pig or something from svn?
Anything interesting about your log setup or classpath we should know about?
D
On Fri, Apr 1, 2011 at 6:19 PM, Dimitris Iliopoulos <
dimitris.iliopoulos@gmail.com> wrote:
> There is a "Fail" link and the error trace that I get from there is as
> follows:
>
> java.io.IOException: Deserialization error:
> org.apache.commons.logging.impl.Log4JLogger;
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
>
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getStores(PigOutputFormat.java:217)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:244)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:274)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:415)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.io.InvalidClassException:
> org.apache.commons.logging.impl.Log4JLogger;
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
> at
> java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:713)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at java.util.LinkedList.readObject(LinkedList.java:964)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
> ... 6 more
> Caused by: java.io.InvalidClassException:
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:587)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
> ... 23 more
>
>
> On Fri, Apr 1, 2011 at 6:13 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > Is there a line that says "Job Setup: Failed" at the top of the job page?
> > Failed (or "Successful") should be a link, click through that to get the
> > setup log.
> >
> > D
> >
> > On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
> > dimitris.iliopoulos@gmail.com> wrote:
> >
> > > Dmitriy,
> > >
> > > The jobtracker does indeed mark the job as failed but the problem is
> that
> > > there are no log files. All map and reduce tasks are flagged as killed
> > and
> > > upon inspection produce no logs. The only logs are the ones I appended
> > > before produced by Pig.
> > >
> > > - Dimitris
> > >
> > > On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> > >
> > > > Dimitris,
> > > > You should be able to look at the jobs submitted to your jobtracker
> and
> > > > find
> > > > out what happened to the pig job (the message " Job terminated with
> > > > anomalous status FAILED" leads me to believe there is some job setup
> > > > problem). Do you see any pig job attempts on your jobtracker UI?
> > > >
> > > > D
> > > >
> > > > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > > > dimitris.iliopoulos@gmail.com> wrote:
> > > >
> > > > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things
> in
> > > > local
> > > > > mode as well as run fs -* commands from within Grunt running in
> > > MapReduce
> > > > > mode. I can't seem to be able to execute any Pig Latin statement in
> > > > > MapReduce mode. The fact that I can browse HDFS and access files
> > leads
> > > me
> > > > > to
> > > > > believe that the setup is correct (also verified by the correct
> > > namenode
> > > > > and
> > > > > jobtracker addresses connected to when firing off any pig command).
> I
> > > > don't
> > > > > know why though I can't run anything else. A simple command like
> the
> > > > > following:
> > > > >
> > > > > data = LOAD 'test.csv' USING PigStorage(',');
> > > > > DUMP data;
> > > > >
> > > > > Produces the following stack trace:
> > > > >
> > > > > Pig Stack Trace
> > > > > ---------------
> > > > > ERROR 1066: Unable to open iterator for alias data
> > > > >
> > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> > Unable
> > > to
> > > > > open iterator for alias data
> > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > > > at
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > > > at org.apache.pig.Main.run(Main.java:465)
> > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > Caused by: java.io.IOException: Job terminated with anomalous
> status
> > > > FAILED
> > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > > > ... 7 more
> > > > >
> > > > > Any other more elaborate collection of Pig statements will produce
> > > this:
> > > > >
> > > > > Pig Stack Trace
> > > > > ---------------
> > > > > ERROR 2244: Job failed, hadoop does not return any error message
> > > > >
> > > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2244:
> Job
> > > > > failed, hadoop does not return any error message
> > > > > at
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > > > at org.apache.pig.Main.run(Main.java:510)
> > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > >
> > > > > Where any input file is flagged as unreadable by the following
> error
> > > > > message
> > > > > during execution:
> > > > >
> > > > > Input(s):
> > > > > Failed to read data from "test.csv"
> > > > >
> > > > > All files are accessible in MapReduce mode via cat, ls etc. I might
> > be
> > > > > missing something very obvious here since this is the first time
> I've
> > > > used
> > > > > Pig so any help would be greatly appreciated.
> > > > >
> > > > > - Dimitris
> > > > >
> > > >
> > >
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Are you running a released version of Pig or something from svn?
Anything interesting about your log setup or classpath we should know about?
D
On Fri, Apr 1, 2011 at 6:19 PM, Dimitris Iliopoulos <
dimitris.iliopoulos@gmail.com> wrote:
> There is a "Fail" link and the error trace that I get from there is as
> follows:
>
> java.io.IOException: Deserialization error:
> org.apache.commons.logging.impl.Log4JLogger;
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
>
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getStores(PigOutputFormat.java:217)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:244)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:274)
> at org.apache.hadoop.mapred.Task.initialize(Task.java:415)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
> at org.apache.hadoop.mapred.Child.main(Child.java:170)
> Caused by: java.io.InvalidClassException:
> org.apache.commons.logging.impl.Log4JLogger;
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
> at
> java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:713)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at java.util.LinkedList.readObject(LinkedList.java:964)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
> at
> org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
> ... 6 more
> Caused by: java.io.InvalidClassException:
> org.apache.commons.logging.impl.Log4JLogger; class invalid for
> deserialization
> at
> java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:587)
> at
> java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
> at
> java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
> ... 23 more
>
>
> On Fri, Apr 1, 2011 at 6:13 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > Is there a line that says "Job Setup: Failed" at the top of the job page?
> > Failed (or "Successful") should be a link, click through that to get the
> > setup log.
> >
> > D
> >
> > On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
> > dimitris.iliopoulos@gmail.com> wrote:
> >
> > > Dmitriy,
> > >
> > > The jobtracker does indeed mark the job as failed but the problem is
> that
> > > there are no log files. All map and reduce tasks are flagged as killed
> > and
> > > upon inspection produce no logs. The only logs are the ones I appended
> > > before produced by Pig.
> > >
> > > - Dimitris
> > >
> > > On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com>
> > wrote:
> > >
> > > > Dimitris,
> > > > You should be able to look at the jobs submitted to your jobtracker
> and
> > > > find
> > > > out what happened to the pig job (the message " Job terminated with
> > > > anomalous status FAILED" leads me to believe there is some job setup
> > > > problem). Do you see any pig job attempts on your jobtracker UI?
> > > >
> > > > D
> > > >
> > > > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > > > dimitris.iliopoulos@gmail.com> wrote:
> > > >
> > > > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things
> in
> > > > local
> > > > > mode as well as run fs -* commands from within Grunt running in
> > > MapReduce
> > > > > mode. I can't seem to be able to execute any Pig Latin statement in
> > > > > MapReduce mode. The fact that I can browse HDFS and access files
> > leads
> > > me
> > > > > to
> > > > > believe that the setup is correct (also verified by the correct
> > > namenode
> > > > > and
> > > > > jobtracker addresses connected to when firing off any pig command).
> I
> > > > don't
> > > > > know why though I can't run anything else. A simple command like
> the
> > > > > following:
> > > > >
> > > > > data = LOAD 'test.csv' USING PigStorage(',');
> > > > > DUMP data;
> > > > >
> > > > > Produces the following stack trace:
> > > > >
> > > > > Pig Stack Trace
> > > > > ---------------
> > > > > ERROR 1066: Unable to open iterator for alias data
> > > > >
> > > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> > Unable
> > > to
> > > > > open iterator for alias data
> > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > > > at
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > > > at org.apache.pig.Main.run(Main.java:465)
> > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > > Caused by: java.io.IOException: Job terminated with anomalous
> status
> > > > FAILED
> > > > > at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > > > ... 7 more
> > > > >
> > > > > Any other more elaborate collection of Pig statements will produce
> > > this:
> > > > >
> > > > > Pig Stack Trace
> > > > > ---------------
> > > > > ERROR 2244: Job failed, hadoop does not return any error message
> > > > >
> > > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2244:
> Job
> > > > > failed, hadoop does not return any error message
> > > > > at
> > > > >
> > >
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > > > at
> > > > >
> > > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > > > at org.apache.pig.Main.run(Main.java:510)
> > > > > at org.apache.pig.Main.main(Main.java:107)
> > > > >
> > > > > Where any input file is flagged as unreadable by the following
> error
> > > > > message
> > > > > during execution:
> > > > >
> > > > > Input(s):
> > > > > Failed to read data from "test.csv"
> > > > >
> > > > > All files are accessible in MapReduce mode via cat, ls etc. I might
> > be
> > > > > missing something very obvious here since this is the first time
> I've
> > > > used
> > > > > Pig so any help would be greatly appreciated.
> > > > >
> > > > > - Dimitris
> > > > >
> > > >
> > >
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dimitris Iliopoulos <di...@gmail.com>.
There is a "Fail" link and the error trace that I get from there is as
follows:
java.io.IOException: Deserialization error:
org.apache.commons.logging.impl.Log4JLogger;
org.apache.commons.logging.impl.Log4JLogger; class invalid for
deserialization
at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:55)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getStores(PigOutputFormat.java:217)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.setupUdfEnvAndStores(PigOutputFormat.java:244)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getOutputCommitter(PigOutputFormat.java:274)
at org.apache.hadoop.mapred.Task.initialize(Task.java:415)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:354)
at org.apache.hadoop.mapred.Child.main(Child.java:170)
Caused by: java.io.InvalidClassException:
org.apache.commons.logging.impl.Log4JLogger;
org.apache.commons.logging.impl.Log4JLogger; class invalid for
deserialization
at java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:713)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1732)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at java.util.LinkedList.readObject(LinkedList.java:964)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:974)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)
at org.apache.pig.impl.util.ObjectSerializer.deserialize(ObjectSerializer.java:53)
... 6 more
Caused by: java.io.InvalidClassException:
org.apache.commons.logging.impl.Log4JLogger; class invalid for
deserialization
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:587)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1582)
at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1731)
... 23 more
On Fri, Apr 1, 2011 at 6:13 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Is there a line that says "Job Setup: Failed" at the top of the job page?
> Failed (or "Successful") should be a link, click through that to get the
> setup log.
>
> D
>
> On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
> dimitris.iliopoulos@gmail.com> wrote:
>
> > Dmitriy,
> >
> > The jobtracker does indeed mark the job as failed but the problem is that
> > there are no log files. All map and reduce tasks are flagged as killed
> and
> > upon inspection produce no logs. The only logs are the ones I appended
> > before produced by Pig.
> >
> > - Dimitris
> >
> > On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com>
> wrote:
> >
> > > Dimitris,
> > > You should be able to look at the jobs submitted to your jobtracker and
> > > find
> > > out what happened to the pig job (the message " Job terminated with
> > > anomalous status FAILED" leads me to believe there is some job setup
> > > problem). Do you see any pig job attempts on your jobtracker UI?
> > >
> > > D
> > >
> > > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > > dimitris.iliopoulos@gmail.com> wrote:
> > >
> > > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things in
> > > local
> > > > mode as well as run fs -* commands from within Grunt running in
> > MapReduce
> > > > mode. I can't seem to be able to execute any Pig Latin statement in
> > > > MapReduce mode. The fact that I can browse HDFS and access files
> leads
> > me
> > > > to
> > > > believe that the setup is correct (also verified by the correct
> > namenode
> > > > and
> > > > jobtracker addresses connected to when firing off any pig command). I
> > > don't
> > > > know why though I can't run anything else. A simple command like the
> > > > following:
> > > >
> > > > data = LOAD 'test.csv' USING PigStorage(',');
> > > > DUMP data;
> > > >
> > > > Produces the following stack trace:
> > > >
> > > > Pig Stack Trace
> > > > ---------------
> > > > ERROR 1066: Unable to open iterator for alias data
> > > >
> > > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066:
> Unable
> > to
> > > > open iterator for alias data
> > > > at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > > at
> > > >
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > > at org.apache.pig.Main.run(Main.java:465)
> > > > at org.apache.pig.Main.main(Main.java:107)
> > > > Caused by: java.io.IOException: Job terminated with anomalous status
> > > FAILED
> > > > at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > > ... 7 more
> > > >
> > > > Any other more elaborate collection of Pig statements will produce
> > this:
> > > >
> > > > Pig Stack Trace
> > > > ---------------
> > > > ERROR 2244: Job failed, hadoop does not return any error message
> > > >
> > > > org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job
> > > > failed, hadoop does not return any error message
> > > > at
> > > >
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > > at org.apache.pig.Main.run(Main.java:510)
> > > > at org.apache.pig.Main.main(Main.java:107)
> > > >
> > > > Where any input file is flagged as unreadable by the following error
> > > > message
> > > > during execution:
> > > >
> > > > Input(s):
> > > > Failed to read data from "test.csv"
> > > >
> > > > All files are accessible in MapReduce mode via cat, ls etc. I might
> be
> > > > missing something very obvious here since this is the first time I've
> > > used
> > > > Pig so any help would be greatly appreciated.
> > > >
> > > > - Dimitris
> > > >
> > >
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Is there a line that says "Job Setup: Failed" at the top of the job page?
Failed (or "Successful") should be a link, click through that to get the
setup log.
D
On Fri, Apr 1, 2011 at 6:02 PM, Dimitris Iliopoulos <
dimitris.iliopoulos@gmail.com> wrote:
> Dmitriy,
>
> The jobtracker does indeed mark the job as failed but the problem is that
> there are no log files. All map and reduce tasks are flagged as killed and
> upon inspection produce no logs. The only logs are the ones I appended
> before produced by Pig.
>
> - Dimitris
>
> On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
>
> > Dimitris,
> > You should be able to look at the jobs submitted to your jobtracker and
> > find
> > out what happened to the pig job (the message " Job terminated with
> > anomalous status FAILED" leads me to believe there is some job setup
> > problem). Do you see any pig job attempts on your jobtracker UI?
> >
> > D
> >
> > On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> > dimitris.iliopoulos@gmail.com> wrote:
> >
> > > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things in
> > local
> > > mode as well as run fs -* commands from within Grunt running in
> MapReduce
> > > mode. I can't seem to be able to execute any Pig Latin statement in
> > > MapReduce mode. The fact that I can browse HDFS and access files leads
> me
> > > to
> > > believe that the setup is correct (also verified by the correct
> namenode
> > > and
> > > jobtracker addresses connected to when firing off any pig command). I
> > don't
> > > know why though I can't run anything else. A simple command like the
> > > following:
> > >
> > > data = LOAD 'test.csv' USING PigStorage(',');
> > > DUMP data;
> > >
> > > Produces the following stack trace:
> > >
> > > Pig Stack Trace
> > > ---------------
> > > ERROR 1066: Unable to open iterator for alias data
> > >
> > > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
> to
> > > open iterator for alias data
> > > at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > > at
> > >
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > > at org.apache.pig.Main.run(Main.java:465)
> > > at org.apache.pig.Main.main(Main.java:107)
> > > Caused by: java.io.IOException: Job terminated with anomalous status
> > FAILED
> > > at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > > ... 7 more
> > >
> > > Any other more elaborate collection of Pig statements will produce
> this:
> > >
> > > Pig Stack Trace
> > > ---------------
> > > ERROR 2244: Job failed, hadoop does not return any error message
> > >
> > > org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job
> > > failed, hadoop does not return any error message
> > > at
> > >
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > > at
> > >
> > >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > > at org.apache.pig.Main.run(Main.java:510)
> > > at org.apache.pig.Main.main(Main.java:107)
> > >
> > > Where any input file is flagged as unreadable by the following error
> > > message
> > > during execution:
> > >
> > > Input(s):
> > > Failed to read data from "test.csv"
> > >
> > > All files are accessible in MapReduce mode via cat, ls etc. I might be
> > > missing something very obvious here since this is the first time I've
> > used
> > > Pig so any help would be greatly appreciated.
> > >
> > > - Dimitris
> > >
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dimitris Iliopoulos <di...@gmail.com>.
Dmitriy,
The jobtracker does indeed mark the job as failed but the problem is that
there are no log files. All map and reduce tasks are flagged as killed and
upon inspection produce no logs. The only logs are the ones I appended
before produced by Pig.
- Dimitris
On Fri, Apr 1, 2011 at 5:55 PM, Dmitriy Ryaboy <dv...@gmail.com> wrote:
> Dimitris,
> You should be able to look at the jobs submitted to your jobtracker and
> find
> out what happened to the pig job (the message " Job terminated with
> anomalous status FAILED" leads me to believe there is some job setup
> problem). Do you see any pig job attempts on your jobtracker UI?
>
> D
>
> On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
> dimitris.iliopoulos@gmail.com> wrote:
>
> > Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things in
> local
> > mode as well as run fs -* commands from within Grunt running in MapReduce
> > mode. I can't seem to be able to execute any Pig Latin statement in
> > MapReduce mode. The fact that I can browse HDFS and access files leads me
> > to
> > believe that the setup is correct (also verified by the correct namenode
> > and
> > jobtracker addresses connected to when firing off any pig command). I
> don't
> > know why though I can't run anything else. A simple command like the
> > following:
> >
> > data = LOAD 'test.csv' USING PigStorage(',');
> > DUMP data;
> >
> > Produces the following stack trace:
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 1066: Unable to open iterator for alias data
> >
> > org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> > open iterator for alias data
> > at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> > at
> > org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> > at
> >
> >
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> > at
> >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> > at
> >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> > at org.apache.pig.Main.run(Main.java:465)
> > at org.apache.pig.Main.main(Main.java:107)
> > Caused by: java.io.IOException: Job terminated with anomalous status
> FAILED
> > at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> > ... 7 more
> >
> > Any other more elaborate collection of Pig statements will produce this:
> >
> > Pig Stack Trace
> > ---------------
> > ERROR 2244: Job failed, hadoop does not return any error message
> >
> > org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job
> > failed, hadoop does not return any error message
> > at
> > org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> > at
> >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> > at
> >
> >
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> > at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> > at org.apache.pig.Main.run(Main.java:510)
> > at org.apache.pig.Main.main(Main.java:107)
> >
> > Where any input file is flagged as unreadable by the following error
> > message
> > during execution:
> >
> > Input(s):
> > Failed to read data from "test.csv"
> >
> > All files are accessible in MapReduce mode via cat, ls etc. I might be
> > missing something very obvious here since this is the first time I've
> used
> > Pig so any help would be greatly appreciated.
> >
> > - Dimitris
> >
>
Re: Unable to run Pig in MapReduce mode
Posted by Dmitriy Ryaboy <dv...@gmail.com>.
Dimitris,
You should be able to look at the jobs submitted to your jobtracker and find
out what happened to the pig job (the message " Job terminated with
anomalous status FAILED" leads me to believe there is some job setup
problem). Do you see any pig job attempts on your jobtracker UI?
D
On Fri, Apr 1, 2011 at 2:16 PM, Dimitris Iliopoulos <
dimitris.iliopoulos@gmail.com> wrote:
> Currently using Pig 0.8 and Hadoop 0.20.2. I'm able to run things in local
> mode as well as run fs -* commands from within Grunt running in MapReduce
> mode. I can't seem to be able to execute any Pig Latin statement in
> MapReduce mode. The fact that I can browse HDFS and access files leads me
> to
> believe that the setup is correct (also verified by the correct namenode
> and
> jobtracker addresses connected to when firing off any pig command). I don't
> know why though I can't run anything else. A simple command like the
> following:
>
> data = LOAD 'test.csv' USING PigStorage(',');
> DUMP data;
>
> Produces the following stack trace:
>
> Pig Stack Trace
> ---------------
> ERROR 1066: Unable to open iterator for alias data
>
> org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> open iterator for alias data
> at org.apache.pig.PigServer.openIterator(PigServer.java:754)
> at
> org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:612)
> at
>
> org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:303)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:165)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:76)
> at org.apache.pig.Main.run(Main.java:465)
> at org.apache.pig.Main.main(Main.java:107)
> Caused by: java.io.IOException: Job terminated with anomalous status FAILED
> at org.apache.pig.PigServer.openIterator(PigServer.java:744)
> ... 7 more
>
> Any other more elaborate collection of Pig statements will produce this:
>
> Pig Stack Trace
> ---------------
> ERROR 2244: Job failed, hadoop does not return any error message
>
> org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job
> failed, hadoop does not return any error message
> at
> org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:116)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
> at
>
> org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:141)
> at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:90)
> at org.apache.pig.Main.run(Main.java:510)
> at org.apache.pig.Main.main(Main.java:107)
>
> Where any input file is flagged as unreadable by the following error
> message
> during execution:
>
> Input(s):
> Failed to read data from "test.csv"
>
> All files are accessible in MapReduce mode via cat, ls etc. I might be
> missing something very obvious here since this is the first time I've used
> Pig so any help would be greatly appreciated.
>
> - Dimitris
>