You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Ankur Kasliwal <an...@gmail.com> on 2015/05/15 23:01:20 UTC
pig UDF error
Hi All,
I am trying to process sequence file using Pig.
Sequence file is provided as input to pig script.
A UDF is written which basically does the processing of meta data and
actual file data.
This UDF returns input sequence file in form of tuples (metadata, actual
file ) as output.
This is working fine on Hadoop :
· Hadoop : 1.0.3-gphd-1.2.0.0 and
· pig version : Apache Pig version 0.9.2-gphd-1.2.0.0
But when I am trying to run same on following version of Hadoop and pig it
is failing with error as below:
· Hadoop : Hadoop 2.2.0-gphd-3.1.0.0
· Pig version : Apache Pig version 0.12.0-gphd-3.1.0.0
Error:
Pig Stack Trace
---------------
ERROR 1066: Unable to open iterator for alias limited. Backend error :
Exception while executing [POUserFunc (Name:
POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
Operator Key: scope-3) children: null at []]: java.lang.NullPointerException
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
open iterator for alias limited. Backend error : Exception while executing
[POUserFunc (Name:
POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
Operator Key: scope-3) children: null at []]: java.lang.NullPointerException
Any pointers on this will be helpful.
I can see several posts on similar issue by many people on google but no
posted solution.
Thanks,
Ankur
Re: pig UDF error
Posted by Daniel Dai <da...@hortonworks.com>.
The stack is in the custom code. Do you have source code for TupleBuilder?
You may find some clue there.
On 5/18/15, 7:22 AM, "Ankur Kasliwal" <an...@gmail.com> wrote:
>Unfortunately i am not able to access Hadoop WebUI,
>but just for reference this is complete error message from the log.
>
>let me know if this gives any idea about the issue.
>
>Thanks,
>Ankur
>
>
>Backend error message
>
>---------------------
>
>AttemptID:attempt_1431714308706_0002_m_000000_0 Info:Container killed by
>the ApplicationMaster.
>
>Container killed on request. Exit code is 143
>
>
>
>Backend error message
>
>---------------------
>
>AttemptID:attempt_1431714308706_0002_r_000000_0 Info:Error:
>org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception
>while executing [POUserFunc (Name:
>POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
>Operator Key: scope-3) children: null at []]:
>java.lang.NullPointerException
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
>or.getNext(PhysicalOperator.java:338)
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
>ators.POForEach.processPlan(POForEach.java:378)
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOper
>ators.POForEach.getNextTuple(POForEach.java:298)
>
> at
>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
>Reduce$Reduce.runPipeline(PigGenericMapReduce.java:464)
>
> at
>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
>Reduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432)
>
> at
>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
>Reduce$Reduce.reduce(PigGenericMapReduce.java:412)
>
> at
>org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMap
>Reduce$Reduce.reduce(PigGenericMapReduce.java:256)
>
> at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
>
> at
>org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:645)
>
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:405)
>
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:415)
>
> at
>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.
>java:1491)
>
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
>
>Caused by: java.lang.NullPointerException
>
> at
>com.company.bigdata.serde.utility.TupleBuilder.generateOperationalMetadata
>List(TupleBuilder.java:76)
>
> at
>com.company.bigdata.serde.utility.TupleBuilder.buildTuple(TupleBuilder.jav
>a:41)
>
> at
>com.company.bigdata.serde.Deserializer.exec(Deserializer.java:28)
>
> at
>com.company.bigdata.serde.Deserializer.exec(Deserializer.java:14)
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
>ators.POUserFunc.getNext(POUserFunc.java:330)
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOper
>ators.POUserFunc.getNextTuple(POUserFunc.java:369)
>
> at
>org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperat
>or.getNext(PhysicalOperator.java:333)
>
> ... 14 more
>
>On Mon, May 18, 2015 at 2:05 AM, Daniel Dai <da...@hortonworks.com> wrote:
>
>> You need to grab the full stack from the Hadoop WebUI. Without the
>>stack,
>> I have very little clue.
>>
>> Daniel
>>
>> On 5/15/15, 2:01 PM, "Ankur Kasliwal" <an...@gmail.com>
>>wrote:
>>
>> >Hi All,
>> >
>> >
>> >
>> >I am trying to process sequence file using Pig.
>> >
>> >Sequence file is provided as input to pig script.
>> >
>> >A UDF is written which basically does the processing of meta data and
>> >actual file data.
>> >
>> >This UDF returns input sequence file in form of tuples (metadata,
>>actual
>> >file ) as output.
>> >
>> >
>> >
>> >This is working fine on Hadoop :
>> >
>> >· Hadoop : 1.0.3-gphd-1.2.0.0 and
>> >
>> >· pig version : Apache Pig version 0.9.2-gphd-1.2.0.0
>> >
>> >
>> >
>> >But when I am trying to run same on following version of Hadoop and
>>pig it
>> >is failing with error as below:
>> >
>> >
>> >
>> >· Hadoop : Hadoop 2.2.0-gphd-3.1.0.0
>> >
>> >· Pig version : Apache Pig version 0.12.0-gphd-3.1.0.0
>> >
>> >
>> >
>> >Error:
>> >
>> >
>> >
>> >Pig Stack Trace
>> >
>> >---------------
>> >
>> >ERROR 1066: Unable to open iterator for alias limited. Backend error :
>> >Exception while executing [POUserFunc (Name:
>> >POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
>> >Operator Key: scope-3) children: null at []]:
>> >java.lang.NullPointerException
>> >
>> >
>> >
>> >org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable
>>to
>> >open iterator for alias limited. Backend error : Exception while
>>executing
>> >[POUserFunc (Name:
>> >POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
>> >Operator Key: scope-3) children: null at []]:
>> >java.lang.NullPointerException
>> >
>> >
>> >
>> >Any pointers on this will be helpful.
>> >
>> >I can see several posts on similar issue by many people on google but
>>no
>> >posted solution.
>> >
>> >
>> >
>> >Thanks,
>> >
>> >Ankur
>>
>>
Re: pig UDF error
Posted by Ankur Kasliwal <an...@gmail.com>.
Unfortunately i am not able to access Hadoop WebUI,
but just for reference this is complete error message from the log.
let me know if this gives any idea about the issue.
Thanks,
Ankur
Backend error message
---------------------
AttemptID:attempt_1431714308706_0002_m_000000_0 Info:Container killed by
the ApplicationMaster.
Container killed on request. Exit code is 143
Backend error message
---------------------
AttemptID:attempt_1431714308706_0002_r_000000_0 Info:Error:
org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception
while executing [POUserFunc (Name:
POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
Operator Key: scope-3) children: null at []]: java.lang.NullPointerException
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:378)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:298)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:464)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:412)
at
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:645)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:405)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.lang.NullPointerException
at
com.company.bigdata.serde.utility.TupleBuilder.generateOperationalMetadataList(TupleBuilder.java:76)
at
com.company.bigdata.serde.utility.TupleBuilder.buildTuple(TupleBuilder.java:41)
at com.company.bigdata.serde.Deserializer.exec(Deserializer.java:28)
at com.company.bigdata.serde.Deserializer.exec(Deserializer.java:14)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:330)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNextTuple(POUserFunc.java:369)
at
org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:333)
... 14 more
On Mon, May 18, 2015 at 2:05 AM, Daniel Dai <da...@hortonworks.com> wrote:
> You need to grab the full stack from the Hadoop WebUI. Without the stack,
> I have very little clue.
>
> Daniel
>
> On 5/15/15, 2:01 PM, "Ankur Kasliwal" <an...@gmail.com> wrote:
>
> >Hi All,
> >
> >
> >
> >I am trying to process sequence file using Pig.
> >
> >Sequence file is provided as input to pig script.
> >
> >A UDF is written which basically does the processing of meta data and
> >actual file data.
> >
> >This UDF returns input sequence file in form of tuples (metadata, actual
> >file ) as output.
> >
> >
> >
> >This is working fine on Hadoop :
> >
> >· Hadoop : 1.0.3-gphd-1.2.0.0 and
> >
> >· pig version : Apache Pig version 0.9.2-gphd-1.2.0.0
> >
> >
> >
> >But when I am trying to run same on following version of Hadoop and pig it
> >is failing with error as below:
> >
> >
> >
> >· Hadoop : Hadoop 2.2.0-gphd-3.1.0.0
> >
> >· Pig version : Apache Pig version 0.12.0-gphd-3.1.0.0
> >
> >
> >
> >Error:
> >
> >
> >
> >Pig Stack Trace
> >
> >---------------
> >
> >ERROR 1066: Unable to open iterator for alias limited. Backend error :
> >Exception while executing [POUserFunc (Name:
> >POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
> >Operator Key: scope-3) children: null at []]:
> >java.lang.NullPointerException
> >
> >
> >
> >org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
> >open iterator for alias limited. Backend error : Exception while executing
> >[POUserFunc (Name:
> >POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
> >Operator Key: scope-3) children: null at []]:
> >java.lang.NullPointerException
> >
> >
> >
> >Any pointers on this will be helpful.
> >
> >I can see several posts on similar issue by many people on google but no
> >posted solution.
> >
> >
> >
> >Thanks,
> >
> >Ankur
>
>
Re: pig UDF error
Posted by Daniel Dai <da...@hortonworks.com>.
You need to grab the full stack from the Hadoop WebUI. Without the stack,
I have very little clue.
Daniel
On 5/15/15, 2:01 PM, "Ankur Kasliwal" <an...@gmail.com> wrote:
>Hi All,
>
>
>
>I am trying to process sequence file using Pig.
>
>Sequence file is provided as input to pig script.
>
>A UDF is written which basically does the processing of meta data and
>actual file data.
>
>This UDF returns input sequence file in form of tuples (metadata, actual
>file ) as output.
>
>
>
>This is working fine on Hadoop :
>
>· Hadoop : 1.0.3-gphd-1.2.0.0 and
>
>· pig version : Apache Pig version 0.9.2-gphd-1.2.0.0
>
>
>
>But when I am trying to run same on following version of Hadoop and pig it
>is failing with error as below:
>
>
>
>· Hadoop : Hadoop 2.2.0-gphd-3.1.0.0
>
>· Pig version : Apache Pig version 0.12.0-gphd-3.1.0.0
>
>
>
>Error:
>
>
>
>Pig Stack Trace
>
>---------------
>
>ERROR 1066: Unable to open iterator for alias limited. Backend error :
>Exception while executing [POUserFunc (Name:
>POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
>Operator Key: scope-3) children: null at []]:
>java.lang.NullPointerException
>
>
>
>org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to
>open iterator for alias limited. Backend error : Exception while executing
>[POUserFunc (Name:
>POUserFunc(com.company.bigdata.serde.Deserializer)[tuple] - scope-3
>Operator Key: scope-3) children: null at []]:
>java.lang.NullPointerException
>
>
>
>Any pointers on this will be helpful.
>
>I can see several posts on similar issue by many people on google but no
>posted solution.
>
>
>
>Thanks,
>
>Ankur