You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@pig.apache.org by Billy Watson <wi...@gmail.com> on 2016/03/10 17:11:04 UTC
HBaseStorage Merge Join NoClassDef Error
Thanks to a bug fix put in by a colleague of mine, merge joins work for
tables loaded into pig via HBaseStorage. In our test environment and in the
test environment for pig itself, I'm able to get all sorts of fairly
complex data merging without issue.
However, when I use that same code on larger data sets in a production
environment, the merge join fails. If I run it on the same exact tables on
the same cluster after trimming the data down to just a few rows, the merge
join works fine.
Here is the most basic I've been able to get the pig script. I've been
taking out pieces and parts trying to narrow it down but it still fails:
If I change the count portion to a limit 5 or something, I'm able to dump
the relation.
The merge join finishes all of its mappers, but when it gets to the reduce
step and starts doing a sort (don't ask me why it's even doing a sort on
pre-sorted data), it throws the following error:
2016-03-09 19:36:01,738 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: Error
while doing final merge
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:160)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.ClassCastException:
org.apache.pig.backend.hadoop.hbase.TableSplitComparable cannot be
cast to org.apache.hadoop.hbase.mapreduce.TableSplit
at org.apache.pig.backend.hadoop.hbase.TableSplitComparable.compareTo(TableSplitComparable.java:26)
at org.apache.pig.data.DataType.compare(DataType.java:566)
at org.apache.pig.data.DataType.compare(DataType.java:464)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareDatum(BinInterSedes.java:1106)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:1082)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:787)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:728)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTupleSortComparator.compare(PigTupleSortComparator.java:100)
at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:587)
at org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:128)
at org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:55)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:678)
at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:596)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:131)
at org.apache.hadoop.mapred.Merger.merge(Merger.java:115)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.finalMerge(MergeManagerImpl.java:722)
at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.close(MergeManagerImpl.java:370)
at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:158)
If I switch the order of the two relations in the merge join, I get a
different error which appears more promising, but I still don't know what
to do about it:
2016-03-09 19:55:24,789 WARN [main]
org.apache.hadoop.mapred.YarnChild: Exception running child :
org.apache.pig.backend.executionengine.ExecException: ERROR 0:
Exception while executing (Name: c: Local
Rearrange[tuple]{chararray}(false) - scope-334 Operator Key:
scope-334): org.apache.pig.backend.executionengine.ExecException:
ERROR 0: Error while executing ForEach at [c[62,4]]
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:316)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:291)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:279)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:274)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR
0: Error while executing ForEach at [c[62,4]]
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:325)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
... 12 more
Caused by: java.lang.NullPointerException
at org.apache.pig.impl.builtin.DefaultIndexableLoader.seekNear(DefaultIndexableLoader.java:190)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.seekInRightStream(POMergeJoin.java:542)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNextTuple(POMergeJoin.java:299)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPreCombinerLocalRearrange.getNextTuple(POPreCombinerLocalRearrange.java:126)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:252)
Again, I've tried replicating the exact scenario (and more complicated
ones) in local environments and I can't get it to fail. I think it's
related to yarn/mapreduce, but I can't figure out why that would matter or
what it's really doing.
I'm trying to set up the e2e (end to end) tests in the pig repo, but I'm
not having any luck there, either. If I can't get a test failure, I'm
afraid I'm not going to be able to fix the bug or issue.
Can anyone help point me in the right direction as far as next debugging
steps or what might be wrong?
William Watson
Lead Software Engineer
Re: HBaseStorage Merge Join NoClassDef Error
Posted by Billy Watson <wi...@gmail.com>.
Sorry to spam, but I also forgot to mention versions of stuff. This fails
in the same way on both:
Hadoop 2.4
HBase 0.98
Pig 0.15.1
and
Hadoop 2.7
HBase 1.1
Pig 0.15.1
William Watson
Lead Software Engineer
On Thu, Mar 10, 2016 at 11:16 AM, Billy Watson <wi...@gmail.com>
wrote:
> Bah, I forgot to paste the pig script like an idiot:
>
> table1 = LOAD 'hbase://table1'
> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
> '',
> '-loadKey -noWAL=true -minTimestamp=1451624400000
> -maxTimestamp=1454302800000') AS
> (uid:chararray);
>
> table2 = LOAD 'hbase://table2'
> USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
> '',
> '-loadKey -noWAL=true -regex=\\\\|ago=156\\\\|
> -minTimestamp=1451624400000 -maxTimestamp=1454302800000') AS
> (uid:chararray);
>
>
> user_segment_with_event = JOIN table1 BY uid, table2 BY uid USING 'merge';
>
> -- fails with TableSplitComparable cannot be cast to TableSplit
> --
> http://ip-10-0-1-180.ec2.internal:19888/jobhistory/logs/ip-10-0-1-14.ec2.internal:45454/container_e10_1457365475473_0248_01_000029/attempt_1457365475473_0248_r_000000_0/hadoop/syslog/?start=0
> ones = FOREACH user_segment_with_event GENERATE (int) 1 AS one:int;
> c = GROUP ones ALL; c = FOREACH c GENERATE COUNT(ones); dump c
>
> William Watson
> Lead Software Engineer
>
> On Thu, Mar 10, 2016 at 11:11 AM, Billy Watson <wi...@gmail.com>
> wrote:
>
>> Thanks to a bug fix put in by a colleague of mine, merge joins work for
>> tables loaded into pig via HBaseStorage. In our test environment and in the
>> test environment for pig itself, I'm able to get all sorts of fairly
>> complex data merging without issue.
>>
>> However, when I use that same code on larger data sets in a production
>> environment, the merge join fails. If I run it on the same exact tables on
>> the same cluster after trimming the data down to just a few rows, the merge
>> join works fine.
>>
>> Here is the most basic I've been able to get the pig script. I've been
>> taking out pieces and parts trying to narrow it down but it still fails:
>>
>>
>>
>> If I change the count portion to a limit 5 or something, I'm able to dump
>> the relation.
>>
>> The merge join finishes all of its mappers, but when it gets to the
>> reduce step and starts doing a sort (don't ask me why it's even doing a
>> sort on pre-sorted data), it throws the following error:
>>
>> 2016-03-09 19:36:01,738 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: Error while doing final merge
>> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:160)
>> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.hbase.TableSplitComparable cannot be cast to org.apache.hadoop.hbase.mapreduce.TableSplit
>> at org.apache.pig.backend.hadoop.hbase.TableSplitComparable.compareTo(TableSplitComparable.java:26)
>> at org.apache.pig.data.DataType.compare(DataType.java:566)
>> at org.apache.pig.data.DataType.compare(DataType.java:464)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareDatum(BinInterSedes.java:1106)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:1082)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:787)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:728)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTupleSortComparator.compare(PigTupleSortComparator.java:100)
>> at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:587)
>> at org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:128)
>> at org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:55)
>> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:678)
>> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:596)
>> at org.apache.hadoop.mapred.Merger.merge(Merger.java:131)
>> at org.apache.hadoop.mapred.Merger.merge(Merger.java:115)
>> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.finalMerge(MergeManagerImpl.java:722)
>> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.close(MergeManagerImpl.java:370)
>> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:158)
>>
>>
>>
>> If I switch the order of the two relations in the merge join, I get a
>> different error which appears more promising, but I still don't know what
>> to do about it:
>>
>> 2016-03-09 19:55:24,789 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: c: Local Rearrange[tuple]{chararray}(false) - scope-334 Operator Key: scope-334): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error while executing ForEach at [c[62,4]]
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:316)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:291)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:279)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:274)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
>> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error while executing ForEach at [c[62,4]]
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:325)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
>> ... 12 more
>> Caused by: java.lang.NullPointerException
>> at org.apache.pig.impl.builtin.DefaultIndexableLoader.seekNear(DefaultIndexableLoader.java:190)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.seekInRightStream(POMergeJoin.java:542)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNextTuple(POMergeJoin.java:299)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPreCombinerLocalRearrange.getNextTuple(POPreCombinerLocalRearrange.java:126)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
>> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:252)
>>
>>
>> Again, I've tried replicating the exact scenario (and more complicated
>> ones) in local environments and I can't get it to fail. I think it's
>> related to yarn/mapreduce, but I can't figure out why that would matter or
>> what it's really doing.
>>
>> I'm trying to set up the e2e (end to end) tests in the pig repo, but I'm
>> not having any luck there, either. If I can't get a test failure, I'm
>> afraid I'm not going to be able to fix the bug or issue.
>>
>> Can anyone help point me in the right direction as far as next debugging
>> steps or what might be wrong?
>>
>>
>> William Watson
>> Lead Software Engineer
>>
>
>
Re: HBaseStorage Merge Join NoClassDef Error
Posted by Billy Watson <wi...@gmail.com>.
Bah, I forgot to paste the pig script like an idiot:
table1 = LOAD 'hbase://table1'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'',
'-loadKey -noWAL=true -minTimestamp=1451624400000
-maxTimestamp=1454302800000') AS
(uid:chararray);
table2 = LOAD 'hbase://table2'
USING org.apache.pig.backend.hadoop.hbase.HBaseStorage(
'',
'-loadKey -noWAL=true -regex=\\\\|ago=156\\\\| -minTimestamp=1451624400000
-maxTimestamp=1454302800000') AS
(uid:chararray);
user_segment_with_event = JOIN table1 BY uid, table2 BY uid USING 'merge';
-- fails with TableSplitComparable cannot be cast to TableSplit
--
http://ip-10-0-1-180.ec2.internal:19888/jobhistory/logs/ip-10-0-1-14.ec2.internal:45454/container_e10_1457365475473_0248_01_000029/attempt_1457365475473_0248_r_000000_0/hadoop/syslog/?start=0
ones = FOREACH user_segment_with_event GENERATE (int) 1 AS one:int;
c = GROUP ones ALL; c = FOREACH c GENERATE COUNT(ones); dump c
William Watson
Lead Software Engineer
On Thu, Mar 10, 2016 at 11:11 AM, Billy Watson <wi...@gmail.com>
wrote:
> Thanks to a bug fix put in by a colleague of mine, merge joins work for
> tables loaded into pig via HBaseStorage. In our test environment and in the
> test environment for pig itself, I'm able to get all sorts of fairly
> complex data merging without issue.
>
> However, when I use that same code on larger data sets in a production
> environment, the merge join fails. If I run it on the same exact tables on
> the same cluster after trimming the data down to just a few rows, the merge
> join works fine.
>
> Here is the most basic I've been able to get the pig script. I've been
> taking out pieces and parts trying to narrow it down but it still fails:
>
>
>
> If I change the count portion to a limit 5 or something, I'm able to dump
> the relation.
>
> The merge join finishes all of its mappers, but when it gets to the reduce
> step and starts doing a sort (don't ask me why it's even doing a sort on
> pre-sorted data), it throws the following error:
>
> 2016-03-09 19:36:01,738 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.hadoop.mapreduce.task.reduce.Shuffle$ShuffleError: Error while doing final merge
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:160)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:376)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassCastException: org.apache.pig.backend.hadoop.hbase.TableSplitComparable cannot be cast to org.apache.hadoop.hbase.mapreduce.TableSplit
> at org.apache.pig.backend.hadoop.hbase.TableSplitComparable.compareTo(TableSplitComparable.java:26)
> at org.apache.pig.data.DataType.compare(DataType.java:566)
> at org.apache.pig.data.DataType.compare(DataType.java:464)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareDatum(BinInterSedes.java:1106)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:1082)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:787)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:728)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTupleSortComparator.compare(PigTupleSortComparator.java:100)
> at org.apache.hadoop.mapred.Merger$MergeQueue.lessThan(Merger.java:587)
> at org.apache.hadoop.util.PriorityQueue.upHeap(PriorityQueue.java:128)
> at org.apache.hadoop.util.PriorityQueue.put(PriorityQueue.java:55)
> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:678)
> at org.apache.hadoop.mapred.Merger$MergeQueue.merge(Merger.java:596)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:131)
> at org.apache.hadoop.mapred.Merger.merge(Merger.java:115)
> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.finalMerge(MergeManagerImpl.java:722)
> at org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl.close(MergeManagerImpl.java:370)
> at org.apache.hadoop.mapreduce.task.reduce.Shuffle.run(Shuffle.java:158)
>
>
>
> If I switch the order of the two relations in the merge join, I get a
> different error which appears more promising, but I still don't know what
> to do about it:
>
> 2016-03-09 19:55:24,789 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing (Name: c: Local Rearrange[tuple]{chararray}(false) - scope-334 Operator Key: scope-334): org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error while executing ForEach at [c[62,4]]
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:316)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POLocalRearrange.getNextTuple(POLocalRearrange.java:291)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:279)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:274)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Error while executing ForEach at [c[62,4]]
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:325)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
> ... 12 more
> Caused by: java.lang.NullPointerException
> at org.apache.pig.impl.builtin.DefaultIndexableLoader.seekNear(DefaultIndexableLoader.java:190)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.seekInRightStream(POMergeJoin.java:542)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POMergeJoin.getNextTuple(POMergeJoin.java:299)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POPreCombinerLocalRearrange.getNextTuple(POPreCombinerLocalRearrange.java:126)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.processInput(PhysicalOperator.java:307)
> at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:252)
>
>
> Again, I've tried replicating the exact scenario (and more complicated
> ones) in local environments and I can't get it to fail. I think it's
> related to yarn/mapreduce, but I can't figure out why that would matter or
> what it's really doing.
>
> I'm trying to set up the e2e (end to end) tests in the pig repo, but I'm
> not having any luck there, either. If I can't get a test failure, I'm
> afraid I'm not going to be able to fix the bug or issue.
>
> Can anyone help point me in the right direction as far as next debugging
> steps or what might be wrong?
>
>
> William Watson
> Lead Software Engineer
>