You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Prasanth J <bu...@gmail.com> on 2012/10/08 05:15:09 UTC
BinInterSedes comparator failing for tuple
Hello everyone
I recently updated my patch(PIG-2831) to work on the latest trunk and noticed that new trunk breaks my code. My code uses secondary sort on a tuple inside a databag. The issue seems to be related to BinInterSedes comparator. I have pasted the stack trace for reference. Can anyone let me know if this is side-effect of some other patch that is committed recently?
java.io.IOException: Spill failed
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Unexpected data type 39 found in stream.
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
Thanks
-- Prasanth
Re: BinInterSedes comparator failing for tuple
Posted by Prasanth J <bu...@gmail.com>.
Thanks for fixing this Jon. It works now.
Thanks
-- Prasanth
On Oct 8, 2012, at 2:15 PM, Jonathan Coveney <jc...@gmail.com> wrote:
> Prashanth,
>
> The fix is in. Please let me know if you have any more issues
>
> Jon
>
> 2012/10/7 Prasanth J <bu...@gmail.com>
>
>> Thanks Jon.
>>
>> Thanks
>> -- Prasanth
>>
>> On Oct 7, 2012, at 11:46 PM, Jon Coveney <jc...@gmail.com> wrote:
>>
>>> My guess is this is my doing. Will tackle this tomorrow.
>>>
>>> Sent from my iPhone
>>>
>>> On Oct 7, 2012, at 8:15 PM, Prasanth J <bu...@gmail.com>
>> wrote:
>>>
>>>> Hello everyone
>>>>
>>>> I recently updated my patch(PIG-2831) to work on the latest trunk and
>> noticed that new trunk breaks my code. My code uses secondary sort on a
>> tuple inside a databag. The issue seems to be related to BinInterSedes
>> comparator. I have pasted the stack trace for reference. Can anyone let me
>> know if this is side-effect of some other patch that is committed recently?
>>>>
>>>> java.io.IOException: Spill failed
>>>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
>>>> at
>> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
>>>> at
>> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>>>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
>>>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
>>>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>>>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>>>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>>>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>> at javax.security.auth.Subject.doAs(Subject.java:396)
>>>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>>>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>>>> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in
>> stream.
>>>> at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
>>>> at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
>>>> at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
>>>> at
>> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
>>>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
>>>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
>>>> at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
>>>> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
>>>> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>>>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
>>>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
>>>> at
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
>>>>
>>>>
>>>> Thanks
>>>> -- Prasanth
>>>>
>>
>>
Re: BinInterSedes comparator failing for tuple
Posted by Jonathan Coveney <jc...@gmail.com>.
Prashanth,
The fix is in. Please let me know if you have any more issues
Jon
2012/10/7 Prasanth J <bu...@gmail.com>
> Thanks Jon.
>
> Thanks
> -- Prasanth
>
> On Oct 7, 2012, at 11:46 PM, Jon Coveney <jc...@gmail.com> wrote:
>
> > My guess is this is my doing. Will tackle this tomorrow.
> >
> > Sent from my iPhone
> >
> > On Oct 7, 2012, at 8:15 PM, Prasanth J <bu...@gmail.com>
> wrote:
> >
> >> Hello everyone
> >>
> >> I recently updated my patch(PIG-2831) to work on the latest trunk and
> noticed that new trunk breaks my code. My code uses secondary sort on a
> tuple inside a databag. The issue seems to be related to BinInterSedes
> comparator. I have pasted the stack trace for reference. Can anyone let me
> know if this is side-effect of some other patch that is committed recently?
> >>
> >> java.io.IOException: Spill failed
> >> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> >> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> >> at
> org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> >> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
> >> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
> >> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> >> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> >> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> >> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> >> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> >> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at javax.security.auth.Subject.doAs(Subject.java:396)
> >> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> >> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> >> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in
> stream.
> >> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
> >> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
> >> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
> >> at
> org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
> >> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
> >> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
> >> at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
> >> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
> >> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> >> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
> >> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> >> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
> >>
> >>
> >> Thanks
> >> -- Prasanth
> >>
>
>
Re: BinInterSedes comparator failing for tuple
Posted by Prasanth J <bu...@gmail.com>.
Thanks Jon.
Thanks
-- Prasanth
On Oct 7, 2012, at 11:46 PM, Jon Coveney <jc...@gmail.com> wrote:
> My guess is this is my doing. Will tackle this tomorrow.
>
> Sent from my iPhone
>
> On Oct 7, 2012, at 8:15 PM, Prasanth J <bu...@gmail.com> wrote:
>
>> Hello everyone
>>
>> I recently updated my patch(PIG-2831) to work on the latest trunk and noticed that new trunk breaks my code. My code uses secondary sort on a tuple inside a databag. The issue seems to be related to BinInterSedes comparator. I have pasted the stack trace for reference. Can anyone let me know if this is side-effect of some other patch that is committed recently?
>>
>> java.io.IOException: Spill failed
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
>> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
>> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
>> at org.apache.hadoop.mapred.Child.main(Child.java:249)
>> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in stream.
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
>> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
>> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
>> at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
>> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
>> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
>> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
>>
>>
>> Thanks
>> -- Prasanth
>>
Re: BinInterSedes comparator failing for tuple
Posted by Jon Coveney <jc...@gmail.com>.
My guess is this is my doing. Will tackle this tomorrow.
Sent from my iPhone
On Oct 7, 2012, at 8:15 PM, Prasanth J <bu...@gmail.com> wrote:
> Hello everyone
>
> I recently updated my patch(PIG-2831) to work on the latest trunk and noticed that new trunk breaks my code. My code uses secondary sort on a tuple inside a databag. The issue seems to be related to BinInterSedes comparator. I have pasted the stack trace for reference. Can anyone let me know if this is side-effect of some other patch that is committed recently?
>
> java.io.IOException: Spill failed
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1029)
> at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:691)
> at org.apache.hadoop.mapreduce.TaskInputOutputContext.write(TaskInputOutputContext.java:80)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map.collect(PigGenericMapReduce.java:123)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:285)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
> at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1083)
> at org.apache.hadoop.mapred.Child.main(Child.java:249)
> Caused by: java.lang.RuntimeException: Unexpected data type 39 found in stream.
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.getGeneralizedDataType(BinInterSedes.java:1166)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinInterSedesDatum(BinInterSedes.java:911)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compareBinSedesTuple(BinInterSedes.java:730)
> at org.apache.pig.data.BinInterSedes$BinInterSedesTupleRawComparator.compare(BinInterSedes.java:695)
> at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigSecondaryKeyComparator.compare(PigSecondaryKeyComparator.java:78)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.compare(MapTask.java:1111)
> at org.apache.hadoop.util.QuickSort.fix(QuickSort.java:30)
> at org.apache.hadoop.util.QuickSort.sortInternal(QuickSort.java:83)
> at org.apache.hadoop.util.QuickSort.sort(QuickSort.java:59)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1399)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.access$1800(MapTask.java:853)
> at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$SpillThread.run(MapTask.java:1344)
>
>
> Thanks
> -- Prasanth
>