You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Rohini Palaniswamy (JIRA)" <ji...@apache.org> on 2019/06/24 22:13:00 UTC
[jira] [Commented] (PIG-5380) SortedDataBag hitting
ConcurrentModificationException or producing incorrect output in a
corner-case
[ https://issues.apache.org/jira/browse/PIG-5380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16871829#comment-16871829 ]
Rohini Palaniswamy commented on PIG-5380:
-----------------------------------------
Patch looks good. I think same change is required in InternalSortedBag as well as code is exactly same and it can spill too - https://github.com/apache/pig/blob/trunk/src/org/apache/pig/data/InternalSortedBag.java#L133 . We most likely haven't seen issues with it as the probability could be very less as it will proactively spill if it exceeds cached memory limit.
> SortedDataBag hitting ConcurrentModificationException or producing incorrect output in a corner-case
> -----------------------------------------------------------------------------------------------------
>
> Key: PIG-5380
> URL: https://issues.apache.org/jira/browse/PIG-5380
> Project: Pig
> Issue Type: Bug
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Major
> Attachments: pig-5380-v01.patch
>
>
> User had a UDF that created large SortedDataBag. This UDF was failing with
> {noformat}
> java.util.ConcurrentModificationException
> at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:901)
> at java.util.ArrayList$Itr.next(ArrayList.java:851)
> at org.apache.pig.data.SortedDataBag$SortedDataBagIterator.readFromPriorityQ(SortedDataBag.java:346)
> at org.apache.pig.data.SortedDataBag$SortedDataBagIterator.next(SortedDataBag.java:322)
> at org.apache.pig.data.SortedDataBag$SortedDataBagIterator.hasNext(SortedDataBag.java:235)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)