[jira] [Comment Edited] (PIG-4922) Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15317274#comment-15317274 ] 

Rohini Palaniswamy edited comment on PIG-4922 at 6/6/16 9:33 PM:
-----------------------------------------------------------------

Moved the sorting of the list that does getMemorySize outside synchronized(spillables) block as we have already made the copy of the list to spillablesSR. 

getMemorySize has always been in synchronized(spillables). But seeing the issue for the first time now. Most likely because in Tez, the input and output take more memory than mapreduce and making it hit the spill code. 


was (Author: rohini):
Moved the sorting of the list that does getMemorySize outside synchronized(spillables) block as we have already made the copy of the list to spillablesSR. 

> Deadlock between SpillableMemoryManager and InternalSortedBag$SortedDataBagIterator
> -----------------------------------------------------------------------------------
>
>                 Key: PIG-4922
>                 URL: https://issues.apache.org/jira/browse/PIG-4922
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.17.0, 0.16.1
>
>         Attachments: PIG-4922-1.patch
>
>
>   This one ran into a deadlock, when the data was really huge and InternalSortedBag was reading spilled data from disk.
> {code}
> grpd = FOREACH (GROUP data BY $0){
>     sorted = ORDER data BY timestamp DESC;
>     latest = LIMIT sorted 1;
>     GENERATE latest;
> };
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)