[jira] [Updated] (PIG-4564) Pig can deadlock in POPartialAgg if there is a bag

["Rohini Palaniswamy (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PIG-4564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Rohini Palaniswamy updated PIG-4564:
------------------------------------
    Attachment: PIG-4564-1.patch

This patch also incorporates fix for PIG-4012. 

The problem is we need to have POPartialAgg to be synchronous spill so that its memory consumption is reduced before we invoke System.gc() in SpillableMemoryManager. But for POPartialAgg to do the aggregation, it needs to read the next record which can block on registerSpillable if there is a bag in the record causing deadlock. This patch allows registering to the spillables if doing POPartialAgg. For other normal spills it just blocks as before. registerSpillable is still synchronized on spillables to avoid adding to spillables when the spillablesSR is being constructed. But it mainly relies on blockRegisterOnSpill variable to block from adding to spillables when the actual spill is in progress.

> Pig can deadlock in POPartialAgg if there is a bag
> --------------------------------------------------
>
>                 Key: PIG-4564
>                 URL: https://issues.apache.org/jira/browse/PIG-4564
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.15.0
>
>         Attachments: PIG-4564-1.patch
>
>
> PIG-3979 made spill of POPartialAgg synchronous, but if there is a bag in the data being grouped then it can deadlock when that tries to register the Spillable with SpillableMemoryManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)