You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by Rohini Palaniswamy <ro...@gmail.com> on 2015/05/23 21:24:49 UTC
Review Request 34635: PIG-4564: Pig can deadlock in POPartialAgg if
there is a bag
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34635/
-----------------------------------------------------------
Review request for pig.
Bugs: PIG-4564
https://issues.apache.org/jira/browse/PIG-4564
Repository: pig
Description
-------
This patch also incorporates PIG-4012 fixing
The problem is we need to have POPartialAgg to be synchronous spill so that its memory consumption is reduced before we invoke System.gc() in SpillableMemoryManager. But for POPartialAgg to do the aggregation, it needs to read the next record which can block on registerSpillable if there is a bag in the record causing deadlock. This patch allows registering to the spillables if doing POPartialAgg. For other normal spills it just blocks as before. registerSpillable is still synchronized on spillables to avoid adding to spillables when the spillablesSR is being constructed. But it mainly relies on blockRegisterOnSpill variable to block from adding to spillables when the actual spill is in progress.
Diffs
-----
http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java 1681368
Diff: https://reviews.apache.org/r/34635/diff/
Testing
-------
Manually tested for a large job with big bags and lot of spills.
Thanks,
Rohini Palaniswamy
Re: Review Request 34635: PIG-4564: Pig can deadlock in POPartialAgg
if there is a bag
Posted by Daniel Dai <da...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34635/#review85079
-----------------------------------------------------------
Ship it!
Ship It!
- Daniel Dai
On May 23, 2015, 7:24 p.m., Rohini Palaniswamy wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34635/
> -----------------------------------------------------------
>
> (Updated May 23, 2015, 7:24 p.m.)
>
>
> Review request for pig.
>
>
> Bugs: PIG-4564
> https://issues.apache.org/jira/browse/PIG-4564
>
>
> Repository: pig
>
>
> Description
> -------
>
> This patch also incorporates PIG-4012 fixing
>
> The problem is we need to have POPartialAgg to be synchronous spill so that its memory consumption is reduced before we invoke System.gc() in SpillableMemoryManager. But for POPartialAgg to do the aggregation, it needs to read the next record which can block on registerSpillable if there is a bag in the record causing deadlock. This patch allows registering to the spillables if doing POPartialAgg. For other normal spills it just blocks as before. registerSpillable is still synchronized on spillables to avoid adding to spillables when the spillablesSR is being constructed. But it mainly relies on blockRegisterOnSpill variable to block from adding to spillables when the actual spill is in progress.
>
>
> Diffs
> -----
>
> http://svn.apache.org/repos/asf/pig/trunk/src/org/apache/pig/impl/util/SpillableMemoryManager.java 1681368
>
> Diff: https://reviews.apache.org/r/34635/diff/
>
>
> Testing
> -------
>
> Manually tested for a large job with big bags and lot of spills.
>
>
> Thanks,
>
> Rohini Palaniswamy
>
>