You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2019/03/21 21:27:00 UTC

[jira] [Updated] (PIG-5384) OOM while spilling large bag

     [ https://issues.apache.org/jira/browse/PIG-5384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Koji Noguchi updated PIG-5384:
------------------------------
    Attachment: pig-5384-v01-halfway.patch

Attaching {{pig-5384-v01-halfway.patch}} to give you an idea.  I only changed DefaultDataBag but if I were to take this route, I need to make similar changes to other bags.

For handling spill failures, calling System.exit() is the reliable way but I think setting mContents to null would let the reader reliably fail (unless users have a custom Bag that is doing something very unique).

> OOM while spilling large bag 
> -----------------------------
>
>                 Key: PIG-5384
>                 URL: https://issues.apache.org/jira/browse/PIG-5384
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Koji Noguchi
>            Assignee: Koji Noguchi
>            Priority: Major
>         Attachments: pig-5384-v01-halfway.patch
>
>
> One of the common OOM issue in Pig is, Pig hitting OOM while trying to spill a large bag. Current solutions is to give higher heapsize or tweak 
> {noformat}
> pig.spill.memory.usage.threshold.fraction
> pig.spill.collection.threshold.fraction
> pig.spill.unused.memory.threshold.size
> {noformat}
> and make sure spilling starts early enough.  These params are still critical but wondering if any improvement can be made to increase the chances of avoiding OOM while spilling a single large bag.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)