[jira] [Commented] (PIG-2812) Spill InternalCachedBag into only 1 file

["Haitao Yao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-2812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13432882#comment-13432882 ] 

Haitao Yao commented on PIG-2812:
---------------------------------

@Alan Gates  
They are cleared in the clear method, but in case some leaks happens, the deleteOnExit is required. Even if the file is deleted, the file path is still stored in java.io.DeleteOnExitHook and still you may get OOM because of this.

                
> Spill InternalCachedBag into only 1 file
> ----------------------------------------
>
>                 Key: PIG-2812
>                 URL: https://issues.apache.org/jira/browse/PIG-2812
>             Project: Pig
>          Issue Type: Bug
>          Components: data
>            Reporter: Haitao Yao
>             Fix For: 0.11
>
>         Attachments: aa.jpg, spill.patch
>
>
> I encountered a reducer's OOM because of java.io.DeleteOnExitHook. And I found out that the InternalCachedBag creates a seperate tmp file, and the tmp files is deleted on exit. So the file delete hook caused the OOM. 
> Why not just hold the tmp file handle and spill only one tmp file?
> Too many tmp files may block the tasktracker start process, if the tmp files are not cleaned on time and the tasktracker restarts at this specific time.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira