[jira] Commented: (PIG-1838) On a large farm, some pigs die of /tmp starvation

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-1838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989481#comment-12989481 ] 

Allen Wittenauer commented on PIG-1838:
---------------------------------------

I have a few thoughts on a better/programmatic way for Pig to be better behaved without depending on users doing the right thing. One or more of these would probably work:

a) Redefine java.io.tmpdir itself after it gets the Hadoop property files loaded
b) In the pig wrapper script, parse mapred-site.xml and pull out the mapred tmp space
c) Override Java's createTempFile method to use Hadoop's tmp location/$TEMPDIR/$TMPDIR/$TEMP/some other value
d) Change the jar assembly such that it goes into a create->submit->delete->repeat pattern.  (From a casual glance, it appears to create all the jars at once rather than just when needed.)



> On a large farm, some pigs die of /tmp starvation
> -------------------------------------------------
>
>                 Key: PIG-1838
>                 URL: https://issues.apache.org/jira/browse/PIG-1838
>             Project: Pig
>          Issue Type: Wish
>          Components: impl
>    Affects Versions: 0.8.0
>            Reporter: Allen Wittenauer
>
> We're starting to issues where interactive/command line pig users blow up due to so many large jar creations in /tmp. (In other words, pig execution prior to the java.io.tmpdir fix that Hadoop makes can kick in.)  Pig should probably not depend upon users being savvy enough to override java.io.tmpdir on their own in these situations and/or a better steward of the space it does use.  

-- 
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira