[jira] [Commented] (MAPREDUCE-4369) Fix streaming job failures with WindowsResourceCalculatorPlugin

["Ivan Mitic (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-4369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402792#comment-13402792 ] 

Ivan Mitic commented on MAPREDUCE-4369:
---------------------------------------

Thanks for the change Bikas!

A few questions/suggestions:
1. In {{WindowsResourceCalculatorPlugin#getProcResourceValues()}} you mention that some tests use JVM_PID. Do you happen to have a list of these tests?
2. Can you please refactor {{ResourceCalculatorPlugin#getResourceCalculatorPlugin()}} to accept processPid, and update call sites to pass the appropriate value (I see only 3 call sites). The cause of this bug in the first place is not having all call sites set the processPid accordingly. And then, if the passed-in processPid is null, you can fallback to {{System.getenv().get("JVM_PID")}}. Make sense? If I'm seeing things correctly, this way you might be able to clean up some of the newly introduced code.

                
> Fix streaming job failures with WindowsResourceCalculatorPlugin
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-4369
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4369
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Bikas Saha
>            Assignee: Bikas Saha
>         Attachments: MAPREDUCE-4369.branch-1-win.1.patch
>
>
> Some streaming jobs use local mode job runs that do not start tasks trackers. In these cases, the jvm context is not setup and hence local mode execution causes the code to crash.
> Fix is to not not use ResourceCalculatorPlugin in such cases or make the local job run creating dummy jvm contexts. Choosing the first option because thats the current implicit behavior in Linux. The ProcfsBasedProcessTree (used inside the LinuxResourceCalculatorPlugin) does no real work when the process pid is not setup correctly. This is what happens when local job mode runs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira