[jira] [Commented] (PIG-4555) Add -XX:+UseNUMA for Tez jobs

["Remi Catherinot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PIG-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14546642#comment-14546642 ] 

Remi Catherinot commented on PIG-4555:
--------------------------------------

note all computers have NUMA architectures and also -XX-like options are not always the same for all JVM, version of JVM. I think adding -XX:+UseNUMA is very specific to one users environnement and won't be supported and may even crash lots of other people evironements where hadoop/tez is runned. This kind of settings are more likely to be configurable and configured by the user to fit its specific needs/environnement. I'm pretty sure there is already some configuration variables one can set in its tez-site.xml file to set this option so no need to have pig force this setting by code.

Can you explain exactly where and how you wanted to set/force the -XX:+UseNUMA ?

besides, UseNUMA should not correct OOM issues, it's more related to memory-performance than to memory-availability so strange activating it corrects an OOM issue. Seems to be accepted by oracle JVM starting from java 6u2.

> Add -XX:+UseNUMA for Tez jobs
> -----------------------------
>
>                 Key: PIG-4555
>                 URL: https://issues.apache.org/jira/browse/PIG-4555
>             Project: Pig
>          Issue Type: Improvement
>            Reporter: Rohini Palaniswamy
>            Assignee: Rohini Palaniswamy
>             Fix For: 0.15.0
>
>
>     For very big Tez jobs (~50K tasks), AM quickly goes OOM without -XX:+UseNUMA. tez.am.launch.cmd-opts default setting has that, but since pig gives preference to yarn.app.mapreduce.am.command-opts if present (which usually it is),  -XX:+UseNUMA is not there. Need to add -XX:+UseNUMA if we are picking up mapreduce setting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)