You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2010/08/09 21:38:17 UTC
[jira] Created: (HIVE-1520) hive.mapred.local.mem should only be
used in case of local mode job submissions
hive.mapred.local.mem should only be used in case of local mode job submissions
-------------------------------------------------------------------------------
Key: HIVE-1520
URL: https://issues.apache.org/jira/browse/HIVE-1520
Project: Hadoop Hive
Issue Type: Bug
Reporter: Joydeep Sen Sarma
Currently - whenever we submit a map-reduce job via a child jvm process, hive sets HADOOP_HEAPSIZE to hive.mapred.local.mem (thereby limiting the max heap memory of the child jvm). the assumption being that we are submitting a job for local mode execution and different memory limits apply for that.
however - one can submit jobs via a child jvm for non local mode execution as well. This is useful, for example, if hive wants to submit jobs via different hadoop clients (for sending jobs to different hadoop clusters). in such case, we can use the 'hive.exec.submitviachild' and 'hadoop.bin.path' to dispatch job via an alternate hadoop client install point. however in such case, we don't need to set HADOOP_HEAPSIZE. all we are using the child jvm is to run the small bit of hive code that submits the job (and not for local mode execution).
in this case - we shouldn't be setting the child jvm's memory limit and should leave it to what the parent's value is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1520) hive.mapred.local.mem should only be
used in case of local mode job submissions
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated HIVE-1520:
------------------------------------
Attachment: 1520.1.patch
> hive.mapred.local.mem should only be used in case of local mode job submissions
> -------------------------------------------------------------------------------
>
> Key: HIVE-1520
> URL: https://issues.apache.org/jira/browse/HIVE-1520
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Joydeep Sen Sarma
> Attachments: 1520.1.patch
>
>
> Currently - whenever we submit a map-reduce job via a child jvm process, hive sets HADOOP_HEAPSIZE to hive.mapred.local.mem (thereby limiting the max heap memory of the child jvm). the assumption being that we are submitting a job for local mode execution and different memory limits apply for that.
> however - one can submit jobs via a child jvm for non local mode execution as well. This is useful, for example, if hive wants to submit jobs via different hadoop clients (for sending jobs to different hadoop clusters). in such case, we can use the 'hive.exec.submitviachild' and 'hadoop.bin.path' to dispatch job via an alternate hadoop client install point. however in such case, we don't need to set HADOOP_HEAPSIZE. all we are using the child jvm is to run the small bit of hive code that submits the job (and not for local mode execution).
> in this case - we shouldn't be setting the child jvm's memory limit and should leave it to what the parent's value is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1520) hive.mapred.local.mem should only be
used in case of local mode job submissions
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain resolved HIVE-1520.
------------------------------
Resolution: Fixed
Fix Version/s: 0.7.0
Hadoop Flags: [Reviewed]
Committed. Thanks Joy
> hive.mapred.local.mem should only be used in case of local mode job submissions
> -------------------------------------------------------------------------------
>
> Key: HIVE-1520
> URL: https://issues.apache.org/jira/browse/HIVE-1520
> Project: Hadoop Hive
> Issue Type: Bug
> Reporter: Joydeep Sen Sarma
> Fix For: 0.7.0
>
> Attachments: 1520.1.patch
>
>
> Currently - whenever we submit a map-reduce job via a child jvm process, hive sets HADOOP_HEAPSIZE to hive.mapred.local.mem (thereby limiting the max heap memory of the child jvm). the assumption being that we are submitting a job for local mode execution and different memory limits apply for that.
> however - one can submit jobs via a child jvm for non local mode execution as well. This is useful, for example, if hive wants to submit jobs via different hadoop clients (for sending jobs to different hadoop clusters). in such case, we can use the 'hive.exec.submitviachild' and 'hadoop.bin.path' to dispatch job via an alternate hadoop client install point. however in such case, we don't need to set HADOOP_HEAPSIZE. all we are using the child jvm is to run the small bit of hive code that submits the job (and not for local mode execution).
> in this case - we shouldn't be setting the child jvm's memory limit and should leave it to what the parent's value is.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.