You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by "Srimanth Gunturi (JIRA)" <ji...@apache.org> on 2014/03/05 21:56:46 UTC
[jira] [Commented] (AMBARI-4964) "hive.tez.container.size" should
be in MB and dynamically calculated
[ https://issues.apache.org/jira/browse/AMBARI-4964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13921385#comment-13921385 ]
Srimanth Gunturi commented on AMBARI-4964:
------------------------------------------
Hive on Tez doesn't distinguish between map an reducer. It just runs vertices that can do whatever. However, the size their set to are similar to what we're doing for maps and reduces and we should be able to steal the magic computation they use.
The relevant settings are hive.tez.container.size and hive.tez.java.opts. The same logic (one needs to be slightly lower than the other needs to be applied here.)
As far as container size goes we'd ideally like to have something like:
if map size > 2GB use map size
else use reduce size
The reason is that up to a certain limit memory buys us more than number of containers, but that tapers off after a certain size.
> "hive.tez.container.size" should be in MB and dynamically calculated
> --------------------------------------------------------------------
>
> Key: AMBARI-4964
> URL: https://issues.apache.org/jira/browse/AMBARI-4964
> Project: Ambari
> Issue Type: Task
> Components: client
> Affects Versions: 1.5.0
> Reporter: Srimanth Gunturi
> Assignee: Srimanth Gunturi
> Fix For: 1.5.0
>
>
> Ambari set the following in {{/etc/hive/conf/hive-site.xml}}
> {code}
> <property>
> <name>hive.tez.container.size</name>
> <value>3000000000</value>
> </property>
> {code}
> and the following error was hit when running a query:
> {code}
> 14/03/03 20:20:32 ERROR exec.Task: Failed to execute tez graph.
> java.lang.NumberFormatException: For input string: "3000000000"
> at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:495)
> at java.lang.Integer.parseInt(Integer.java:527)
> at org.apache.hadoop.conf.Configuration.getInt(Configuration.java:1094)
> at org.apache.hadoop.hive.conf.HiveConf.getIntVar(HiveConf.java:1173)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.getContainerResource(DagUtils.java:350)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:508)
> at org.apache.hadoop.hive.ql.exec.tez.DagUtils.createVertex(DagUtils.java:850)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.build(TezTask.java:260)
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:144)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1456)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1229)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1047)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:874)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:864)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:794)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:687)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
> Ambari should set the value in MB
--
This message was sent by Atlassian JIRA
(v6.2#6252)