You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Ashutosh Chauhan (Jira)" <ji...@apache.org> on 2021/01/30 03:42:00 UTC
[jira] [Commented] (HIVE-24695) Clean up session resources, if
TezSession is unable to start
[ https://issues.apache.org/jira/browse/HIVE-24695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17275470#comment-17275470 ]
Ashutosh Chauhan commented on HIVE-24695:
-----------------------------------------
+1
> Clean up session resources, if TezSession is unable to start
> ------------------------------------------------------------
>
> Key: HIVE-24695
> URL: https://issues.apache.org/jira/browse/HIVE-24695
> Project: Hive
> Issue Type: Improvement
> Reporter: Rajesh Balamohan
> Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are cases when TezSessionState would not be able to start. (e.g resource constraints on YARN queues).
> {noformat}
> Caused by: org.apache.hadoop.yarn.exceptions.YarnException: Failed to submit application_1611791897439_0058 to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has X applications from user hive cannot accept submission of application: application_1611791897439_0058
> at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.submitApplication(YarnClientImpl.java:322) ~[hadoop-yarn-client-3.x.jar:?]
> at org.apache.tez.client.TezYarnClient.submitApplication(TezYarnClient.java:77) ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.tez.client.TezClient.start(TezClient.java:405) ~[tez-api-0.9.x.jar:0.9.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.startSessionAndContainers(TezSessionState.java:535) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.openInternal(TezSessionState.java:373) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionState.open(TezSessionState.java:298) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezSessionPoolSession.open(TezSessionPoolSession.java:106) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.ensureSessionHasResources(TezTask.java:403) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.tez.TezTask.execute(TezTask.java:209) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:213) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:105) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Executor.launchTask(Executor.java:359) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Executor.launchTasks(Executor.java:330) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Executor.runTasks(Executor.java:246) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Executor.execute(Executor.java:109) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:721) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:488) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:482) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:166) ~[hive-exec-3.x.jar:3.x]
> at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:225) ~[hive-service-3.x.jar:3.x]
> {noformat}
> However by this time, session directories & certain resources are localized. (e.g, hive-exec jars are stored in hdfs:///tmp/hive/hive/_tez_session_dir/*/hive-exec*.jar). When tezClient is not started, it does not clear up the resources. This leaks ~100MB data in HDFS per failure.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)