You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Wenping Zhan (Jira)" <ji...@apache.org> on 2021/03/25 03:07:00 UTC
[jira] [Commented] (HIVE-24800) Yarn cluster resource can't be
released when close jdbc connection for hiveserver2
[ https://issues.apache.org/jira/browse/HIVE-24800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17308318#comment-17308318 ]
Wenping Zhan commented on HIVE-24800:
-------------------------------------
i consider two methods to slove the issues:
1. enable spark dynamicAllocation, it won't trigger the code
2. the other is throw InterruptException directly,eg:
{code:java}
private void getSparkMemoryAndCores(OptimizeSparkProcContext context) throws SemanticException {
if (sparkMemoryAndCores != null) {
return;
}
if (context.getConf().getBoolean(SPARK_DYNAMIC_ALLOCATION_ENABLED, false)) {
// If dynamic allocation is enabled, numbers for memory and cores are meaningless. So, we don't
// try to get it.
sparkMemoryAndCores = null;
return;
}
SparkSessionManager sparkSessionManager = null;
SparkSession sparkSession = null;
try {
sparkSessionManager = SparkSessionManagerImpl.getInstance();
sparkSession = SparkUtilities.getSparkSession(
context.getConf(), sparkSessionManager);
sparkMemoryAndCores = sparkSession.getMemoryAndCores();
} catch (HiveException e) {
throw new SemanticException("Failed to get a spark session: " + e);
} catch (Exception e) {
LOG.warn("Failed to get spark memory/core info", e);
// throw InterruptedException
if(e instanceof InterruptedException){
throw new SemanticException("Semantic analysis failed as current thread is interrupted: " + e);
}
} finally {
if (sparkSession != null && sparkSessionManager != null) {
try {
sparkSessionManager.returnSession(sparkSession);
} catch (HiveException ex) {
LOG.error("Failed to return the session to SessionManager: " + ex, ex);
}
}
}
}
{code}
who can give me more suggestm, thx!
> Yarn cluster resource can't be released when close jdbc connection for hiveserver2
> ----------------------------------------------------------------------------------
>
> Key: HIVE-24800
> URL: https://issues.apache.org/jira/browse/HIVE-24800
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2, JDBC, Spark
> Affects Versions: 2.3.6
> Reporter: Wenping Zhan
> Priority: Critical
>
> When I submit an application job on yarn cluster by Hive on Spark. I found yarn cluster resource which app had apply for can't be released after closing jdbc connection for hiveserver2. I think multi thread operate the same connect would cause this issue, one thread create jdbc connection and start compile Hql, the other thread interrupt the connection when create DAG . I get error info below and alway recurrent this issue, can you help me confirm if it is an issue?[~xuefuz], How I do if indeed?
> *error info:*
> 2020-12-10T10:24:30,069 INFO [HiveServer2-Handler-Pool: Thread-824055] session.SessionState: Updating thread name to 11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055
> 2020-12-10T10:24:30,069 INFO [11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055] operation.OperationManager: Closing operation: OperationHandle [opType=EXECUTE_STATEMENT, getHandleIdentifier()=df4e1322-c60a-4c28-83e3-c19ae9243404]
> 2020-12-10T10:24:30,069 INFO [11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055] operation.Operation: The running operation has been successfully interrupted: null
> 2020-12-10T10:24:30,070 INFO [11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055] session.HiveSessionImpl: Operation log session directory is deleted: /cloud/data/hive/tmp/hadoop/operation_logs/11bf6a51-9a3b-488b-885e-fc50af485a64
> 2020-12-10T10:24:30,070 INFO [11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055] conf.HiveConf: Using the default value passed in for log id: 11bf6a51-9a3b-488b-885e-fc50af485a64
> 2020-12-10T10:24:30,070 INFO [11bf6a51-9a3b-488b-885e-fc50af485a64 HiveServer2-Handler-Pool: Thread-824055] session.SessionState: Resetting thread name to HiveServer2-Handler-Pool: Thread-824055
> 2020-12-10T10:24:30,071 WARN [RPC-Handler-7] rpc.RpcDispatcher: [ClientProtocol] Closing RPC channel with 2 outstanding RPCs.
> 2020-12-10T10:24:30,070 WARN [HiveServer2-Background-Pool: Thread-824059] spark.SetSparkReducerParallelism: Failed to get spark memory/core info
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method) ~[?:1.8.0_231]
> at java.lang.Object.wait(Object.java:460) ~[?:1.8.0_231]
> at io.netty.util.concurrent.DefaultPromise.await0(DefaultPromise.java:600) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
> at io.netty.util.concurrent.DefaultPromise.await(DefaultPromise.java:273) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
> at io.netty.util.concurrent.AbstractFuture.get(AbstractFuture.java:46) ~[netty-all-4.0.52.Final.jar:4.0.52.Final]
> at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.getExecutorCount(RemoteHiveSparkClient.java:156) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient.getExecutorCount(RemoteHiveSparkClient.java:166) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:77) ~[hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.getSparkMemoryAndCores(SetSparkReducerParallelism.java:238) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:173) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:56) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.PreOrderWalker.walk(PreOrderWalker.java:61) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.runSetReducerParallelism(SparkCompiler.java:288) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.spark.SparkCompiler.optimizeOperatorPlan(SparkCompiler.java:122) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:140) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:11273) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:286) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:512) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1317) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1295) [hive-exec-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:204) [hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:346) [hive-service-2.3.6.jar:2.3.6]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_231]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_231]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) [hadoop-common-2.8.5.jar:?]
> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362) [hive-service-2.3.6.jar:2.3.6]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_231]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
> ...
> ...
> ...
> 2020-12-10T10:24:43,770 INFO [HiveServer2-Background-Pool: Thread-824059] parse.CalcitePlanner: Completed plan generation
> 2020-12-10T10:24:43,770 INFO [HiveServer2-Background-Pool: Thread-824059] ql.Driver: Semantic Analysis Completed
> 2020-12-10T10:24:43,770 ERROR [HiveServer2-Background-Pool: Thread-824059] ql.Driver: FAILED: command has been interrupted: after analyzing query.
> 2020-12-10T10:24:43,774 INFO [HiveServer2-Background-Pool: Thread-824059] ql.Driver: Compiling command(queryId=hadoop_20201210102414_27d30b57-1633-4c8a-b326-852b70bb2c2d) has been interrupted after 28.804 seconds
> 2020-12-10T10:24:43,774 ERROR [HiveServer2-Background-Pool: Thread-824059] operation.Operation: Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Illegal Operation state transition from CLOSED to ERROR
> at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:96) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.OperationState.validateTransition(OperationState.java:102) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.operation.Operation.setState(Operation.java:164) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.operation.SQLOperation.prepare(SQLOperation.java:234) ~[hive-service-2.3.6.jar:2.3.6]
> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork$1.run(SQLOperation.java:346) [hive-service-2.3.6.jar:2.3.6]
> at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_231]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_231]
> at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844) [hadoop-common-2.8.5.jar:?]
> at org.apache.hive.service.cli.operation.SQLOperation$BackgroundWork.run(SQLOperation.java:362) [hive-service-2.3.6.jar:2.3.6]
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_231]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)