You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2022/03/08 05:25:04 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #2060] Clear job group for init SQL
This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new f8d9010 [KYUUBI #2060] Clear job group for init SQL
f8d9010 is described below
commit f8d9010ba57f7c4e12daf4626376edc7bc0421c0
Author: Cheng Pan <ch...@apache.org>
AuthorDate: Tue Mar 8 13:24:52 2022 +0800
[KYUUBI #2060] Clear job group for init SQL
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
We need to clear job group information for init SQL, otherwise in incremental collection mode, the fetchNext will always display wrong description.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Before this patch:
<img width="1916" alt="incremental_collect" src="https://user-images.githubusercontent.com/26535726/157165964-502ad809-9d99-44de-b08c-131d135ce33d.png">
After this patch:
<img width="1917" alt="Xnip2022-03-08_12-04-50" src="https://user-images.githubusercontent.com/26535726/157165975-68eddf24-e0ad-4db2-931c-c75541802a7b.png">
- [ ] [Run test](https://kyuubi.apache.org/docs/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #2060 from pan3793/clear.
Closes #2060
76a67372 [Cheng Pan] Clear job group for init SQL
Authored-by: Cheng Pan <ch...@apache.org>
Signed-off-by: ulysses-you <ul...@apache.org>
---
.../main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 1 +
.../kyuubi/engine/spark/session/SparkSQLSessionManager.scala | 7 ++++++-
2 files changed, 7 insertions(+), 1 deletion(-)
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index 31a2828..b6de175 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -133,6 +133,7 @@ object SparkSQLEngine extends Logging {
interruptOnCancel = true)
debug(s"Execute session initializing sql: $sqlStr")
session.sql(sqlStr).isEmpty
+ session.sparkContext.clearJobGroup()
}
session
}
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala
index 4a8769b..fc039d8 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/session/SparkSQLSessionManager.scala
@@ -66,8 +66,13 @@ class SparkSQLSessionManager private (name: String, spark: SparkSession)
} else {
val ss = spark.newSession()
this.conf.get(ENGINE_SESSION_INITIALIZE_SQL).foreach { sqlStr =>
- ss.sparkContext.setJobDescription(sqlStr)
+ ss.sparkContext.setJobGroup(
+ "engine_initializing_queries",
+ sqlStr,
+ interruptOnCancel = true)
+ debug(s"Execute session initializing sql: $sqlStr")
ss.sql(sqlStr).isEmpty
+ ss.sparkContext.clearJobGroup()
}
ss
}