You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by ul...@apache.org on 2023/03/23 12:49:03 UTC
[kyuubi] branch branch-1.7 updated: [KYUUBI #4587] [SPARK][EXT] Take care of table cache query stage
This is an automated email from the ASF dual-hosted git repository.
ulyssesyou pushed a commit to branch branch-1.7
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.7 by this push:
new 7ce1af90d [KYUUBI #4587] [SPARK][EXT] Take care of table cache query stage
7ce1af90d is described below
commit 7ce1af90d50b22244ed282c96b331e9b8fbe266d
Author: ulysses-you <ul...@gmail.com>
AuthorDate: Thu Mar 23 20:48:47 2023 +0800
[KYUUBI #4587] [SPARK][EXT] Take care of table cache query stage
### _Why are the changes needed?_
compatible with Spark3.5
see https://issues.apache.org/jira/browse/SPARK-42101
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #4587 from ulysses-you/FinalStageConfigIsolation.
Closes #4587
d456f9c99 [Cheng Pan] Update extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala
34db59d6e [ulysses-you] take care of table cache query stage
Lead-authored-by: ulysses-you <ul...@gmail.com>
Co-authored-by: Cheng Pan <pa...@gmail.com>
Signed-off-by: ulyssesyou <ul...@apache.org>
(cherry picked from commit 1457a2f82aea2513d0261c6699e735f853d0418c)
Signed-off-by: ulyssesyou <ul...@apache.org>
---
.../scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala b/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala
index 360a2645e..fee65b350 100644
--- a/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala
+++ b/extensions/spark/kyuubi-extension-spark-common/src/main/scala/org/apache/kyuubi/sql/KyuubiQueryStagePreparation.scala
@@ -133,7 +133,9 @@ case class FinalStageConfigIsolation(session: SparkSession) extends Rule[SparkPl
reusedExchangeExec
// query stage is leaf node so we need to transform it manually
- case queryStage: QueryStageExec =>
+ // compatible with Spark 3.5:
+ // SPARK-42101: table cache is a independent query stage, so do not need include it.
+ case queryStage: QueryStageExec if queryStage.nodeName != "TableCacheQueryStage" =>
queryStageNum += 1
collectNumber(queryStage.plan)
queryStage