You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kyuubi.apache.org by GitBox <gi...@apache.org> on 2021/09/03 06:02:27 UTC
[GitHub] [incubator-kyuubi] yaooqinn opened a new issue #1018: Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value
yaooqinn opened a new issue #1018:
URL: https://github.com/apache/incubator-kyuubi/issues/1018
### 1. Describe the feature
In https://github.com/apache/spark/pull/33904, I faced a performance issue with the topK scenario where K is very large which can be avoided by setting spark.sql.execution.topKSortFallbackThreshold less than the K here.
For Kyuubi users who run SQL only, improper topK SQLs will hold engines for quite a long time and they don't know why.
With a proper value for spark.sql.execution.topKSortFallbackThreshold is more suite for Kyuubi's cases
I'd suggest this value be set to 10000
### 2. Motivation
### 3. Describe the solution
### 4. Additional context
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] byyue commented on issue #1018: Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value
Posted by GitBox <gi...@apache.org>.
byyue commented on issue #1018:
URL: https://github.com/apache/incubator-kyuubi/issues/1018#issuecomment-914390300
Hi, could you please elaborate a little bit more on how to implement this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] byyue commented on issue #1018: Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value
Posted by GitBox <gi...@apache.org>.
byyue commented on issue #1018:
URL: https://github.com/apache/incubator-kyuubi/issues/1018#issuecomment-914957634
Thanks! I think spark.sql.execution.topKSortFallbackThreshold is now a property name of Spark Runtime SQL Configuration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn commented on issue #1018: Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value
Posted by GitBox <gi...@apache.org>.
yaooqinn commented on issue #1018:
URL: https://github.com/apache/incubator-kyuubi/issues/1018#issuecomment-914888754
```diff
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index 6968e07c..9c2832c5 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -96,6 +96,7 @@ object SparkSQLEngine extends Logging {
def createSpark(): SparkSession = {
val sparkConf = new SparkConf()
+ sparkConf.setIfMissing("spark.sql.execution.topKSortFallbackThreshold", "10000")
sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
sparkConf.setIfMissing("spark.master", "local")
sparkConf.setIfMissing("spark.ui.port", "0")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [incubator-kyuubi] yaooqinn closed issue #1018: Set spark.sql.execution.topKSortFallbackThreshold to a reasonable value
Posted by GitBox <gi...@apache.org>.
yaooqinn closed issue #1018:
URL: https://github.com/apache/incubator-kyuubi/issues/1018
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@kyuubi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org