You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@kyuubi.apache.org by fe...@apache.org on 2021/12/04 10:31:19 UTC
[incubator-kyuubi] branch master updated: [KYUUBI #1493] Make a reasonable thread pool size for listing file statuses
This is an automated email from the ASF dual-hosted git repository.
feiwang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/incubator-kyuubi.git
The following commit(s) were added to refs/heads/master by this push:
new a876678 [KYUUBI #1493] Make a reasonable thread pool size for listing file statuses
a876678 is described below
commit a876678a9143bb2f82a5901840e3ba0da992efcf
Author: Brian Yue <co...@gmail.com>
AuthorDate: Sat Dec 4 18:31:07 2021 +0800
[KYUUBI #1493] Make a reasonable thread pool size for listing file statuses
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://kyuubi.readthedocs.io/en/latest/community/contributions.html
2. If the PR is related to an issue in https://github.com/apache/incubator-kyuubi/issues, add '[KYUUBI #XXXX]' in your PR title, e.g., '[KYUUBI #XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][KYUUBI #XXXX] Your PR title ...'.
-->
### _Why are the changes needed?_
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about the use case of it.
2. If you fix a bug, you can clarify why it is a bug.
-->
Close #1493.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] [Run test](https://kyuubi.readthedocs.io/en/latest/develop_tools/testing.html#running-tests) locally before make a pull request
Closes #1494 from byyue/feature-1493.
Closes #1493
9a280926 [Brian Yue] Make a reasonable thread pool size for listing file statuses
Authored-by: Brian Yue <co...@gmail.com>
Signed-off-by: Fei Wang <fw...@ebay.com>
---
.../src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala | 3 +++
1 file changed, 3 insertions(+)
diff --git a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
index ab7c589..8a5b951 100644
--- a/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
+++ b/externals/kyuubi-spark-sql-engine/src/main/scala/org/apache/kyuubi/engine/spark/SparkSQLEngine.scala
@@ -81,6 +81,9 @@ object SparkSQLEngine extends Logging {
sparkConf.setIfMissing("spark.sql.legacy.castComplexTypesToString.enabled", "true")
sparkConf.setIfMissing("spark.master", "local")
sparkConf.setIfMissing("spark.ui.port", "0")
+ sparkConf.setIfMissing(
+ "spark.hadoop.mapreduce.input.fileinputformat.list-status.num-threads",
+ "20")
val appName = s"kyuubi_${user}_spark_${Instant.now}"
sparkConf.setIfMissing("spark.app.name", appName)