You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by hv...@apache.org on 2024/08/30 12:04:16 UTC
(spark) branch master updated: [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
This is an automated email from the ASF dual-hosted git repository.
hvanhovell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 343471dac4b9 [SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
343471dac4b9 is described below
commit 343471dac4b96b43a09763d759b6c30760fb626e
Author: Changgyoo Park <ch...@databricks.com>
AuthorDate: Fri Aug 30 08:04:04 2024 -0400
[SPARK-49409][CONNECT] Adjust the default value of CONNECT_SESSION_PLAN_CACHE_SIZE
### What changes were proposed in this pull request?
Set the default value of CONNECT_SESSION_PLAN_CACHE_SIZE to 16 which was previously 5.
### Why are the changes needed?
There is a rare use case where a data frame is built from multiple dependent data frames, and while creating the last dependent data frame, all the other dependent data frames are invalidated from the plan cache, causing them to be analysed again. For this particular case, the performance of creating the data frame was improved from 41 minutes to 1 minute.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Performance test.
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #47937 from changgyoopark-db/SPARK-49409.
Authored-by: Changgyoo Park <ch...@databricks.com>
Signed-off-by: Herman van Hovell <he...@databricks.com>
---
.../src/main/scala/org/apache/spark/sql/connect/config/Connect.scala | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
index dc45684a75eb..92709ff29a1c 100644
--- a/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
+++ b/sql/connect/server/src/main/scala/org/apache/spark/sql/connect/config/Connect.scala
@@ -289,7 +289,7 @@ object Connect {
.version("4.0.0")
.internal()
.intConf
- .createWithDefault(5)
+ .createWithDefault(16)
val CONNECT_SESSION_PLAN_CACHE_ENABLED =
buildConf("spark.connect.session.planCache.enabled")
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org