[spark] branch master updated: [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value

[we...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

wenchen pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new afcccb42c96 [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value
afcccb42c96 is described below

commit afcccb42c96cc7785a85c9463523f34d7a900a1d
Author: Zamil Majdy <za...@databricks.com>
AuthorDate: Tue Aug 15 18:10:23 2023 +0800

    [SPARK-44718][SQL] Match ColumnVector memory-mode config default to OffHeapMemoryMode config value
    
    ### What changes were proposed in this pull request?
    
    Set the column vector default memory mode to depend on the off-heap memory mode flag. This is to prevent a user from using Vectorized-Reader with an on-heap column-vector by default when the off-heap memory mode is enabled on the cluster.
    
    ### Why are the changes needed?
    Avoid the unintentional usage of on-heap memory in vectorized-reader when off-heap memory mode is enabled by the user.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    Manual & existing tests.
    
    Closes #42394 from majdyz/offheap-colvec-mode-default-value.
    
    Lead-authored-by: Zamil Majdy <za...@databricks.com>
    Co-authored-by: Zamil Majdy <za...@gmail.com>
    Signed-off-by: Wenchen Fan <we...@databricks.com>
---
 .../src/main/scala/org/apache/spark/sql/internal/SQLConf.scala      | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index ac1b4918d97..33b28f60f2a 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -513,10 +513,10 @@ object SQLConf {
   val COLUMN_VECTOR_OFFHEAP_ENABLED =
     buildConf("spark.sql.columnVector.offheap.enabled")
       .internal()
-      .doc("When true, use OffHeapColumnVector in ColumnarBatch.")
+      .doc("When true, use OffHeapColumnVector in ColumnarBatch. " +
+        s"Defaults to $MEMORY_OFFHEAP_ENABLED.")
       .version("2.3.0")
-      .booleanConf
-      .createWithDefault(false)
+      .fallbackConf(MEMORY_OFFHEAP_ENABLED)
 
   val PREFER_SORTMERGEJOIN = buildConf("spark.sql.join.preferSortMergeJoin")
     .internal()


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org