[spark] branch branch-3.2 updated: [SPARK-36659][SQL] Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-3.2 by this push:
     new aa96a37  [SPARK-36659][SQL] Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config
aa96a37 is described below

commit aa96a374b2f8b0bf4b004792d6b7729e27e349e5
Author: Kent Yao <ya...@apache.org>
AuthorDate: Fri Sep 3 19:11:37 2021 +0800

    [SPARK-36659][SQL] Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config
    
    ### What changes were proposed in this pull request?
    
    Promote spark.sql.execution.topKSortFallbackThreshold to a user-facing config
    
    ### Why are the changes needed?
    
    spark.sql.execution.topKSortFallbackThreshold now is an internal config hidden from users Integer.MAX_VALUE - 15 as its default. In many real-world cases, if the K is very big,  there would be performance issues.
    
    It's better to leave this choice to users
    
    ### Does this PR introduce _any_ user-facing change?
    
     spark.sql.execution.topKSortFallbackThreshold is now user-facing
    
    ### How was this patch tested?
    
    passing GA
    
    Closes #33904 from yaooqinn/SPARK-36659.
    
    Authored-by: Kent Yao <ya...@apache.org>
    Signed-off-by: Kent Yao <ya...@apache.org>
    (cherry picked from commit 7f1ad7be18b07d880ba92c470e64bf7458b8366a)
    Signed-off-by: Dongjoon Hyun <dh...@apple.com>
---
 sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala | 1 -
 1 file changed, 1 deletion(-)

diff --git a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index b137fd2..11b3097 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -2637,7 +2637,6 @@ object SQLConf {
 
   val TOP_K_SORT_FALLBACK_THRESHOLD =
     buildConf("spark.sql.execution.topKSortFallbackThreshold")
-      .internal()
       .doc("In SQL queries with a SORT followed by a LIMIT like " +
           "'SELECT x FROM t ORDER BY y LIMIT m', if m is under this threshold, do a top-K sort" +
           " in memory, otherwise do a global sort which spills to disk if necessary.")

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org