You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@datafusion.apache.org by gi...@apache.org on 2026/08/22 14:59:56 UTC

(datafusion) 01/01: Raise `hash_join_single_partition_threshold` to 4MB (#24558)

This is an automated email from the ASF dual-hosted git repository.

github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-24558-fc15e25cbb49d457c319dbf522f2982d32c794cb
in repository https://gitbox.apache.org/repos/asf/datafusion.git

commit 7b7a9e84db1d528f0b2769122a02f82780a66327
Author: Daniël Heres <da...@gmail.com>
AuthorDate: Sat Aug 22 14:59:33 2026 +0000

    Raise `hash_join_single_partition_threshold` to 4MB (#24558)
    
    ## Which issue does this PR close?
    
    None; split out of #24456 so the default change can be reviewed on its
    own.
    
    ## Rationale for this change
    
    We can increase the default value as generally a larger value performs
    better (avoiding a repartioning)
    
    ## What changes are included in this PR?
    
    - `hash_join_single_partition_threshold` defaults to 4MB instead of 1MB,
    and the
      generated `configs.md` row follows.
    - The two sqllogictest files that print or restore the value are
    updated.
    - The `join_selection` tests build their inputs from the threshold but
    asserted the byte
    sizes the 1MB default happened to produce (`8192` / `2097152`). They now
    derive those
    expectations from the configured threshold, so the assertions stay
    correct wherever the
      default sits.
    
    ```
    TPC-H 10
    
    │ Total Time (HEAD)                                            │ 6500.41ms │
    │ Total Time (perf_hash-join-single-partition-threshold-4mb)   │ 6262.84ms │
    
    
    TPC-DS
    │ Total Time (HEAD)                                            │ 9919.35ms │
    │ Total Time (perf_hash-join-single-partition-threshold-4mb)   │ 9572.04ms │
    ```
    
    ## Are these changes tested?
    
    Yes, by existing coverage: the full sqllogictest suite (504 files)
    passes, as do the
    `datafusion`, `datafusion-common` and `datafusion-physical-optimizer`
    unit and
    integration tests. No new test is added -- the change is a default, and
    the tests that
    pin plan shapes around it already exist.
    
    ## Are there any user-facing changes?
    
    Yes: a changed default, so a query whose build side is between 1MB and
    4MB now plans as
    `CollectLeft` where it previously repartitioned. `configs.md` documents
    the new value,
    and anyone relying on the old behaviour can set the option back to
    `1048576`.
    
    Co-authored-by: Claude Opus 5 <no...@anthropic.com>
---
 datafusion/common/src/config.rs                    |  2 +-
 .../tests/physical_optimizer/join_selection.rs     | 38 ++++++++++++++--------
 .../sqllogictest/test_files/information_schema.slt |  4 +--
 .../test_files/statistics_registry.slt             |  2 +-
 docs/source/user-guide/configs.md                  |  2 +-
 5 files changed, 29 insertions(+), 19 deletions(-)

diff --git a/datafusion/common/src/config.rs b/datafusion/common/src/config.rs
index 9fc1db841e..b340ec8628 100644
--- a/datafusion/common/src/config.rs
+++ b/datafusion/common/src/config.rs
@@ -1775,7 +1775,7 @@ config_namespace! {
 
         /// The maximum estimated size in bytes for one input side of a HashJoin
         /// will be collected into a single partition
-        pub hash_join_single_partition_threshold: usize, default = 1024 * 1024
+        pub hash_join_single_partition_threshold: usize, default = 4 * 1024 * 1024
 
         /// The maximum estimated size in rows for one input side of a HashJoin
         /// will be collected into a single partition
diff --git a/datafusion/core/tests/physical_optimizer/join_selection.rs b/datafusion/core/tests/physical_optimizer/join_selection.rs
index 63654ae048..5bf457ed5d 100644
--- a/datafusion/core/tests/physical_optimizer/join_selection.rs
+++ b/datafusion/core/tests/physical_optimizer/join_selection.rs
@@ -75,6 +75,16 @@ fn get_thresholds() -> (usize, usize) {
     )
 }
 
+/// The byte size [`small_statistics`] derives from the configured threshold.
+fn small_byte_size() -> usize {
+    get_thresholds().1 / 128
+}
+
+/// The byte size [`big_statistics`] derives from the configured threshold.
+fn big_byte_size() -> usize {
+    get_thresholds().1 * 2
+}
+
 /// Return statistics for small table
 fn small_statistics() -> Statistics {
     let (threshold_num_rows, threshold_byte_size) = get_thresholds();
@@ -258,14 +268,14 @@ async fn test_join_with_swap() {
             .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(8192)
+        Precision::Inexact(small_byte_size())
     );
     assert_eq!(
         StatisticsContext::new()
             .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(2097152)
+        Precision::Inexact(big_byte_size())
     );
 }
 
@@ -382,14 +392,14 @@ async fn test_left_join_no_swap() {
             .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(8192)
+        Precision::Inexact(small_byte_size())
     );
     assert_eq!(
         StatisticsContext::new()
             .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(2097152)
+        Precision::Inexact(big_byte_size())
     );
 }
 
@@ -431,14 +441,14 @@ async fn test_join_with_swap_semi() {
                 .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
                 .unwrap()
                 .total_byte_size,
-            Precision::Inexact(8192)
+            Precision::Inexact(small_byte_size())
         );
         assert_eq!(
             StatisticsContext::new()
                 .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
                 .unwrap()
                 .total_byte_size,
-            Precision::Inexact(2097152)
+            Precision::Inexact(big_byte_size())
         );
         assert_eq!(original_schema, swapped_join.schema());
     }
@@ -668,14 +678,14 @@ async fn test_join_with_swap_mark() {
                 .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
                 .unwrap()
                 .total_byte_size,
-            Precision::Inexact(8192)
+            Precision::Inexact(small_byte_size())
         );
         assert_eq!(
             StatisticsContext::new()
                 .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
                 .unwrap()
                 .total_byte_size,
-            Precision::Inexact(2097152)
+            Precision::Inexact(big_byte_size())
         );
         assert_eq!(original_schema, swapped_join.schema());
     }
@@ -794,14 +804,14 @@ async fn test_join_no_swap() {
             .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(8192)
+        Precision::Inexact(small_byte_size())
     );
     assert_eq!(
         StatisticsContext::new()
             .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(2097152)
+        Precision::Inexact(big_byte_size())
     );
 }
 
@@ -867,14 +877,14 @@ async fn test_nl_join_with_swap(join_type: JoinType) {
             .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(8192)
+        Precision::Inexact(small_byte_size())
     );
     assert_eq!(
         StatisticsContext::new()
             .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(2097152)
+        Precision::Inexact(big_byte_size())
     );
 }
 
@@ -938,14 +948,14 @@ async fn test_nl_join_with_swap_no_proj(join_type: JoinType) {
             .compute(swapped_join.left().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(8192)
+        Precision::Inexact(small_byte_size())
     );
     assert_eq!(
         StatisticsContext::new()
             .compute(swapped_join.right().as_ref(), &StatisticsArgs::new())
             .unwrap()
             .total_byte_size,
-        Precision::Inexact(2097152)
+        Precision::Inexact(big_byte_size())
     );
 }
 
diff --git a/datafusion/sqllogictest/test_files/information_schema.slt b/datafusion/sqllogictest/test_files/information_schema.slt
index 573fb04b34..261027cb92 100644
--- a/datafusion/sqllogictest/test_files/information_schema.slt
+++ b/datafusion/sqllogictest/test_files/information_schema.slt
@@ -324,7 +324,7 @@ datafusion.optimizer.expand_views_at_output false
 datafusion.optimizer.filter_null_join_keys false
 datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values 150
 datafusion.optimizer.hash_join_inlist_pushdown_max_size 131072
-datafusion.optimizer.hash_join_single_partition_threshold 1048576
+datafusion.optimizer.hash_join_single_partition_threshold 4194304
 datafusion.optimizer.hash_join_single_partition_threshold_rows 131072
 datafusion.optimizer.join_reordering true
 datafusion.optimizer.max_passes 3
@@ -484,7 +484,7 @@ datafusion.optimizer.expand_views_at_output false When set to true, if the retur
 datafusion.optimizer.filter_null_join_keys false When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.
 datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values 150 Maximum number of distinct values (rows) in the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides with more rows than this will use hash table lookups instead. Set to 0 to always use hash table lookups. This provides an additional limit beyond `hash_join_inlist_pushdown_max_size` to prevent very large IN lists that might not provide much benefit over hash table look [...]
 datafusion.optimizer.hash_join_inlist_pushdown_max_size 131072 Maximum size in bytes for the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides larger than this will use hash table lookups instead. Set to 0 to always use hash table lookups. InList pushdown can be more efficient for small build sides because it can result in better statistics pruning as well as use any bloom filters present on the scan side. InList expressions are also m [...]
-datafusion.optimizer.hash_join_single_partition_threshold 1048576 The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition
+datafusion.optimizer.hash_join_single_partition_threshold 4194304 The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition
 datafusion.optimizer.hash_join_single_partition_threshold_rows 131072 The maximum estimated size in rows for one input side of a HashJoin will be collected into a single partition
 datafusion.optimizer.join_reordering true When set to true, the physical plan optimizer may swap join inputs based on statistics. When set to false, statistics-driven join input reordering is disabled and the original join order in the query is used.
 datafusion.optimizer.max_passes 3 Number of times that the optimizer will attempt to optimize the plan
diff --git a/datafusion/sqllogictest/test_files/statistics_registry.slt b/datafusion/sqllogictest/test_files/statistics_registry.slt
index 89258bec29..f68eeac65e 100644
--- a/datafusion/sqllogictest/test_files/statistics_registry.slt
+++ b/datafusion/sqllogictest/test_files/statistics_registry.slt
@@ -162,7 +162,7 @@ statement ok
 set datafusion.optimizer.use_statistics_registry = false;
 
 statement ok
-set datafusion.optimizer.hash_join_single_partition_threshold = 1048576;
+set datafusion.optimizer.hash_join_single_partition_threshold = 4194304;
 
 statement ok
 set datafusion.optimizer.hash_join_single_partition_threshold_rows = 131072;
diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.md
index e02ada03fc..dec013a88d 100644
--- a/docs/source/user-guide/configs.md
+++ b/docs/source/user-guide/configs.md
@@ -173,7 +173,7 @@ The following configuration settings are available:
 | datafusion.optimizer.use_statistics_registry                            | false                     | When set to true, the physical plan optimizer uses the pluggable `StatisticsRegistry` for statistics propagation across operators. This enables more accurate cardinality estimates compared to each operator's built-in `partition_statistics`.                                                                                                                                                     [...]
 | datafusion.optimizer.prefer_hash_join                                   | true                      | When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory                                                                                                                                                                                                                                [...]
 | datafusion.optimizer.enable_piecewise_merge_join                        | false                     | When set to true, piecewise merge join is enabled. PiecewiseMergeJoin is currently experimental. Physical planner will opt for PiecewiseMergeJoin when there is only one range filter.                                                                                                                                                                                                               [...]
-| datafusion.optimizer.hash_join_single_partition_threshold               | 1048576                   | The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition                                                                                                                                                                                                                                                                                       [...]
+| datafusion.optimizer.hash_join_single_partition_threshold               | 4194304                   | The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition                                                                                                                                                                                                                                                                                       [...]
 | datafusion.optimizer.hash_join_single_partition_threshold_rows          | 131072                    | The maximum estimated size in rows for one input side of a HashJoin will be collected into a single partition                                                                                                                                                                                                                                                                                        [...]
 | datafusion.optimizer.hash_join_inlist_pushdown_max_size                 | 131072                    | Maximum size in bytes for the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides larger than this will use hash table lookups instead. Set to 0 to always use hash table lookups. InList pushdown can be more efficient for small build sides because it can result in better statistics pruning as well as use any bloom filters present on the [...]
 | datafusion.optimizer.hash_join_inlist_pushdown_max_distinct_values      | 150                       | Maximum number of distinct values (rows) in the build side of a hash join to be pushed down as an InList expression for dynamic filtering. Build sides with more rows than this will use hash table lookups instead. Set to 0 to always use hash table lookups. This provides an additional limit beyond `hash_join_inlist_pushdown_max_size` to prevent very large IN lists that might not provide  [...]


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org