You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "bert-beyondloops (via GitHub)" <gi...@apache.org> on 2026/10/13 11:37:04 UTC

[PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

bert-beyondloops opened a new pull request, #26261:
URL: https://github.com/apache/datafusion/pull/26261

   ## Which issue does this PR close?
   
   - Closes #26260 
   
   ## Rationale for this change
   
    With `prefer_hash_join = false`, the planner adds `SortExec`s under `SortMergeJoinExec` even when both inputs are already sorted on the join keys:                                                                               
                                                                                                                                                                                                                                      
     1. The sort options are hardcoded to `SortOptions::default()` (`ASC NULLS FIRST`) for every key. Inputs sorted `ASC NULLS LAST`, `DESC`, or with different options per key don't satisfy this, so `EnforceSorting` sorts both sides again.                                                                                                                                                                                                                     
     2. SMJ is only planned when `target_partitions > 1 && repartition_joins`. With `target_partitions = 1`, `prefer_hash_join = false` is silently ignored and a `HashJoinExec` is planned. With a single partition no repartitioning happens, so `repartition_joins` has no effect there.                
   
   ## What changes are included in this PR?
   
   Both changes are in `physical_planner.rs`.                                                                                                                                                                                       
                                                                                                                                                                                                                                      
     1. The sort options are derived from the inputs' declared ordering (`EquivalenceProperties::ordering_satisfy`):                                                                                                                  
        - Per key, try these candidates in order: default, flipped nulls, flipped direction, both flipped.                                                                                                                            
        - Backtrack across keys until both inputs satisfy the whole key list.                                                                                                                                                         
        - If no combination works, fall back to `SortOptions::default()`, so existing plans are unchanged.                                                                                                                            
     2. The SMJ condition changes from `target_partitions() > 1 && repartition_joins() && ...` to `(target_partitions() == 1 || repartition_joins()) && ...`. For `target_partitions > 1`, behavior is unchanged.                     
            
   
   ## What is the testing strategy for this PR?
   
   New SLT's :                                                                                                                                                                                                                             
     - New `sort_merge_join_sort_options.slt` covers inputs ordered `ASC NULLS LAST`, `DESC`, and mixed options per key, checking that no extra `SortExec` is added.                                                                  
     - New cases in `sort_merge_join.slt` for `target_partitions = 1`.                                                                                                                          
                          
   ## Are there any user-facing changes?
   
   No
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4584521438


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3728,6 +3732,117 @@ enum PlanAsyncExpr {
     Async(AsyncMapper, PlannedExprResult),
 }
 
+/// Chooses the join key order and per-key [`SortOptions`] of a [`SortMergeJoinExec`], so that as
+/// few inputs as possible need a `SortExec`.
+///
+/// The join keys are an unordered set of equalities, so they can be permuted freely. Candidates
+/// are derived from each known output ordering of each input: its entries that (after normalizing
+/// through the equivalence group) are join keys become the leading keys, in that order and with
+/// those options, and the remaining keys follow with the default options. Each candidate is
+/// scored by how many inputs `ordering_satisfy` confirms are already sorted on it, so constants,
+/// equal columns and any key order are handled. The default (original key order, default options)
+/// wins ties, so plans that already match are unchanged.
+///
+/// Cost: nothing is done if neither input has an ordering; otherwise one candidate per known
+/// ordering, each verified with two `ordering_satisfy` calls.
+fn sort_merge_join_keys(

Review Comment:
   indeed, nice hint. I adopted your proposed change. πŸ™ 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4267235546


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]

Review Comment:
   This fallback case has the same projection mismatch.
   ```suggestion
   01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
   02)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   04)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join.slt:
##########
@@ -1029,6 +1029,35 @@ DROP TABLE c;
 statement ok
 set datafusion.execution.target_partitions = 4;
 
+# prefer_hash_join = false also selects a sort merge join with a single target partition
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a = t2.a
+----
+logical_plan
+01)Projection: t1.a, t1.b, t2.b
+02)--Inner Join: t1.a = t2.a
+03)----TableScan: t1 projection=[a, b]
+04)----TableScan: t2 projection=[a, b]
+physical_plan
+01)ProjectionExec: expr=[a@0 as a, b@1 as b, b@3 as b]
+02)--SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0)]
+03)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]
+
+query III rowsort
+SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a = t2.a
+----

Review Comment:
   Could we restore the test runner’s initial value of 4 explicitly here and in `sort_merge_join_sort_options.slt`? `RESET` restores the machine default instead. After correcting the snapshots, both files fail with target_partitions: 4 -> 8 on my machine. Setting this back to 4 makes the focused suite pass.
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+
+statement ok
+DROP TABLE l;
+
+statement ok
+DROP TABLE r;
+
+statement ok
+DROP TABLE r_asc;
+
+statement ok
+reset datafusion.execution.target_partitions;

Review Comment:
   This location as well
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4584533799


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right table declares a different ordering than the file holds, so only the left ordering is adopted
+# and the right input is sorted
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET

Review Comment:
   replaced with an unsorted right input.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6858808685

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6858530257)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚    5.31 ms β”‚                                        5.20 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚   80.69 ms β”‚                                       78.24 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚   28.86 ms β”‚                                       28.77 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚  436.22 ms β”‚                                      424.17 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚   50.53 ms β”‚                                       49.13 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚   35.02 ms β”‚                                       34.29 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚   71.76 ms β”‚                                       71.40 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚   36.95 ms β”‚                                       36.14 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚   52.60 ms β”‚                                       50.40 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚   60.84 ms β”‚                                       60.41 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚  264.20 ms β”‚                                      273.78 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚   28.25 ms β”‚                                       28.28 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚  111.12 ms β”‚                                      111.97 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚  372.62 ms β”‚                                      370.84 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚   53.88 ms β”‚                                       52.11 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚    5.75 ms β”‚                                        5.74 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚   74.90 ms β”‚                                       74.50 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚  102.73 ms β”‚                                      103.13 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚   40.43 ms β”‚                                       40.45 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚   35.41 ms β”‚                                       35.24 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚   16.38 ms β”‚                                       17.42 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 22 β”‚   61.48 ms β”‚                                       62.35 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚  289.39 ms β”‚                                      288.75 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚  186.96 ms β”‚                                      187.35 ms β”‚     no change β”‚
   β”‚ QQuery 25 β”‚  104.17 ms β”‚                                      103.15 ms β”‚     no change β”‚
   β”‚ QQuery 26 β”‚   45.60 ms β”‚                                       44.87 ms β”‚     no change β”‚
   β”‚ QQuery 27 β”‚    4.80 ms β”‚                                        4.80 ms β”‚     no change β”‚
   β”‚ QQuery 28 β”‚   60.70 ms β”‚                                       56.60 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 29 β”‚   91.31 ms β”‚                                       90.57 ms β”‚     no change β”‚
   β”‚ QQuery 30 β”‚   31.44 ms β”‚                                       32.13 ms β”‚     no change β”‚
   β”‚ QQuery 31 β”‚  105.87 ms β”‚                                      106.96 ms β”‚     no change β”‚
   β”‚ QQuery 32 β”‚   19.77 ms β”‚                                       19.96 ms β”‚     no change β”‚
   β”‚ QQuery 33 β”‚   36.56 ms β”‚                                       36.35 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚    5.43 ms β”‚                                        5.38 ms β”‚     no change β”‚
   β”‚ QQuery 35 β”‚   69.81 ms β”‚                                       70.55 ms β”‚     no change β”‚
   β”‚ QQuery 36 β”‚    5.42 ms β”‚                                        5.41 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚    6.83 ms β”‚                                        6.60 ms β”‚     no change β”‚
   β”‚ QQuery 38 β”‚   59.69 ms β”‚                                       59.54 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚   72.48 ms β”‚                                       70.76 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚   22.07 ms β”‚                                       22.01 ms β”‚     no change β”‚
   β”‚ QQuery 41 β”‚   12.33 ms β”‚                                       12.25 ms β”‚     no change β”‚
   β”‚ QQuery 42 β”‚   23.72 ms β”‚                                       23.14 ms β”‚     no change β”‚
   β”‚ QQuery 43 β”‚    4.46 ms β”‚                                        4.36 ms β”‚     no change β”‚
   β”‚ QQuery 44 β”‚    5.38 ms β”‚                                        5.31 ms β”‚     no change β”‚
   β”‚ QQuery 45 β”‚   36.08 ms β”‚                                       35.05 ms β”‚     no change β”‚
   β”‚ QQuery 46 β”‚   11.51 ms β”‚                                       11.24 ms β”‚     no change β”‚
   β”‚ QQuery 47 β”‚  220.06 ms β”‚                                      214.14 ms β”‚     no change β”‚
   β”‚ QQuery 48 β”‚   90.92 ms β”‚                                       90.84 ms β”‚     no change β”‚
   β”‚ QQuery 49 β”‚   69.19 ms β”‚                                       69.29 ms β”‚     no change β”‚
   β”‚ QQuery 50 β”‚   56.51 ms β”‚                                       56.29 ms β”‚     no change β”‚
   β”‚ QQuery 51 β”‚   91.56 ms β”‚                                       93.52 ms β”‚     no change β”‚
   β”‚ QQuery 52 β”‚   23.73 ms β”‚                                       23.97 ms β”‚     no change β”‚
   β”‚ QQuery 53 β”‚   28.69 ms β”‚                                       29.25 ms β”‚     no change β”‚
   β”‚ QQuery 54 β”‚   52.80 ms β”‚                                       52.86 ms β”‚     no change β”‚
   β”‚ QQuery 55 β”‚   22.90 ms β”‚                                       23.30 ms β”‚     no change β”‚
   β”‚ QQuery 56 β”‚   37.19 ms β”‚                                       37.47 ms β”‚     no change β”‚
   β”‚ QQuery 57 β”‚  165.89 ms β”‚                                      162.65 ms β”‚     no change β”‚
   β”‚ QQuery 58 β”‚  106.55 ms β”‚                                      104.02 ms β”‚     no change β”‚
   β”‚ QQuery 59 β”‚  113.81 ms β”‚                                      113.38 ms β”‚     no change β”‚
   β”‚ QQuery 60 β”‚   37.81 ms β”‚                                       38.08 ms β”‚     no change β”‚
   β”‚ QQuery 61 β”‚    8.71 ms β”‚                                        8.94 ms β”‚     no change β”‚
   β”‚ QQuery 62 β”‚   43.65 ms β”‚                                       42.22 ms β”‚     no change β”‚
   β”‚ QQuery 63 β”‚   29.25 ms β”‚                                       29.12 ms β”‚     no change β”‚
   β”‚ QQuery 64 β”‚  345.00 ms β”‚                                      344.96 ms β”‚     no change β”‚
   β”‚ QQuery 65 β”‚   66.38 ms β”‚                                       66.18 ms β”‚     no change β”‚
   β”‚ QQuery 66 β”‚   72.58 ms β”‚                                       71.39 ms β”‚     no change β”‚
   β”‚ QQuery 67 β”‚  242.62 ms β”‚                                      237.30 ms β”‚     no change β”‚
   β”‚ QQuery 68 β”‚    6.30 ms β”‚                                        6.25 ms β”‚     no change β”‚
   β”‚ QQuery 69 β”‚   54.81 ms β”‚                                       54.88 ms β”‚     no change β”‚
   β”‚ QQuery 70 β”‚  102.19 ms β”‚                                      101.65 ms β”‚     no change β”‚
   β”‚ QQuery 71 β”‚   34.95 ms β”‚                                       34.62 ms β”‚     no change β”‚
   β”‚ QQuery 72 β”‚ 1623.30 ms β”‚                                     1668.33 ms β”‚     no change β”‚
   β”‚ QQuery 73 β”‚    5.02 ms β”‚                                        5.09 ms β”‚     no change β”‚
   β”‚ QQuery 74 β”‚  185.97 ms β”‚                                      188.37 ms β”‚     no change β”‚
   β”‚ QQuery 75 β”‚  136.50 ms β”‚                                      136.84 ms β”‚     no change β”‚
   β”‚ QQuery 76 β”‚   33.94 ms β”‚                                       33.92 ms β”‚     no change β”‚
   β”‚ QQuery 77 β”‚   58.42 ms β”‚                                       59.52 ms β”‚     no change β”‚
   β”‚ QQuery 78 β”‚  262.32 ms β”‚                                      258.69 ms β”‚     no change β”‚
   β”‚ QQuery 79 β”‚   65.40 ms β”‚                                       65.48 ms β”‚     no change β”‚
   β”‚ QQuery 80 β”‚   92.21 ms β”‚                                       92.40 ms β”‚     no change β”‚
   β”‚ QQuery 81 β”‚   25.39 ms β”‚                                       25.33 ms β”‚     no change β”‚
   β”‚ QQuery 82 β”‚   16.59 ms β”‚                                       16.16 ms β”‚     no change β”‚
   β”‚ QQuery 83 β”‚   32.52 ms β”‚                                       31.77 ms β”‚     no change β”‚
   β”‚ QQuery 84 β”‚   29.45 ms β”‚                                       28.72 ms β”‚     no change β”‚
   β”‚ QQuery 85 β”‚   96.19 ms β”‚                                       95.95 ms β”‚     no change β”‚
   β”‚ QQuery 86 β”‚   24.78 ms β”‚                                       25.23 ms β”‚     no change β”‚
   β”‚ QQuery 87 β”‚   59.71 ms β”‚                                       60.03 ms β”‚     no change β”‚
   β”‚ QQuery 88 β”‚   58.64 ms β”‚                                       59.18 ms β”‚     no change β”‚
   β”‚ QQuery 89 β”‚   35.19 ms β”‚                                       34.86 ms β”‚     no change β”‚
   β”‚ QQuery 90 β”‚   15.81 ms β”‚                                       16.10 ms β”‚     no change β”‚
   β”‚ QQuery 91 β”‚   39.90 ms β”‚                                       40.30 ms β”‚     no change β”‚
   β”‚ QQuery 92 β”‚   28.33 ms β”‚                                       28.57 ms β”‚     no change β”‚
   β”‚ QQuery 93 β”‚   48.46 ms β”‚                                       48.17 ms β”‚     no change β”‚
   β”‚ QQuery 94 β”‚   37.12 ms β”‚                                       37.67 ms β”‚     no change β”‚
   β”‚ QQuery 95 β”‚   77.39 ms β”‚                                       78.02 ms β”‚     no change β”‚
   β”‚ QQuery 96 β”‚   23.20 ms β”‚                                       23.29 ms β”‚     no change β”‚
   β”‚ QQuery 97 β”‚   50.08 ms β”‚                                       50.03 ms β”‚     no change β”‚
   β”‚ QQuery 98 β”‚   42.16 ms β”‚                                       42.09 ms β”‚     no change β”‚
   β”‚ QQuery 99 β”‚   64.96 ms β”‚                                       63.04 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 8730.71ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 8736.74ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚   88.19ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚   88.25ms β”‚
   β”‚ Queries Faster                                                β”‚         1 β”‚
   β”‚ Queries Slower                                                β”‚         1 β”‚
   β”‚ Queries with No Change                                        β”‚        97 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   Distribution per query (min / mean Β±stddev / max):
   
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚           5.31 / 5.91 Β±1.01 / 7.92 ms β”‚                    5.20 / 5.75 Β±1.03 / 7.81 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚        80.69 / 81.10 Β±0.33 / 81.55 ms β”‚                 78.24 / 78.64 Β±0.23 / 78.91 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚        28.86 / 29.26 Β±0.23 / 29.47 ms β”‚                 28.77 / 28.98 Β±0.15 / 29.15 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚     436.22 / 439.89 Β±4.71 / 449.04 ms β”‚              424.17 / 434.21 Β±6.14 / 442.07 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚        50.53 / 50.63 Β±0.10 / 50.80 ms β”‚                 49.13 / 49.53 Β±0.24 / 49.74 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚        35.02 / 35.57 Β±0.38 / 36.07 ms β”‚                 34.29 / 36.15 Β±2.68 / 41.48 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚        71.76 / 73.44 Β±1.24 / 75.24 ms β”‚                 71.40 / 72.10 Β±0.65 / 73.30 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚        36.95 / 37.28 Β±0.43 / 38.12 ms β”‚                 36.14 / 36.53 Β±0.27 / 36.84 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚        52.60 / 54.66 Β±1.11 / 55.62 ms β”‚                 50.40 / 52.48 Β±1.13 / 53.72 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚        60.84 / 61.27 Β±0.46 / 62.02 ms β”‚                 60.41 / 61.98 Β±2.51 / 66.98 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚     264.20 / 269.09 Β±4.88 / 277.44 ms β”‚              273.78 / 276.00 Β±2.01 / 278.78 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚        28.25 / 28.65 Β±0.35 / 29.19 ms β”‚                 28.28 / 29.08 Β±0.46 / 29.54 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚     111.12 / 111.88 Β±0.53 / 112.43 ms β”‚              111.97 / 113.37 Β±1.03 / 115.19 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚     372.62 / 376.65 Β±2.66 / 379.59 ms β”‚              370.84 / 375.12 Β±2.59 / 378.94 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚        53.88 / 55.82 Β±3.32 / 62.44 ms β”‚                 52.11 / 54.17 Β±1.76 / 57.08 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚           5.75 / 5.87 Β±0.21 / 6.29 ms β”‚                    5.74 / 5.92 Β±0.27 / 6.45 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚        74.90 / 75.60 Β±0.61 / 76.35 ms β”‚                 74.50 / 75.23 Β±0.41 / 75.63 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚     102.73 / 104.34 Β±1.79 / 107.65 ms β”‚              103.13 / 104.36 Β±1.13 / 106.11 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚        40.43 / 40.65 Β±0.13 / 40.82 ms β”‚                 40.45 / 41.25 Β±0.47 / 41.91 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚        35.41 / 36.24 Β±0.49 / 36.87 ms β”‚                 35.24 / 35.99 Β±0.74 / 36.97 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚        16.38 / 16.63 Β±0.21 / 16.97 ms β”‚                 17.42 / 17.48 Β±0.06 / 17.58 ms β”‚  1.05x slower β”‚
   β”‚ QQuery 22 β”‚        61.48 / 62.44 Β±0.71 / 63.69 ms β”‚                 62.35 / 63.81 Β±1.46 / 65.72 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚     289.39 / 291.60 Β±2.03 / 294.70 ms β”‚              288.75 / 289.82 Β±1.33 / 292.40 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚     186.96 / 192.49 Β±4.39 / 200.41 ms β”‚              187.35 / 188.54 Β±0.97 / 190.33 ms β”‚     no change β”‚
   β”‚ QQuery 25 β”‚     104.17 / 105.90 Β±1.18 / 107.30 ms β”‚              103.15 / 104.86 Β±1.26 / 106.38 ms β”‚     no change β”‚
   β”‚ QQuery 26 β”‚        45.60 / 46.96 Β±2.39 / 51.74 ms β”‚                 44.87 / 45.34 Β±0.41 / 46.00 ms β”‚     no change β”‚
   β”‚ QQuery 27 β”‚           4.80 / 4.93 Β±0.20 / 5.32 ms β”‚                    4.80 / 4.98 Β±0.17 / 5.31 ms β”‚     no change β”‚
   β”‚ QQuery 28 β”‚        60.70 / 61.29 Β±0.42 / 61.73 ms β”‚                 56.60 / 59.24 Β±1.96 / 60.95 ms β”‚     no change β”‚
   β”‚ QQuery 29 β”‚        91.31 / 92.02 Β±0.92 / 93.80 ms β”‚                 90.57 / 92.39 Β±1.37 / 94.50 ms β”‚     no change β”‚
   β”‚ QQuery 30 β”‚        31.44 / 31.84 Β±0.35 / 32.27 ms β”‚                 32.13 / 32.52 Β±0.31 / 32.90 ms β”‚     no change β”‚
   β”‚ QQuery 31 β”‚     105.87 / 107.45 Β±2.49 / 112.42 ms β”‚              106.96 / 108.11 Β±0.71 / 109.17 ms β”‚     no change β”‚
   β”‚ QQuery 32 β”‚        19.77 / 20.56 Β±0.78 / 22.03 ms β”‚                 19.96 / 20.34 Β±0.23 / 20.64 ms β”‚     no change β”‚
   β”‚ QQuery 33 β”‚        36.56 / 36.97 Β±0.33 / 37.44 ms β”‚                 36.35 / 36.71 Β±0.22 / 37.06 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚           5.43 / 5.68 Β±0.26 / 6.03 ms β”‚                    5.38 / 5.53 Β±0.23 / 5.98 ms β”‚     no change β”‚
   β”‚ QQuery 35 β”‚        69.81 / 71.96 Β±2.39 / 76.63 ms β”‚                 70.55 / 72.63 Β±2.45 / 77.05 ms β”‚     no change β”‚
   β”‚ QQuery 36 β”‚           5.42 / 5.56 Β±0.21 / 5.98 ms β”‚                    5.41 / 5.59 Β±0.25 / 6.08 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚           6.83 / 6.97 Β±0.18 / 7.32 ms β”‚                    6.60 / 6.91 Β±0.17 / 7.08 ms β”‚     no change β”‚
   β”‚ QQuery 38 β”‚        59.69 / 60.20 Β±0.40 / 60.71 ms β”‚                 59.54 / 60.35 Β±0.64 / 61.13 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚        72.48 / 73.66 Β±1.26 / 75.93 ms β”‚                 70.76 / 71.83 Β±1.00 / 73.35 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚        22.07 / 23.74 Β±2.57 / 28.86 ms β”‚                 22.01 / 22.39 Β±0.45 / 23.24 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 41 β”‚        12.33 / 12.61 Β±0.19 / 12.83 ms β”‚                 12.25 / 13.42 Β±2.01 / 17.43 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 42 β”‚        23.72 / 24.30 Β±0.41 / 24.97 ms β”‚                 23.14 / 23.71 Β±0.41 / 24.35 ms β”‚     no change β”‚
   β”‚ QQuery 43 β”‚           4.46 / 4.61 Β±0.23 / 5.07 ms β”‚                    4.36 / 4.47 Β±0.15 / 4.78 ms β”‚     no change β”‚
   β”‚ QQuery 44 β”‚           5.38 / 5.48 Β±0.14 / 5.76 ms β”‚                    5.31 / 5.42 Β±0.12 / 5.62 ms β”‚     no change β”‚
   β”‚ QQuery 45 β”‚        36.08 / 36.95 Β±0.79 / 38.01 ms β”‚                 35.05 / 36.03 Β±0.76 / 37.18 ms β”‚     no change β”‚
   β”‚ QQuery 46 β”‚        11.51 / 12.05 Β±0.33 / 12.52 ms β”‚                 11.24 / 11.51 Β±0.15 / 11.67 ms β”‚     no change β”‚
   β”‚ QQuery 47 β”‚     220.06 / 224.80 Β±2.55 / 227.37 ms β”‚              214.14 / 217.91 Β±2.72 / 221.63 ms β”‚     no change β”‚
   β”‚ QQuery 48 β”‚        90.92 / 93.11 Β±3.42 / 99.89 ms β”‚                 90.84 / 91.29 Β±0.42 / 91.95 ms β”‚     no change β”‚
   β”‚ QQuery 49 β”‚        69.19 / 70.36 Β±0.64 / 71.06 ms β”‚                 69.29 / 70.35 Β±1.18 / 72.41 ms β”‚     no change β”‚
   β”‚ QQuery 50 β”‚        56.51 / 57.13 Β±0.49 / 57.69 ms β”‚                 56.29 / 56.64 Β±0.21 / 56.89 ms β”‚     no change β”‚
   β”‚ QQuery 51 β”‚       91.56 / 95.84 Β±3.25 / 100.48 ms β”‚                 93.52 / 95.84 Β±1.49 / 97.96 ms β”‚     no change β”‚
   β”‚ QQuery 52 β”‚        23.73 / 23.93 Β±0.28 / 24.49 ms β”‚                 23.97 / 24.50 Β±0.49 / 25.30 ms β”‚     no change β”‚
   β”‚ QQuery 53 β”‚        28.69 / 28.79 Β±0.11 / 28.98 ms β”‚                 29.25 / 29.51 Β±0.25 / 29.85 ms β”‚     no change β”‚
   β”‚ QQuery 54 β”‚        52.80 / 53.40 Β±0.39 / 53.87 ms β”‚                 52.86 / 53.33 Β±0.39 / 54.02 ms β”‚     no change β”‚
   β”‚ QQuery 55 β”‚        22.90 / 23.26 Β±0.25 / 23.66 ms β”‚                 23.30 / 24.20 Β±1.19 / 26.55 ms β”‚     no change β”‚
   β”‚ QQuery 56 β”‚        37.19 / 39.24 Β±2.71 / 44.57 ms β”‚                 37.47 / 37.72 Β±0.23 / 38.11 ms β”‚     no change β”‚
   β”‚ QQuery 57 β”‚     165.89 / 167.41 Β±1.48 / 170.11 ms β”‚              162.65 / 167.48 Β±3.84 / 174.33 ms β”‚     no change β”‚
   β”‚ QQuery 58 β”‚     106.55 / 109.25 Β±3.51 / 116.08 ms β”‚              104.02 / 105.73 Β±2.11 / 109.85 ms β”‚     no change β”‚
   β”‚ QQuery 59 β”‚     113.81 / 115.00 Β±0.80 / 116.30 ms β”‚              113.38 / 114.13 Β±0.56 / 114.86 ms β”‚     no change β”‚
   β”‚ QQuery 60 β”‚        37.81 / 38.01 Β±0.17 / 38.29 ms β”‚                 38.08 / 39.10 Β±1.30 / 41.65 ms β”‚     no change β”‚
   β”‚ QQuery 61 β”‚           8.71 / 8.92 Β±0.26 / 9.43 ms β”‚                    8.94 / 9.08 Β±0.24 / 9.57 ms β”‚     no change β”‚
   β”‚ QQuery 62 β”‚        43.65 / 43.98 Β±0.27 / 44.42 ms β”‚                 42.22 / 42.61 Β±0.23 / 42.90 ms β”‚     no change β”‚
   β”‚ QQuery 63 β”‚        29.25 / 29.47 Β±0.21 / 29.83 ms β”‚                 29.12 / 29.43 Β±0.25 / 29.84 ms β”‚     no change β”‚
   β”‚ QQuery 64 β”‚     345.00 / 351.30 Β±6.55 / 363.25 ms β”‚              344.96 / 349.76 Β±3.42 / 354.70 ms β”‚     no change β”‚
   β”‚ QQuery 65 β”‚        66.38 / 67.11 Β±0.47 / 67.54 ms β”‚                 66.18 / 66.90 Β±0.65 / 68.05 ms β”‚     no change β”‚
   β”‚ QQuery 66 β”‚        72.58 / 74.30 Β±2.60 / 79.48 ms β”‚                 71.39 / 72.38 Β±0.92 / 73.95 ms β”‚     no change β”‚
   β”‚ QQuery 67 β”‚     242.62 / 249.47 Β±8.18 / 264.49 ms β”‚              237.30 / 244.47 Β±5.54 / 253.40 ms β”‚     no change β”‚
   β”‚ QQuery 68 β”‚           6.30 / 6.53 Β±0.21 / 6.90 ms β”‚                    6.25 / 6.41 Β±0.22 / 6.83 ms β”‚     no change β”‚
   β”‚ QQuery 69 β”‚        54.81 / 55.47 Β±0.43 / 56.00 ms β”‚                 54.88 / 55.24 Β±0.32 / 55.69 ms β”‚     no change β”‚
   β”‚ QQuery 70 β”‚     102.19 / 105.32 Β±5.00 / 115.24 ms β”‚              101.65 / 102.87 Β±1.04 / 104.20 ms β”‚     no change β”‚
   β”‚ QQuery 71 β”‚        34.95 / 35.27 Β±0.33 / 35.84 ms β”‚                 34.62 / 38.18 Β±5.70 / 49.54 ms β”‚  1.08x slower β”‚
   β”‚ QQuery 72 β”‚ 1623.30 / 1659.13 Β±31.82 / 1700.01 ms β”‚          1668.33 / 1743.19 Β±62.78 / 1820.04 ms β”‚  1.05x slower β”‚
   β”‚ QQuery 73 β”‚           5.02 / 5.19 Β±0.24 / 5.66 ms β”‚                    5.09 / 5.24 Β±0.24 / 5.72 ms β”‚     no change β”‚
   β”‚ QQuery 74 β”‚     185.97 / 187.47 Β±1.24 / 188.78 ms β”‚              188.37 / 189.96 Β±2.79 / 195.54 ms β”‚     no change β”‚
   β”‚ QQuery 75 β”‚     136.50 / 137.71 Β±1.15 / 139.45 ms β”‚              136.84 / 139.25 Β±3.22 / 145.36 ms β”‚     no change β”‚
   β”‚ QQuery 76 β”‚        33.94 / 34.23 Β±0.26 / 34.57 ms β”‚                 33.92 / 34.46 Β±0.47 / 35.34 ms β”‚     no change β”‚
   β”‚ QQuery 77 β”‚        58.42 / 58.99 Β±0.67 / 60.28 ms β”‚                 59.52 / 59.82 Β±0.24 / 60.18 ms β”‚     no change β”‚
   β”‚ QQuery 78 β”‚    262.32 / 277.47 Β±12.03 / 295.80 ms β”‚             258.69 / 269.65 Β±10.05 / 288.47 ms β”‚     no change β”‚
   β”‚ QQuery 79 β”‚        65.40 / 65.89 Β±0.54 / 66.90 ms β”‚                 65.48 / 66.05 Β±0.59 / 67.10 ms β”‚     no change β”‚
   β”‚ QQuery 80 β”‚       92.21 / 95.69 Β±5.03 / 105.66 ms β”‚                92.40 / 97.10 Β±6.96 / 110.96 ms β”‚     no change β”‚
   β”‚ QQuery 81 β”‚        25.39 / 27.36 Β±3.40 / 34.16 ms β”‚                 25.33 / 25.49 Β±0.18 / 25.80 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 82 β”‚        16.59 / 17.49 Β±0.87 / 19.16 ms β”‚                 16.16 / 16.46 Β±0.31 / 17.04 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 83 β”‚        32.52 / 33.02 Β±0.28 / 33.30 ms β”‚                 31.77 / 32.21 Β±0.30 / 32.54 ms β”‚     no change β”‚
   β”‚ QQuery 84 β”‚        29.45 / 29.81 Β±0.31 / 30.24 ms β”‚                 28.72 / 29.08 Β±0.20 / 29.33 ms β”‚     no change β”‚
   β”‚ QQuery 85 β”‚       96.19 / 98.37 Β±3.45 / 105.25 ms β”‚                95.95 / 99.60 Β±4.13 / 107.09 ms β”‚     no change β”‚
   β”‚ QQuery 86 β”‚        24.78 / 26.55 Β±3.08 / 32.70 ms β”‚                 25.23 / 25.77 Β±0.47 / 26.53 ms β”‚     no change β”‚
   β”‚ QQuery 87 β”‚        59.71 / 61.15 Β±1.61 / 64.06 ms β”‚                 60.03 / 61.35 Β±0.80 / 62.51 ms β”‚     no change β”‚
   β”‚ QQuery 88 β”‚        58.64 / 59.95 Β±2.00 / 63.93 ms β”‚                 59.18 / 59.83 Β±0.49 / 60.31 ms β”‚     no change β”‚
   β”‚ QQuery 89 β”‚        35.19 / 35.71 Β±0.66 / 36.98 ms β”‚                 34.86 / 35.69 Β±0.82 / 36.89 ms β”‚     no change β”‚
   β”‚ QQuery 90 β”‚        15.81 / 16.04 Β±0.27 / 16.54 ms β”‚                 16.10 / 16.34 Β±0.22 / 16.67 ms β”‚     no change β”‚
   β”‚ QQuery 91 β”‚        39.90 / 40.20 Β±0.25 / 40.48 ms β”‚                 40.30 / 40.63 Β±0.26 / 40.95 ms β”‚     no change β”‚
   β”‚ QQuery 92 β”‚        28.33 / 28.85 Β±0.39 / 29.35 ms β”‚                 28.57 / 29.24 Β±0.45 / 29.75 ms β”‚     no change β”‚
   β”‚ QQuery 93 β”‚        48.46 / 49.43 Β±0.65 / 50.33 ms β”‚                 48.17 / 49.53 Β±1.50 / 52.41 ms β”‚     no change β”‚
   β”‚ QQuery 94 β”‚        37.12 / 38.17 Β±1.05 / 40.12 ms β”‚                 37.67 / 37.85 Β±0.17 / 38.17 ms β”‚     no change β”‚
   β”‚ QQuery 95 β”‚        77.39 / 78.70 Β±1.26 / 80.94 ms β”‚                 78.02 / 79.63 Β±1.88 / 83.21 ms β”‚     no change β”‚
   β”‚ QQuery 96 β”‚        23.20 / 23.50 Β±0.21 / 23.75 ms β”‚                 23.29 / 23.52 Β±0.23 / 23.94 ms β”‚     no change β”‚
   β”‚ QQuery 97 β”‚        50.08 / 51.22 Β±1.09 / 53.08 ms β”‚                 50.03 / 51.66 Β±1.20 / 52.99 ms β”‚     no change β”‚
   β”‚ QQuery 98 β”‚        42.16 / 42.33 Β±0.18 / 42.68 ms β”‚                 42.09 / 43.05 Β±0.64 / 44.07 ms β”‚     no change β”‚
   β”‚ QQuery 99 β”‚        64.96 / 65.12 Β±0.27 / 65.65 ms β”‚                 63.04 / 65.14 Β±3.27 / 71.62 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 8900.11ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 8942.62ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚   89.90ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚   90.33ms β”‚
   β”‚ Queries Faster                                                β”‚         3 β”‚
   β”‚ Queries Slower                                                β”‚         4 β”‚
   β”‚ Queries with No Change                                        β”‚        92 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 45.0s |
   | Peak memory | 2.3 GiB |
   | Avg memory | 1.7 GiB |
   | CPU user | 185.2s |
   | CPU sys | 5.4s |
   | Peak spill | 0 B |
   
   **tpcds β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.0 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 186.7s |
   | CPU sys | 5.3s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4584536313


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right table declares a different ordering than the file holds, so only the left ordering is adopted
+# and the right input is sorted
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--SortExec: expr=[k1@0 DESC NULLS LAST, k2@1 ASC], preserve_partitioning=[false]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 DESC NULLS LAST, k2@1 ASC], reverse_row_groups=true
+
+# the join keys are listed in a different order than the input ordering: the keys are reordered
+# to follow the ordering, so no SortExec is needed
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k2 = r.k2, l.k1 = r.k1
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+2 b 20 200
+3 c 30 300
+
+# only the leading ordering column is a join key: its options are adopted and no SortExec is needed
+query TT
+EXPLAIN SELECT l.k1, v, w FROM l JOIN r ON l.k1 = r.k1
+----
+logical_plan
+01)Projection: l.k1, l.v, r.w
+02)--Inner Join: l.k1 = r.k1
+03)----TableScan: l projection=[k1, v]
+04)----TableScan: r projection=[k1, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0)], projection=[k1@0, v@1, w@3]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, v], output_ordering=[k1@0 DESC NULLS LAST], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, w], output_ordering=[k1@0 DESC NULLS LAST], file_type=parquet
+
+query III rowsort
+SELECT l.k1, v, w FROM l JOIN r ON l.k1 = r.k1
+----
+2 20 200
+3 30 300
+
+# k2 is constant after the filter, so it does not need to be in the ordering: the join adopts the
+# options of the k1 ordering and no SortExec is needed (a single partition avoids a repartition)
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1 WHERE l.k2 = 'b'
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k2 = r.k2, l.k1 = r.k1
+03)----Filter: l.k2 = Utf8View("b")
+04)------TableScan: l projection=[k1, k2, v], partial_filters=[l.k2 = Utf8View("b")]
+05)----Filter: r.k2 = Utf8View("b")
+06)------TableScan: r projection=[k1, k2, w], partial_filters=[r.k2 = Utf8View("b")]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--FilterExec: k2@1 = b
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet, predicate=k2@1 = b, pruning_predicate=k2_null_count@2 != row_count@3 AND k2_min@0 <= b AND b <= k2_max@1, required_guarantees=[k2 in (b)]
+04)--FilterExec: k2@1 = b
+05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet, predicate=k2@1 = b, pruning_predicate=k2_null_count@2 != row_count@3 AND k2_min@0 <= b AND b <= k2_max@1, required_guarantees=[k2 in (b)]
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1 WHERE l.k2 = 'b'
+----
+2 b 20 200
+
+statement ok
+set datafusion.execution.target_partitions = 2;

Review Comment:
   removed.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4267255730


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+
+statement ok
+DROP TABLE l;
+
+statement ok
+DROP TABLE r;
+
+statement ok
+DROP TABLE r_asc;
+
+statement ok
+reset datafusion.execution.target_partitions;

Review Comment:
   This location as well
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4276265044


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet

Review Comment:
   Thanks for resolving this!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "comphead (via GitHub) via github" <gi...@datafusion.apache.org>.
comphead commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859968362

   Thanks @bert-beyondloops looks like SMJ performs better, I filtered out noise. I'll review the PR soon


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub)" <gi...@apache.org>.
bert-beyondloops commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6164927931

   I applied a patch built on top of the 55.1 release. Since then, SortMergeJoin on main has gained embedded projection support, so I updated the affected explain plans to reflect that change.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4584509270


##########
datafusion/core/src/physical_planner.rs:
##########
@@ -1902,14 +1902,18 @@ impl DefaultPhysicalPlanner {
                     // Use ordinary sort-merge join when hash join is not preferred.
                     // Null-aware joins not handled above fall back to CollectLeft
                     // HashJoin because their semantics are not supported by sort-merge join.
-                    let join_on_len = join_on.len();
+                    // The join keys are an unordered set of equalities, so they may be
+                    // reshuffled (together with their sort options) to match the input
+                    // orderings and avoid sorting.

Review Comment:
   removed the comments



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859861099

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃       HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚    5.72 ms β”‚                                        5.72 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚   80.33 ms β”‚                                       79.43 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚   29.12 ms β”‚                                       29.00 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚  483.26 ms β”‚                                      459.64 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚   51.17 ms β”‚                                       50.61 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚   35.63 ms β”‚                                       35.53 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚   72.21 ms β”‚                                       72.42 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚   36.57 ms β”‚                                       36.57 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚   52.54 ms β”‚                                       52.65 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚   60.67 ms β”‚                                       60.52 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚  310.04 ms β”‚                                      287.79 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 12 β”‚   29.65 ms β”‚                                       29.55 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚  113.75 ms β”‚                                      113.75 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚  384.21 ms β”‚                                      376.71 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚   59.33 ms β”‚                                       56.51 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚    6.53 ms β”‚                                        6.43 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚   76.83 ms β”‚                                       76.37 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚  107.98 ms β”‚                                      105.81 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚   41.70 ms β”‚                                       41.46 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚   36.56 ms β”‚                                       35.87 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚   17.42 ms β”‚                                       17.07 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚   67.02 ms β”‚                                       65.03 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚  305.67 ms β”‚                                      300.38 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚  200.93 ms β”‚                                      198.60 ms β”‚     no change β”‚
   β”‚ QQuery 25 β”‚  104.84 ms β”‚                                      105.77 ms β”‚     no change β”‚
   β”‚ QQuery 26 β”‚   46.22 ms β”‚                                       46.05 ms β”‚     no change β”‚
   β”‚ QQuery 27 β”‚    5.31 ms β”‚                                        5.28 ms β”‚     no change β”‚
   β”‚ QQuery 28 β”‚   61.98 ms β”‚                                       58.12 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 29 β”‚   92.12 ms β”‚                                       91.99 ms β”‚     no change β”‚
   β”‚ QQuery 30 β”‚   33.34 ms β”‚                                       32.85 ms β”‚     no change β”‚
   β”‚ QQuery 31 β”‚  109.38 ms β”‚                                      108.77 ms β”‚     no change β”‚
   β”‚ QQuery 32 β”‚   20.66 ms β”‚                                       20.38 ms β”‚     no change β”‚
   β”‚ QQuery 33 β”‚   37.64 ms β”‚                                       37.30 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚    5.90 ms β”‚                                        5.59 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 35 β”‚   74.56 ms β”‚                                       72.33 ms β”‚     no change β”‚
   β”‚ QQuery 36 β”‚    5.94 ms β”‚                                        5.67 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚    7.24 ms β”‚                                        7.10 ms β”‚     no change β”‚
   β”‚ QQuery 38 β”‚   62.56 ms β”‚                                       60.88 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚   75.03 ms β”‚                                       72.48 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚   23.84 ms β”‚                                       22.92 ms β”‚     no change β”‚
   β”‚ QQuery 41 β”‚   13.11 ms β”‚                                       12.62 ms β”‚     no change β”‚
   β”‚ QQuery 42 β”‚   24.21 ms β”‚                                       23.97 ms β”‚     no change β”‚
   β”‚ QQuery 43 β”‚    4.95 ms β”‚                                        4.42 ms β”‚ +1.12x faster β”‚
   β”‚ QQuery 44 β”‚    5.97 ms β”‚                                        5.42 ms β”‚ +1.10x faster β”‚
   β”‚ QQuery 45 β”‚   39.72 ms β”‚                                       36.94 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 46 β”‚   12.10 ms β”‚                                       11.81 ms β”‚     no change β”‚
   β”‚ QQuery 47 β”‚  237.72 ms β”‚                                      221.08 ms β”‚ +1.08x faster β”‚
   β”‚ QQuery 48 β”‚   91.61 ms β”‚                                       90.48 ms β”‚     no change β”‚
   β”‚ QQuery 49 β”‚   70.87 ms β”‚                                       69.53 ms β”‚     no change β”‚
   β”‚ QQuery 50 β”‚   56.98 ms β”‚                                       56.61 ms β”‚     no change β”‚
   β”‚ QQuery 51 β”‚   94.65 ms β”‚                                       92.35 ms β”‚     no change β”‚
   β”‚ QQuery 52 β”‚   24.60 ms β”‚                                       24.37 ms β”‚     no change β”‚
   β”‚ QQuery 53 β”‚   29.55 ms β”‚                                       29.71 ms β”‚     no change β”‚
   β”‚ QQuery 54 β”‚   54.17 ms β”‚                                       53.55 ms β”‚     no change β”‚
   β”‚ QQuery 55 β”‚   23.76 ms β”‚                                       24.06 ms β”‚     no change β”‚
   β”‚ QQuery 56 β”‚   38.79 ms β”‚                                       38.69 ms β”‚     no change β”‚
   β”‚ QQuery 57 β”‚  172.31 ms β”‚                                      166.49 ms β”‚     no change β”‚
   β”‚ QQuery 58 β”‚  110.11 ms β”‚                                      105.65 ms β”‚     no change β”‚
   β”‚ QQuery 59 β”‚  114.90 ms β”‚                                      115.27 ms β”‚     no change β”‚
   β”‚ QQuery 60 β”‚   38.85 ms β”‚                                       38.76 ms β”‚     no change β”‚
   β”‚ QQuery 61 β”‚    9.25 ms β”‚                                        9.26 ms β”‚     no change β”‚
   β”‚ QQuery 62 β”‚   44.05 ms β”‚                                       43.29 ms β”‚     no change β”‚
   β”‚ QQuery 63 β”‚   29.37 ms β”‚                                       29.83 ms β”‚     no change β”‚
   β”‚ QQuery 64 β”‚  354.45 ms β”‚                                      353.40 ms β”‚     no change β”‚
   β”‚ QQuery 65 β”‚   70.27 ms β”‚                                       68.81 ms β”‚     no change β”‚
   β”‚ QQuery 66 β”‚   75.80 ms β”‚                                       73.55 ms β”‚     no change β”‚
   β”‚ QQuery 67 β”‚  259.15 ms β”‚                                      255.63 ms β”‚     no change β”‚
   β”‚ QQuery 68 β”‚    6.74 ms β”‚                                        6.62 ms β”‚     no change β”‚
   β”‚ QQuery 69 β”‚   56.03 ms β”‚                                       56.32 ms β”‚     no change β”‚
   β”‚ QQuery 70 β”‚  103.26 ms β”‚                                      103.08 ms β”‚     no change β”‚
   β”‚ QQuery 71 β”‚   35.00 ms β”‚                                       35.27 ms β”‚     no change β”‚
   β”‚ QQuery 72 β”‚ 1692.17 ms β”‚                                     1698.45 ms β”‚     no change β”‚
   β”‚ QQuery 73 β”‚    5.43 ms β”‚                                        5.68 ms β”‚     no change β”‚
   β”‚ QQuery 74 β”‚  210.73 ms β”‚                                      203.42 ms β”‚     no change β”‚
   β”‚ QQuery 75 β”‚  139.46 ms β”‚                                      139.79 ms β”‚     no change β”‚
   β”‚ QQuery 76 β”‚   34.46 ms β”‚                                       34.89 ms β”‚     no change β”‚
   β”‚ QQuery 77 β”‚   60.63 ms β”‚                                       60.81 ms β”‚     no change β”‚
   β”‚ QQuery 78 β”‚  285.02 ms β”‚                                      273.54 ms β”‚     no change β”‚
   β”‚ QQuery 79 β”‚   67.48 ms β”‚                                       67.70 ms β”‚     no change β”‚
   β”‚ QQuery 80 β”‚   96.76 ms β”‚                                       94.67 ms β”‚     no change β”‚
   β”‚ QQuery 81 β”‚   26.21 ms β”‚                                       26.01 ms β”‚     no change β”‚
   β”‚ QQuery 82 β”‚   16.78 ms β”‚                                       16.96 ms β”‚     no change β”‚
   β”‚ QQuery 83 β”‚   33.11 ms β”‚                                       33.28 ms β”‚     no change β”‚
   β”‚ QQuery 84 β”‚   29.90 ms β”‚                                       29.66 ms β”‚     no change β”‚
   β”‚ QQuery 85 β”‚   98.08 ms β”‚                                       97.85 ms β”‚     no change β”‚
   β”‚ QQuery 86 β”‚   25.66 ms β”‚                                       25.77 ms β”‚     no change β”‚
   β”‚ QQuery 87 β”‚   63.45 ms β”‚                                       63.74 ms β”‚     no change β”‚
   β”‚ QQuery 88 β”‚   60.09 ms β”‚                                       60.68 ms β”‚     no change β”‚
   β”‚ QQuery 89 β”‚   35.66 ms β”‚                                       35.79 ms β”‚     no change β”‚
   β”‚ QQuery 90 β”‚   16.58 ms β”‚                                       16.80 ms β”‚     no change β”‚
   β”‚ QQuery 91 β”‚   41.15 ms β”‚                                       41.01 ms β”‚     no change β”‚
   β”‚ QQuery 92 β”‚   29.68 ms β”‚                                       30.17 ms β”‚     no change β”‚
   β”‚ QQuery 93 β”‚   49.73 ms β”‚                                       49.29 ms β”‚     no change β”‚
   β”‚ QQuery 94 β”‚   39.58 ms β”‚                                       39.46 ms β”‚     no change β”‚
   β”‚ QQuery 95 β”‚   79.76 ms β”‚                                       79.36 ms β”‚     no change β”‚
   β”‚ QQuery 96 β”‚   23.92 ms β”‚                                       24.03 ms β”‚     no change β”‚
   β”‚ QQuery 97 β”‚   51.20 ms β”‚                                       51.49 ms β”‚     no change β”‚
   β”‚ QQuery 98 β”‚   43.82 ms β”‚                                       42.89 ms β”‚     no change β”‚
   β”‚ QQuery 99 β”‚   65.66 ms β”‚                                       64.62 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 9155.84ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 9011.56ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚   92.48ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚   91.03ms β”‚
   β”‚ Queries Faster                                                β”‚         7 β”‚
   β”‚ Queries Slower                                                β”‚         0 β”‚
   β”‚ Queries with No Change                                        β”‚        92 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   Distribution per query (min / mean Β±stddev / max):
   
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpcds_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                                  HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚           5.72 / 6.26 Β±0.95 / 8.16 ms β”‚                    5.72 / 6.28 Β±0.99 / 8.27 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚        80.33 / 80.61 Β±0.18 / 80.89 ms β”‚                 79.43 / 80.23 Β±0.43 / 80.64 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚        29.12 / 29.42 Β±0.18 / 29.67 ms β”‚                 29.00 / 29.23 Β±0.17 / 29.40 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚     483.26 / 487.22 Β±3.62 / 492.09 ms β”‚              459.64 / 468.53 Β±6.03 / 478.47 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚        51.17 / 52.00 Β±0.72 / 53.29 ms β”‚                 50.61 / 52.33 Β±2.53 / 57.29 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚        35.63 / 35.84 Β±0.20 / 36.22 ms β”‚                 35.53 / 35.88 Β±0.26 / 36.20 ms β”‚     no change β”‚
   β”‚ QQuery 7  β”‚        72.21 / 73.02 Β±0.45 / 73.49 ms β”‚                 72.42 / 72.90 Β±0.42 / 73.39 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚        36.57 / 36.99 Β±0.27 / 37.34 ms β”‚                 36.57 / 37.02 Β±0.32 / 37.35 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚        52.54 / 54.88 Β±2.87 / 60.38 ms β”‚                 52.65 / 54.50 Β±1.56 / 57.39 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚        60.67 / 61.11 Β±0.25 / 61.39 ms β”‚                 60.52 / 61.14 Β±0.34 / 61.56 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚     310.04 / 312.00 Β±1.91 / 315.65 ms β”‚              287.79 / 292.30 Β±2.85 / 295.96 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 12 β”‚        29.65 / 30.11 Β±0.33 / 30.56 ms β”‚                 29.55 / 30.70 Β±1.58 / 33.71 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚     113.75 / 114.79 Β±0.83 / 116.03 ms β”‚              113.75 / 114.19 Β±0.42 / 114.94 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚     384.21 / 388.60 Β±3.26 / 393.86 ms β”‚              376.71 / 381.91 Β±3.84 / 385.63 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚        59.33 / 61.56 Β±2.94 / 67.27 ms β”‚                 56.51 / 57.34 Β±0.56 / 58.14 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 16 β”‚           6.53 / 6.67 Β±0.11 / 6.85 ms β”‚                    6.43 / 6.48 Β±0.05 / 6.53 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚        76.83 / 78.55 Β±1.55 / 81.33 ms β”‚                 76.37 / 76.60 Β±0.29 / 77.14 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚     107.98 / 109.11 Β±1.00 / 110.40 ms β”‚              105.81 / 106.91 Β±1.22 / 109.21 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚        41.70 / 42.20 Β±0.29 / 42.50 ms β”‚                 41.46 / 41.76 Β±0.23 / 42.04 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚        36.56 / 37.33 Β±1.11 / 39.49 ms β”‚                 35.87 / 36.25 Β±0.49 / 37.21 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚        17.42 / 17.54 Β±0.10 / 17.68 ms β”‚                 17.07 / 17.23 Β±0.13 / 17.42 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚        67.02 / 67.85 Β±0.93 / 69.47 ms β”‚                 65.03 / 65.74 Β±0.52 / 66.56 ms β”‚     no change β”‚
   β”‚ QQuery 23 β”‚     305.67 / 311.32 Β±3.78 / 317.35 ms β”‚              300.38 / 304.33 Β±4.54 / 313.17 ms β”‚     no change β”‚
   β”‚ QQuery 24 β”‚     200.93 / 203.52 Β±1.68 / 205.76 ms β”‚              198.60 / 203.38 Β±5.71 / 214.26 ms β”‚     no change β”‚
   β”‚ QQuery 25 β”‚     104.84 / 105.95 Β±0.86 / 107.41 ms β”‚              105.77 / 107.47 Β±1.77 / 109.91 ms β”‚     no change β”‚
   β”‚ QQuery 26 β”‚        46.22 / 47.06 Β±0.87 / 48.71 ms β”‚                 46.05 / 46.52 Β±0.29 / 46.97 ms β”‚     no change β”‚
   β”‚ QQuery 27 β”‚           5.31 / 5.40 Β±0.07 / 5.50 ms β”‚                    5.28 / 5.39 Β±0.10 / 5.53 ms β”‚     no change β”‚
   β”‚ QQuery 28 β”‚        61.98 / 62.48 Β±0.34 / 63.01 ms β”‚                 58.12 / 61.38 Β±1.72 / 63.24 ms β”‚     no change β”‚
   β”‚ QQuery 29 β”‚        92.12 / 95.27 Β±2.44 / 99.26 ms β”‚                91.99 / 95.00 Β±4.63 / 104.14 ms β”‚     no change β”‚
   β”‚ QQuery 30 β”‚        33.34 / 34.41 Β±0.77 / 35.70 ms β”‚                 32.85 / 33.04 Β±0.10 / 33.13 ms β”‚     no change β”‚
   β”‚ QQuery 31 β”‚     109.38 / 109.79 Β±0.48 / 110.43 ms β”‚              108.77 / 109.01 Β±0.22 / 109.42 ms β”‚     no change β”‚
   β”‚ QQuery 32 β”‚        20.66 / 20.91 Β±0.18 / 21.21 ms β”‚                 20.38 / 20.61 Β±0.18 / 20.86 ms β”‚     no change β”‚
   β”‚ QQuery 33 β”‚        37.64 / 37.96 Β±0.32 / 38.52 ms β”‚                 37.30 / 37.80 Β±0.33 / 38.23 ms β”‚     no change β”‚
   β”‚ QQuery 34 β”‚           5.90 / 6.62 Β±1.23 / 9.08 ms β”‚                    5.59 / 5.75 Β±0.19 / 6.09 ms β”‚ +1.15x faster β”‚
   β”‚ QQuery 35 β”‚        74.56 / 75.29 Β±0.75 / 76.70 ms β”‚                 72.33 / 72.87 Β±0.68 / 74.18 ms β”‚     no change β”‚
   β”‚ QQuery 36 β”‚           5.94 / 6.04 Β±0.11 / 6.25 ms β”‚                    5.67 / 5.83 Β±0.21 / 6.24 ms β”‚     no change β”‚
   β”‚ QQuery 37 β”‚           7.24 / 7.34 Β±0.06 / 7.42 ms β”‚                    7.10 / 7.24 Β±0.08 / 7.33 ms β”‚     no change β”‚
   β”‚ QQuery 38 β”‚        62.56 / 63.59 Β±0.59 / 64.38 ms β”‚                 60.88 / 61.64 Β±0.41 / 62.14 ms β”‚     no change β”‚
   β”‚ QQuery 39 β”‚        75.03 / 75.76 Β±0.81 / 77.32 ms β”‚                 72.48 / 74.32 Β±2.90 / 80.04 ms β”‚     no change β”‚
   β”‚ QQuery 40 β”‚        23.84 / 24.54 Β±0.52 / 25.16 ms β”‚                 22.92 / 23.32 Β±0.34 / 23.83 ms β”‚     no change β”‚
   β”‚ QQuery 41 β”‚        13.11 / 13.21 Β±0.05 / 13.27 ms β”‚                 12.62 / 13.20 Β±0.77 / 14.73 ms β”‚     no change β”‚
   β”‚ QQuery 42 β”‚        24.21 / 24.56 Β±0.31 / 24.98 ms β”‚                 23.97 / 24.40 Β±0.48 / 25.23 ms β”‚     no change β”‚
   β”‚ QQuery 43 β”‚           4.95 / 5.06 Β±0.08 / 5.19 ms β”‚                    4.42 / 4.58 Β±0.24 / 5.05 ms β”‚ +1.10x faster β”‚
   β”‚ QQuery 44 β”‚           5.97 / 6.01 Β±0.05 / 6.09 ms β”‚                    5.42 / 5.47 Β±0.05 / 5.54 ms β”‚ +1.10x faster β”‚
   β”‚ QQuery 45 β”‚        39.72 / 42.61 Β±3.18 / 48.78 ms β”‚                 36.94 / 37.50 Β±0.55 / 38.19 ms β”‚ +1.14x faster β”‚
   β”‚ QQuery 46 β”‚        12.10 / 12.54 Β±0.38 / 13.14 ms β”‚                 11.81 / 12.13 Β±0.18 / 12.31 ms β”‚     no change β”‚
   β”‚ QQuery 47 β”‚     237.72 / 244.63 Β±4.03 / 249.26 ms β”‚              221.08 / 228.48 Β±5.43 / 237.88 ms β”‚ +1.07x faster β”‚
   β”‚ QQuery 48 β”‚        91.61 / 92.44 Β±0.67 / 93.59 ms β”‚                 90.48 / 91.70 Β±0.79 / 92.76 ms β”‚     no change β”‚
   β”‚ QQuery 49 β”‚        70.87 / 73.13 Β±2.55 / 77.95 ms β”‚                 69.53 / 70.12 Β±0.54 / 71.05 ms β”‚     no change β”‚
   β”‚ QQuery 50 β”‚        56.98 / 58.18 Β±1.02 / 60.02 ms β”‚                 56.61 / 58.99 Β±2.89 / 64.59 ms β”‚     no change β”‚
   β”‚ QQuery 51 β”‚        94.65 / 96.26 Β±0.91 / 97.13 ms β”‚                 92.35 / 95.31 Β±1.93 / 97.71 ms β”‚     no change β”‚
   β”‚ QQuery 52 β”‚        24.60 / 25.07 Β±0.26 / 25.33 ms β”‚                 24.37 / 24.60 Β±0.18 / 24.85 ms β”‚     no change β”‚
   β”‚ QQuery 53 β”‚        29.55 / 29.83 Β±0.27 / 30.31 ms β”‚                 29.71 / 29.84 Β±0.13 / 30.06 ms β”‚     no change β”‚
   β”‚ QQuery 54 β”‚        54.17 / 55.02 Β±1.02 / 56.96 ms β”‚                 53.55 / 55.76 Β±3.11 / 61.89 ms β”‚     no change β”‚
   β”‚ QQuery 55 β”‚        23.76 / 23.92 Β±0.13 / 24.10 ms β”‚                 24.06 / 24.74 Β±0.78 / 26.16 ms β”‚     no change β”‚
   β”‚ QQuery 56 β”‚        38.79 / 39.02 Β±0.26 / 39.50 ms β”‚                 38.69 / 38.93 Β±0.21 / 39.30 ms β”‚     no change β”‚
   β”‚ QQuery 57 β”‚     172.31 / 173.97 Β±2.39 / 178.67 ms β”‚              166.49 / 170.65 Β±5.20 / 180.68 ms β”‚     no change β”‚
   β”‚ QQuery 58 β”‚     110.11 / 111.62 Β±2.08 / 115.68 ms β”‚              105.65 / 107.10 Β±1.18 / 109.18 ms β”‚     no change β”‚
   β”‚ QQuery 59 β”‚     114.90 / 115.85 Β±1.03 / 117.82 ms β”‚              115.27 / 117.24 Β±2.97 / 123.10 ms β”‚     no change β”‚
   β”‚ QQuery 60 β”‚        38.85 / 39.88 Β±1.20 / 42.16 ms β”‚                 38.76 / 39.43 Β±0.63 / 40.62 ms β”‚     no change β”‚
   β”‚ QQuery 61 β”‚           9.25 / 9.34 Β±0.12 / 9.58 ms β”‚                    9.26 / 9.46 Β±0.12 / 9.63 ms β”‚     no change β”‚
   β”‚ QQuery 62 β”‚        44.05 / 44.38 Β±0.30 / 44.74 ms β”‚                 43.29 / 43.82 Β±0.37 / 44.39 ms β”‚     no change β”‚
   β”‚ QQuery 63 β”‚        29.37 / 29.82 Β±0.42 / 30.55 ms β”‚                 29.83 / 30.52 Β±0.44 / 30.93 ms β”‚     no change β”‚
   β”‚ QQuery 64 β”‚     354.45 / 360.14 Β±4.09 / 365.80 ms β”‚              353.40 / 355.87 Β±2.80 / 360.13 ms β”‚     no change β”‚
   β”‚ QQuery 65 β”‚        70.27 / 71.60 Β±1.19 / 73.62 ms β”‚                 68.81 / 69.78 Β±0.70 / 70.72 ms β”‚     no change β”‚
   β”‚ QQuery 66 β”‚        75.80 / 77.58 Β±2.26 / 82.03 ms β”‚                 73.55 / 74.13 Β±0.37 / 74.57 ms β”‚     no change β”‚
   β”‚ QQuery 67 β”‚     259.15 / 268.46 Β±7.54 / 279.22 ms β”‚              255.63 / 260.99 Β±4.05 / 265.19 ms β”‚     no change β”‚
   β”‚ QQuery 68 β”‚           6.74 / 7.49 Β±1.23 / 9.92 ms β”‚                    6.62 / 6.76 Β±0.12 / 6.97 ms β”‚ +1.11x faster β”‚
   β”‚ QQuery 69 β”‚        56.03 / 56.65 Β±0.43 / 57.27 ms β”‚                 56.32 / 57.40 Β±1.07 / 59.45 ms β”‚     no change β”‚
   β”‚ QQuery 70 β”‚     103.26 / 106.81 Β±5.12 / 116.98 ms β”‚              103.08 / 106.16 Β±2.38 / 110.07 ms β”‚     no change β”‚
   β”‚ QQuery 71 β”‚        35.00 / 35.76 Β±0.62 / 36.79 ms β”‚                 35.27 / 35.56 Β±0.19 / 35.80 ms β”‚     no change β”‚
   β”‚ QQuery 72 β”‚ 1692.17 / 1731.83 Β±39.54 / 1800.13 ms β”‚          1698.45 / 1755.60 Β±35.18 / 1798.48 ms β”‚     no change β”‚
   β”‚ QQuery 73 β”‚           5.43 / 5.55 Β±0.13 / 5.79 ms β”‚                    5.68 / 5.83 Β±0.18 / 6.17 ms β”‚  1.05x slower β”‚
   β”‚ QQuery 74 β”‚     210.73 / 215.82 Β±3.74 / 219.88 ms β”‚              203.42 / 206.18 Β±2.28 / 209.42 ms β”‚     no change β”‚
   β”‚ QQuery 75 β”‚     139.46 / 140.41 Β±0.92 / 142.13 ms β”‚              139.79 / 143.60 Β±5.22 / 153.89 ms β”‚     no change β”‚
   β”‚ QQuery 76 β”‚        34.46 / 36.64 Β±3.42 / 43.46 ms β”‚                 34.89 / 35.62 Β±0.96 / 37.52 ms β”‚     no change β”‚
   β”‚ QQuery 77 β”‚        60.63 / 61.51 Β±0.95 / 62.96 ms β”‚                 60.81 / 61.24 Β±0.48 / 62.09 ms β”‚     no change β”‚
   β”‚ QQuery 78 β”‚     285.02 / 291.89 Β±6.25 / 299.99 ms β”‚             273.54 / 289.03 Β±13.96 / 307.49 ms β”‚     no change β”‚
   β”‚ QQuery 79 β”‚        67.48 / 71.51 Β±6.63 / 84.69 ms β”‚                 67.70 / 70.38 Β±4.36 / 79.06 ms β”‚     no change β”‚
   β”‚ QQuery 80 β”‚       96.76 / 98.46 Β±1.86 / 101.34 ms β”‚                 94.67 / 95.81 Β±0.86 / 96.84 ms β”‚     no change β”‚
   β”‚ QQuery 81 β”‚        26.21 / 26.47 Β±0.17 / 26.63 ms β”‚                 26.01 / 26.40 Β±0.21 / 26.58 ms β”‚     no change β”‚
   β”‚ QQuery 82 β”‚        16.78 / 17.47 Β±0.95 / 19.31 ms β”‚                 16.96 / 17.19 Β±0.25 / 17.64 ms β”‚     no change β”‚
   β”‚ QQuery 83 β”‚        33.11 / 35.59 Β±4.20 / 43.98 ms β”‚                 33.28 / 33.85 Β±0.81 / 35.44 ms β”‚     no change β”‚
   β”‚ QQuery 84 β”‚        29.90 / 30.75 Β±0.84 / 31.90 ms β”‚                 29.66 / 30.56 Β±0.76 / 31.71 ms β”‚     no change β”‚
   β”‚ QQuery 85 β”‚       98.08 / 99.44 Β±0.99 / 100.99 ms β”‚               97.85 / 101.64 Β±5.60 / 112.48 ms β”‚     no change β”‚
   β”‚ QQuery 86 β”‚        25.66 / 25.80 Β±0.07 / 25.86 ms β”‚                 25.77 / 26.25 Β±0.35 / 26.86 ms β”‚     no change β”‚
   β”‚ QQuery 87 β”‚        63.45 / 64.24 Β±0.74 / 65.42 ms β”‚                 63.74 / 64.36 Β±0.42 / 64.90 ms β”‚     no change β”‚
   β”‚ QQuery 88 β”‚        60.09 / 60.76 Β±0.35 / 61.11 ms β”‚                 60.68 / 61.12 Β±0.32 / 61.46 ms β”‚     no change β”‚
   β”‚ QQuery 89 β”‚        35.66 / 35.86 Β±0.22 / 36.19 ms β”‚                 35.79 / 37.86 Β±3.48 / 44.79 ms β”‚  1.06x slower β”‚
   β”‚ QQuery 90 β”‚        16.58 / 16.84 Β±0.19 / 17.06 ms β”‚                 16.80 / 16.90 Β±0.06 / 16.96 ms β”‚     no change β”‚
   β”‚ QQuery 91 β”‚        41.15 / 41.43 Β±0.23 / 41.67 ms β”‚                 41.01 / 41.62 Β±0.43 / 42.11 ms β”‚     no change β”‚
   β”‚ QQuery 92 β”‚        29.68 / 32.08 Β±4.44 / 40.96 ms β”‚                 30.17 / 31.02 Β±0.97 / 32.72 ms β”‚     no change β”‚
   β”‚ QQuery 93 β”‚        49.73 / 50.67 Β±1.10 / 52.77 ms β”‚                 49.29 / 50.04 Β±0.73 / 51.13 ms β”‚     no change β”‚
   β”‚ QQuery 94 β”‚        39.58 / 40.61 Β±1.40 / 43.36 ms β”‚                 39.46 / 41.45 Β±2.73 / 46.56 ms β”‚     no change β”‚
   β”‚ QQuery 95 β”‚        79.76 / 80.88 Β±0.73 / 81.65 ms β”‚                 79.36 / 81.67 Β±1.55 / 84.04 ms β”‚     no change β”‚
   β”‚ QQuery 96 β”‚        23.92 / 24.49 Β±0.71 / 25.86 ms β”‚                 24.03 / 24.35 Β±0.46 / 25.27 ms β”‚     no change β”‚
   β”‚ QQuery 97 β”‚        51.20 / 52.65 Β±2.32 / 57.23 ms β”‚                 51.49 / 52.44 Β±0.93 / 53.94 ms β”‚     no change β”‚
   β”‚ QQuery 98 β”‚        43.82 / 44.65 Β±0.80 / 45.71 ms β”‚                 42.89 / 44.91 Β±2.24 / 49.13 ms β”‚     no change β”‚
   β”‚ QQuery 99 β”‚        65.66 / 66.06 Β±0.29 / 66.57 ms β”‚                 64.62 / 65.07 Β±0.37 / 65.46 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 9328.07ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 9212.91ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚   94.22ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚   93.06ms β”‚
   β”‚ Queries Faster                                                β”‚         8 β”‚
   β”‚ Queries Slower                                                β”‚         2 β”‚
   β”‚ Queries with No Change                                        β”‚        89 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpcds β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.0 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 190.9s |
   | CPU sys | 5.7s |
   | Peak spill | 0 B |
   
   **tpcds β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 50.0s |
   | Peak memory | 2.2 GiB |
   | Avg memory | 1.5 GiB |
   | CPU user | 193.8s |
   | CPU sys | 5.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub)" <gi...@apache.org>.
bert-beyondloops commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6189479270

   @quwin Thanks for doing some benchmarks. 
   
   I adopted another strategy for the implementation:
   
     `sort_merge_join_keys` in `physical_planner.rs` picks the join key order and per-key SortOptions. It works as follows:                                                                                                               
                                                                                                                                                                                                                                      
     - Candidates.                                                                                                                                                                                                                    
       - The start is the provided key order with default sort options.                                                                                                                                                               
       - Further candidates come from each known ordering (oeq_class) of each input. The ordering's entries that are join keys, normalized through the equivalence group, become the leading keys, in that order and with those  options. The remaining keys follow in their provided order with default options.                                                                                                                                             
     - Scoring. Each candidate is scored 0, 1 or 2: the number of inputs that ordering_satisfy confirms are already sorted on it. The default wins ties, so existing plans are unchanged, and the search stops as soon as a candidate scoresΒ 2.                                                                                                                                                                                                                      
     - Cost. There is nothing to do if neither input has an ordering. Otherwise it costs one candidate per known ordering, each checked with two ordering_satisfy calls.                                                              
                                                                                                                                                                                                                                      
     Note on the approach. An earlier iteration of this PR searched all combinations of sort options per key with backtracking, in the provided key order, and adopted them only if both inputs agreed. This version replaces it with 
     the candidate approach above, to avoid the exponential search and the CPU cost. It also covers more cases:                                                                                                                       
                                                                                                                                                                                                                                      
     - keys listed in a different order than the input ordering                                                                                                                                                                       
     - keys that are constant or equal to another column                                                                                                                                                                              
     - adopting the ordering of only one input, so that only the other side needs a SortExec


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6853505729

    Hi @andygrove  and @comphead , could you take a look when you have time? You both worked on the SortMergeJoin perf changes (#26277, #26192), so I'd value your read on this.                                 
                                                                                                                                                                                                                                      
     This PR does two things:                                                                                                                                                                                                         
                                                                                                                                                                                                                                      
     1. **Derives SortMergeJoin sort options from the input ordering.** The join keys are an unordered set of equalities, so the planner now reorders them, together with their sort options, to match the ordering both inputs       
     already declare. This avoids adding a `SortExec` on inputs that are already sorted that way (see `sort_merge_join_sort_options.slt`).                                                                                            
     2. **Allows SortMergeJoin with `prefer_hash_join=false` for a single target partition.** Before, it was only chosen when `target_partitions > 1` and `repartition_joins` was on.   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859718911

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6859664529-4973-6w2x8 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6171014524

   ## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/26261?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `92.18750%` with `5 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 82.79%. Comparing base ([`2c568b4`](https://app.codecov.io/gh/apache/datafusion/commit/2c568b478b3aff18de52609c7237f1c3f494d5b1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`a9d122f`](https://app.codecov.io/gh/apache/datafusion/commit/a9d122f0e312fadf1881f45734cd82bbd6010ab1?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   :warning: Report is 9 commits behind head on main.
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/26261?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/26261?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 92.18% | [1 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/26261?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main   #26261      +/-   ##
   ==========================================
   - Coverage   82.83%   82.79%   -0.04%     
   ==========================================
     Files        1149     1149              
     Lines      452798   454206    +1408     
     Branches   452798   454206    +1408     
   ==========================================
   + Hits       375082   376079     +997     
   - Misses      54893    55267     +374     
   - Partials    22823    22860      +37     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/26261?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4267235546


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]

Review Comment:
   This fallback case has the same projection mismatch.
   ```suggestion
   01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
   02)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   04)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join.slt:
##########
@@ -1029,6 +1029,35 @@ DROP TABLE c;
 statement ok
 set datafusion.execution.target_partitions = 4;
 
+# prefer_hash_join = false also selects a sort merge join with a single target partition
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a = t2.a
+----
+logical_plan
+01)Projection: t1.a, t1.b, t2.b
+02)--Inner Join: t1.a = t2.a
+03)----TableScan: t1 projection=[a, b]
+04)----TableScan: t2 projection=[a, b]
+physical_plan
+01)ProjectionExec: expr=[a@0 as a, b@1 as b, b@3 as b]
+02)--SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0)]
+03)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]
+
+query III rowsort
+SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a = t2.a
+----

Review Comment:
   Could we restore the test runner’s initial value of 4 explicitly here and in `sort_merge_join_sort_options.slt`? `RESET` restores the machine default instead. After correcting the snapshots, both files fail with target_partitions: 4 -> 8 on my machine. Setting this back to 4 makes the focused suite pass.
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+
+statement ok
+DROP TABLE l;
+
+statement ok
+DROP TABLE r;
+
+statement ok
+DROP TABLE r_asc;
+
+statement ok
+reset datafusion.execution.target_partitions;

Review Comment:
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6858773064

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6858530257)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃    Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 40.57 ms β”‚                                       40.52 ms β”‚ no change β”‚
   β”‚ QQuery 2  β”‚ 18.24 ms β”‚                                       18.19 ms β”‚ no change β”‚
   β”‚ QQuery 3  β”‚ 28.52 ms β”‚                                       28.46 ms β”‚ no change β”‚
   β”‚ QQuery 4  β”‚ 17.24 ms β”‚                                       17.16 ms β”‚ no change β”‚
   β”‚ QQuery 5  β”‚ 34.98 ms β”‚                                       34.75 ms β”‚ no change β”‚
   β”‚ QQuery 6  β”‚ 13.59 ms β”‚                                       13.49 ms β”‚ no change β”‚
   β”‚ QQuery 7  β”‚ 40.54 ms β”‚                                       40.83 ms β”‚ no change β”‚
   β”‚ QQuery 8  β”‚ 40.51 ms β”‚                                       40.29 ms β”‚ no change β”‚
   β”‚ QQuery 9  β”‚ 47.47 ms β”‚                                       48.80 ms β”‚ no change β”‚
   β”‚ QQuery 10 β”‚ 39.94 ms β”‚                                       40.16 ms β”‚ no change β”‚
   β”‚ QQuery 11 β”‚ 13.20 ms β”‚                                       13.10 ms β”‚ no change β”‚
   β”‚ QQuery 12 β”‚ 17.59 ms β”‚                                       17.94 ms β”‚ no change β”‚
   β”‚ QQuery 13 β”‚ 39.38 ms β”‚                                       39.01 ms β”‚ no change β”‚
   β”‚ QQuery 14 β”‚ 24.40 ms β”‚                                       24.25 ms β”‚ no change β”‚
   β”‚ QQuery 15 β”‚ 30.99 ms β”‚                                       30.79 ms β”‚ no change β”‚
   β”‚ QQuery 16 β”‚ 13.42 ms β”‚                                       13.51 ms β”‚ no change β”‚
   β”‚ QQuery 17 β”‚ 67.34 ms β”‚                                       67.06 ms β”‚ no change β”‚
   β”‚ QQuery 18 β”‚ 40.34 ms β”‚                                       40.45 ms β”‚ no change β”‚
   β”‚ QQuery 19 β”‚ 31.91 ms β”‚                                       32.00 ms β”‚ no change β”‚
   β”‚ QQuery 20 β”‚ 31.82 ms β”‚                                       31.78 ms β”‚ no change β”‚
   β”‚ QQuery 21 β”‚ 54.84 ms β”‚                                       54.83 ms β”‚ no change β”‚
   β”‚ QQuery 22 β”‚ 13.39 ms β”‚                                       13.39 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃          ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 700.20ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 700.77ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  31.83ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  31.85ms β”‚
   β”‚ Queries Faster                                                β”‚        0 β”‚
   β”‚ Queries Slower                                                β”‚        0 β”‚
   β”‚ Queries with No Change                                        β”‚       22 β”‚
   β”‚ Queries with Failure                                          β”‚        0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   Distribution per query (min / mean Β±stddev / max):
   
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃        Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 40.57 / 41.31 Β±1.09 / 43.44 ms β”‚                 40.52 / 41.82 Β±1.51 / 43.84 ms β”‚     no change β”‚
   β”‚ QQuery 2  β”‚ 18.24 / 18.49 Β±0.27 / 18.83 ms β”‚                 18.19 / 18.56 Β±0.22 / 18.82 ms β”‚     no change β”‚
   β”‚ QQuery 3  β”‚ 28.52 / 29.16 Β±0.85 / 30.82 ms β”‚                 28.46 / 29.33 Β±0.65 / 30.30 ms β”‚     no change β”‚
   β”‚ QQuery 4  β”‚ 17.24 / 17.98 Β±0.82 / 19.14 ms β”‚                 17.16 / 17.35 Β±0.16 / 17.63 ms β”‚     no change β”‚
   β”‚ QQuery 5  β”‚ 34.98 / 35.14 Β±0.11 / 35.31 ms β”‚                 34.75 / 34.95 Β±0.19 / 35.20 ms β”‚     no change β”‚
   β”‚ QQuery 6  β”‚ 13.59 / 14.71 Β±1.34 / 16.71 ms β”‚                 13.49 / 13.91 Β±0.65 / 15.20 ms β”‚ +1.06x faster β”‚
   β”‚ QQuery 7  β”‚ 40.54 / 41.75 Β±0.98 / 42.94 ms β”‚                 40.83 / 42.02 Β±1.33 / 44.57 ms β”‚     no change β”‚
   β”‚ QQuery 8  β”‚ 40.51 / 41.98 Β±2.44 / 46.86 ms β”‚                 40.29 / 41.18 Β±1.04 / 43.12 ms β”‚     no change β”‚
   β”‚ QQuery 9  β”‚ 47.47 / 49.26 Β±1.21 / 50.95 ms β”‚                 48.80 / 49.84 Β±1.31 / 52.26 ms β”‚     no change β”‚
   β”‚ QQuery 10 β”‚ 39.94 / 40.29 Β±0.19 / 40.46 ms β”‚                 40.16 / 40.27 Β±0.08 / 40.38 ms β”‚     no change β”‚
   β”‚ QQuery 11 β”‚ 13.20 / 13.51 Β±0.25 / 13.80 ms β”‚                 13.10 / 13.47 Β±0.40 / 14.14 ms β”‚     no change β”‚
   β”‚ QQuery 12 β”‚ 17.59 / 18.19 Β±0.32 / 18.53 ms β”‚                 17.94 / 18.38 Β±0.36 / 18.82 ms β”‚     no change β”‚
   β”‚ QQuery 13 β”‚ 39.38 / 40.35 Β±1.31 / 42.93 ms β”‚                 39.01 / 39.90 Β±1.22 / 42.31 ms β”‚     no change β”‚
   β”‚ QQuery 14 β”‚ 24.40 / 24.62 Β±0.21 / 25.02 ms β”‚                 24.25 / 24.50 Β±0.21 / 24.89 ms β”‚     no change β”‚
   β”‚ QQuery 15 β”‚ 30.99 / 31.41 Β±0.47 / 32.32 ms β”‚                 30.79 / 31.50 Β±0.61 / 32.54 ms β”‚     no change β”‚
   β”‚ QQuery 16 β”‚ 13.42 / 13.59 Β±0.16 / 13.89 ms β”‚                 13.51 / 13.69 Β±0.13 / 13.82 ms β”‚     no change β”‚
   β”‚ QQuery 17 β”‚ 67.34 / 69.07 Β±1.49 / 70.98 ms β”‚                 67.06 / 67.49 Β±0.23 / 67.73 ms β”‚     no change β”‚
   β”‚ QQuery 18 β”‚ 40.34 / 40.84 Β±0.29 / 41.17 ms β”‚                 40.45 / 41.44 Β±0.77 / 42.82 ms β”‚     no change β”‚
   β”‚ QQuery 19 β”‚ 31.91 / 33.28 Β±2.04 / 37.29 ms β”‚                 32.00 / 32.64 Β±0.93 / 34.47 ms β”‚     no change β”‚
   β”‚ QQuery 20 β”‚ 31.82 / 32.10 Β±0.24 / 32.49 ms β”‚                 31.78 / 32.03 Β±0.21 / 32.42 ms β”‚     no change β”‚
   β”‚ QQuery 21 β”‚ 54.84 / 56.46 Β±1.06 / 58.14 ms β”‚                 54.83 / 56.00 Β±0.95 / 57.57 ms β”‚     no change β”‚
   β”‚ QQuery 22 β”‚ 13.39 / 13.86 Β±0.56 / 14.93 ms β”‚                 13.39 / 13.71 Β±0.49 / 14.69 ms β”‚     no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃          ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 717.34ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 713.97ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  32.61ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  32.45ms β”‚
   β”‚ Queries Faster                                                β”‚        1 β”‚
   β”‚ Queries Slower                                                β”‚        0 β”‚
   β”‚ Queries with No Change                                        β”‚       21 β”‚
   β”‚ Queries with Failure                                          β”‚        0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 488.9 MiB |
   | CPU user | 19.2s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   **tpch β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 496.6 MiB |
   | CPU user | 19.1s |
   | CPU sys | 1.6s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4267083950


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet

Review Comment:
   This expected plan seems to be stale. The focused SQL logic run fails the `EXPLAIN`: DataFusion puts the projection on `SortMergeJoinExec`, rather than emitting a separate `ProjectionExec`. 
   
   ```suggestion
   01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
   02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
   03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join.slt:
##########
@@ -1029,6 +1029,35 @@ DROP TABLE c;
 statement ok
 set datafusion.execution.target_partitions = 4;
 
+# prefer_hash_join = false also selects a sort merge join with a single target partition
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT t1.a, t1.b, t2.b FROM t1 JOIN t2 ON t1.a = t2.a
+----
+logical_plan
+01)Projection: t1.a, t1.b, t2.b
+02)--Inner Join: t1.a = t2.a
+03)----TableScan: t1 projection=[a, b]
+04)----TableScan: t2 projection=[a, b]
+physical_plan
+01)ProjectionExec: expr=[a@0 as a, b@1 as b, b@3 as b]
+02)--SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0)]
+03)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]

Review Comment:
   This single-partition `EXPLAIN` snapshot seems to fail. The actual plan puts ```projection=[a@0, b@1, b@3]``` on `SortMergeJoinExec`, rather than emitting the expected `ProjectionExec`.
   ```suggestion
   01)SortMergeJoinExec: join_type=Inner, on=[(a@0, a@0)], projection=[a@0, b@1, b@3]
   02)--SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
   03)----DataSourceExec: partitions=1, partition_sizes=[1]
   04)--SortExec: expr=[a@0 ASC], preserve_partitioning=[false]
   05)----DataSourceExec: partitions=1, partition_sizes=[1]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "comphead (via GitHub) via github" <gi...@datafusion.apache.org>.
comphead commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529

   run benchmark tpch tpcds tpch10
   
   ```
   env:
       PREFER_HASH_JOIN: false
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4584529637


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)

Review Comment:
   Added a full join test and a test which covers NULL keys as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6858593825

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6858530257)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6858530257-4956-nqm8x 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859800253

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃     HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃    Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 41.23 ms β”‚                                       41.09 ms β”‚ no change β”‚
   β”‚ QQuery 2  β”‚ 18.78 ms β”‚                                       18.91 ms β”‚ no change β”‚
   β”‚ QQuery 3  β”‚ 28.86 ms β”‚                                       29.19 ms β”‚ no change β”‚
   β”‚ QQuery 4  β”‚ 17.93 ms β”‚                                       18.18 ms β”‚ no change β”‚
   β”‚ QQuery 5  β”‚ 36.49 ms β”‚                                       36.66 ms β”‚ no change β”‚
   β”‚ QQuery 6  β”‚ 13.99 ms β”‚                                       14.04 ms β”‚ no change β”‚
   β”‚ QQuery 7  β”‚ 43.22 ms β”‚                                       42.29 ms β”‚ no change β”‚
   β”‚ QQuery 8  β”‚ 40.99 ms β”‚                                       41.08 ms β”‚ no change β”‚
   β”‚ QQuery 9  β”‚ 50.91 ms β”‚                                       49.83 ms β”‚ no change β”‚
   β”‚ QQuery 10 β”‚ 41.93 ms β”‚                                       41.41 ms β”‚ no change β”‚
   β”‚ QQuery 11 β”‚ 14.30 ms β”‚                                       13.90 ms β”‚ no change β”‚
   β”‚ QQuery 12 β”‚ 17.94 ms β”‚                                       18.22 ms β”‚ no change β”‚
   β”‚ QQuery 13 β”‚ 42.41 ms β”‚                                       42.49 ms β”‚ no change β”‚
   β”‚ QQuery 14 β”‚ 24.83 ms β”‚                                       24.91 ms β”‚ no change β”‚
   β”‚ QQuery 15 β”‚ 31.75 ms β”‚                                       31.86 ms β”‚ no change β”‚
   β”‚ QQuery 16 β”‚ 13.99 ms β”‚                                       14.19 ms β”‚ no change β”‚
   β”‚ QQuery 17 β”‚ 72.34 ms β”‚                                       72.08 ms β”‚ no change β”‚
   β”‚ QQuery 18 β”‚ 42.63 ms β”‚                                       41.84 ms β”‚ no change β”‚
   β”‚ QQuery 19 β”‚ 32.50 ms β”‚                                       32.67 ms β”‚ no change β”‚
   β”‚ QQuery 20 β”‚ 32.51 ms β”‚                                       32.77 ms β”‚ no change β”‚
   β”‚ QQuery 21 β”‚ 57.66 ms β”‚                                       57.27 ms β”‚ no change β”‚
   β”‚ QQuery 22 β”‚ 14.21 ms β”‚                                       14.31 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃          ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 731.38ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 729.18ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  33.24ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  33.14ms β”‚
   β”‚ Queries Faster                                                β”‚        0 β”‚
   β”‚ Queries Slower                                                β”‚        0 β”‚
   β”‚ Queries with No Change                                        β”‚       22 β”‚
   β”‚ Queries with Failure                                          β”‚        0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   Distribution per query (min / mean Β±stddev / max):
   
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf1.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃                           HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃    Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 41.23 / 42.82 Β±1.26 / 44.37 ms β”‚                 41.09 / 42.49 Β±1.50 / 44.39 ms β”‚ no change β”‚
   β”‚ QQuery 2  β”‚ 18.78 / 19.12 Β±0.20 / 19.38 ms β”‚                 18.91 / 19.09 Β±0.17 / 19.38 ms β”‚ no change β”‚
   β”‚ QQuery 3  β”‚ 28.86 / 29.09 Β±0.14 / 29.22 ms β”‚                 29.19 / 29.68 Β±0.68 / 31.02 ms β”‚ no change β”‚
   β”‚ QQuery 4  β”‚ 17.93 / 18.01 Β±0.07 / 18.11 ms β”‚                 18.18 / 18.40 Β±0.34 / 19.07 ms β”‚ no change β”‚
   β”‚ QQuery 5  β”‚ 36.49 / 37.24 Β±1.26 / 39.77 ms β”‚                 36.66 / 36.79 Β±0.14 / 37.00 ms β”‚ no change β”‚
   β”‚ QQuery 6  β”‚ 13.99 / 14.89 Β±0.99 / 16.10 ms β”‚                 14.04 / 14.22 Β±0.19 / 14.58 ms β”‚ no change β”‚
   β”‚ QQuery 7  β”‚ 43.22 / 44.95 Β±1.55 / 47.26 ms β”‚                 42.29 / 43.94 Β±1.64 / 47.05 ms β”‚ no change β”‚
   β”‚ QQuery 8  β”‚ 40.99 / 41.96 Β±1.74 / 45.43 ms β”‚                 41.08 / 41.49 Β±0.35 / 42.12 ms β”‚ no change β”‚
   β”‚ QQuery 9  β”‚ 50.91 / 51.59 Β±0.36 / 51.95 ms β”‚                 49.83 / 51.22 Β±1.29 / 53.48 ms β”‚ no change β”‚
   β”‚ QQuery 10 β”‚ 41.93 / 43.42 Β±0.93 / 44.25 ms β”‚                 41.41 / 43.23 Β±2.20 / 47.06 ms β”‚ no change β”‚
   β”‚ QQuery 11 β”‚ 14.30 / 14.67 Β±0.56 / 15.77 ms β”‚                 13.90 / 14.14 Β±0.17 / 14.40 ms β”‚ no change β”‚
   β”‚ QQuery 12 β”‚ 17.94 / 18.56 Β±0.38 / 19.11 ms β”‚                 18.22 / 18.44 Β±0.18 / 18.65 ms β”‚ no change β”‚
   β”‚ QQuery 13 β”‚ 42.41 / 43.50 Β±1.29 / 45.96 ms β”‚                 42.49 / 43.78 Β±2.40 / 48.57 ms β”‚ no change β”‚
   β”‚ QQuery 14 β”‚ 24.83 / 24.92 Β±0.08 / 25.05 ms β”‚                 24.91 / 24.98 Β±0.08 / 25.11 ms β”‚ no change β”‚
   β”‚ QQuery 15 β”‚ 31.75 / 32.61 Β±0.85 / 34.00 ms β”‚                 31.86 / 33.48 Β±1.17 / 35.19 ms β”‚ no change β”‚
   β”‚ QQuery 16 β”‚ 13.99 / 14.40 Β±0.39 / 14.92 ms β”‚                 14.19 / 14.32 Β±0.10 / 14.46 ms β”‚ no change β”‚
   β”‚ QQuery 17 β”‚ 72.34 / 73.75 Β±1.94 / 77.54 ms β”‚                 72.08 / 73.03 Β±1.05 / 74.35 ms β”‚ no change β”‚
   β”‚ QQuery 18 β”‚ 42.63 / 43.94 Β±1.09 / 45.76 ms β”‚                 41.84 / 42.50 Β±0.53 / 43.36 ms β”‚ no change β”‚
   β”‚ QQuery 19 β”‚ 32.50 / 32.90 Β±0.59 / 34.07 ms β”‚                 32.67 / 33.40 Β±1.25 / 35.89 ms β”‚ no change β”‚
   β”‚ QQuery 20 β”‚ 32.51 / 32.70 Β±0.23 / 33.12 ms β”‚                 32.77 / 33.26 Β±0.46 / 34.12 ms β”‚ no change β”‚
   β”‚ QQuery 21 β”‚ 57.66 / 58.28 Β±0.39 / 58.88 ms β”‚                 57.27 / 57.88 Β±0.62 / 58.65 ms β”‚ no change β”‚
   β”‚ QQuery 22 β”‚ 14.21 / 14.29 Β±0.06 / 14.38 ms β”‚                 14.31 / 14.41 Β±0.07 / 14.53 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃          ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 747.64ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 744.18ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  33.98ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  33.83ms β”‚
   β”‚ Queries Faster                                                β”‚        0 β”‚
   β”‚ Queries Slower                                                β”‚        0 β”‚
   β”‚ Queries with No Change                                        β”‚       22 β”‚
   β”‚ Queries with Failure                                          β”‚        0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 497.5 MiB |
   | CPU user | 20.1s |
   | CPU sys | 1.8s |
   | Peak spill | 0 B |
   
   **tpch β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 5.0s |
   | Peak memory | 1.2 GiB |
   | Avg memory | 525.1 MiB |
   | CPU user | 20.1s |
   | CPU sys | 1.8s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859738676

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6859664529-4975-nsnwc 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch10
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859860445

   πŸ€– Benchmark completed (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB)
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpch10
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   <details><summary>Details</summary>
   <p>
   
   ```
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃      HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃    Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚ 335.32 ms β”‚                                      336.81 ms β”‚ no change β”‚
   β”‚ QQuery 2  β”‚  91.97 ms β”‚                                       91.01 ms β”‚ no change β”‚
   β”‚ QQuery 3  β”‚ 229.83 ms β”‚                                      229.46 ms β”‚ no change β”‚
   β”‚ QQuery 4  β”‚ 117.54 ms β”‚                                      116.57 ms β”‚ no change β”‚
   β”‚ QQuery 5  β”‚ 360.41 ms β”‚                                      361.33 ms β”‚ no change β”‚
   β”‚ QQuery 6  β”‚ 102.87 ms β”‚                                      103.63 ms β”‚ no change β”‚
   β”‚ QQuery 7  β”‚ 490.60 ms β”‚                                      480.23 ms β”‚ no change β”‚
   β”‚ QQuery 8  β”‚ 366.81 ms β”‚                                      364.94 ms β”‚ no change β”‚
   β”‚ QQuery 9  β”‚ 542.68 ms β”‚                                      542.07 ms β”‚ no change β”‚
   β”‚ QQuery 10 β”‚ 301.36 ms β”‚                                      298.24 ms β”‚ no change β”‚
   β”‚ QQuery 11 β”‚  63.63 ms β”‚                                       63.87 ms β”‚ no change β”‚
   β”‚ QQuery 12 β”‚ 121.22 ms β”‚                                      122.00 ms β”‚ no change β”‚
   β”‚ QQuery 13 β”‚ 320.48 ms β”‚                                      327.81 ms β”‚ no change β”‚
   β”‚ QQuery 14 β”‚ 176.02 ms β”‚                                      174.85 ms β”‚ no change β”‚
   β”‚ QQuery 15 β”‚ 312.74 ms β”‚                                      311.82 ms β”‚ no change β”‚
   β”‚ QQuery 16 β”‚  66.04 ms β”‚                                       66.07 ms β”‚ no change β”‚
   β”‚ QQuery 17 β”‚ 640.83 ms β”‚                                      639.67 ms β”‚ no change β”‚
   β”‚ QQuery 18 β”‚ 446.91 ms β”‚                                      463.56 ms β”‚ no change β”‚
   β”‚ QQuery 19 β”‚ 238.90 ms β”‚                                      237.45 ms β”‚ no change β”‚
   β”‚ QQuery 20 β”‚ 269.04 ms β”‚                                      276.34 ms β”‚ no change β”‚
   β”‚ QQuery 21 β”‚ 692.00 ms β”‚                                      685.35 ms β”‚ no change β”‚
   β”‚ QQuery 22 β”‚  64.23 ms β”‚                                       63.23 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 6351.44ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 6356.32ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  288.70ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  288.92ms β”‚
   β”‚ Queries Faster                                                β”‚         0 β”‚
   β”‚ Queries Slower                                                β”‚         0 β”‚
   β”‚ Queries with No Change                                        β”‚        22 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   
   Distribution per query (min / mean Β±stddev / max):
   
   Comparing HEAD and smj-derive-sort-options-and-target-partition-1
   --------------------
   Benchmark tpch_sf10.json
   --------------------
   ┏━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Query     ┃                               HEAD ┃ smj-derive-sort-options-and-target-partition-1 ┃    Change ┃
   ┑━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ QQuery 1  β”‚  335.32 / 338.29 Β±1.83 / 340.15 ms β”‚              336.81 / 337.99 Β±1.20 / 340.14 ms β”‚ no change β”‚
   β”‚ QQuery 2  β”‚    91.97 / 96.13 Β±4.30 / 104.31 ms β”‚                 91.01 / 93.18 Β±1.49 / 95.00 ms β”‚ no change β”‚
   β”‚ QQuery 3  β”‚  229.83 / 231.90 Β±2.30 / 236.08 ms β”‚              229.46 / 232.51 Β±2.45 / 235.89 ms β”‚ no change β”‚
   β”‚ QQuery 4  β”‚  117.54 / 121.09 Β±2.40 / 124.66 ms β”‚              116.57 / 118.67 Β±1.23 / 119.94 ms β”‚ no change β”‚
   β”‚ QQuery 5  β”‚  360.41 / 367.33 Β±5.44 / 375.14 ms β”‚              361.33 / 372.65 Β±7.87 / 383.01 ms β”‚ no change β”‚
   β”‚ QQuery 6  β”‚  102.87 / 106.24 Β±2.82 / 111.16 ms β”‚              103.63 / 105.65 Β±1.91 / 109.09 ms β”‚ no change β”‚
   β”‚ QQuery 7  β”‚  490.60 / 499.87 Β±7.59 / 509.23 ms β”‚              480.23 / 490.90 Β±5.52 / 496.00 ms β”‚ no change β”‚
   β”‚ QQuery 8  β”‚  366.81 / 369.29 Β±2.91 / 374.27 ms β”‚              364.94 / 369.02 Β±3.65 / 375.39 ms β”‚ no change β”‚
   β”‚ QQuery 9  β”‚ 542.68 / 557.00 Β±19.03 / 593.24 ms β”‚             542.07 / 557.15 Β±17.07 / 588.97 ms β”‚ no change β”‚
   β”‚ QQuery 10 β”‚ 301.36 / 317.03 Β±16.96 / 349.19 ms β”‚              298.24 / 309.53 Β±8.72 / 323.15 ms β”‚ no change β”‚
   β”‚ QQuery 11 β”‚     63.63 / 64.88 Β±1.06 / 66.47 ms β”‚                 63.87 / 67.17 Β±3.31 / 73.12 ms β”‚ no change β”‚
   β”‚ QQuery 12 β”‚  121.22 / 127.62 Β±9.04 / 145.47 ms β”‚              122.00 / 128.67 Β±8.88 / 146.26 ms β”‚ no change β”‚
   β”‚ QQuery 13 β”‚  320.48 / 332.32 Β±7.86 / 340.40 ms β”‚             327.81 / 348.86 Β±30.93 / 410.23 ms β”‚ no change β”‚
   β”‚ QQuery 14 β”‚  176.02 / 180.86 Β±4.47 / 187.73 ms β”‚              174.85 / 182.23 Β±6.80 / 192.79 ms β”‚ no change β”‚
   β”‚ QQuery 15 β”‚  312.74 / 316.40 Β±3.41 / 321.80 ms β”‚              311.82 / 316.75 Β±2.95 / 320.53 ms β”‚ no change β”‚
   β”‚ QQuery 16 β”‚     66.04 / 68.96 Β±3.16 / 74.74 ms β”‚                 66.07 / 71.52 Β±7.30 / 86.00 ms β”‚ no change β”‚
   β”‚ QQuery 17 β”‚  640.83 / 650.21 Β±6.19 / 655.91 ms β”‚              639.67 / 643.63 Β±6.48 / 656.54 ms β”‚ no change β”‚
   β”‚ QQuery 18 β”‚ 446.91 / 475.54 Β±19.44 / 505.52 ms β”‚             463.56 / 471.36 Β±10.09 / 490.97 ms β”‚ no change β”‚
   β”‚ QQuery 19 β”‚ 238.90 / 249.47 Β±10.49 / 263.69 ms β”‚             237.45 / 247.65 Β±12.73 / 270.78 ms β”‚ no change β”‚
   β”‚ QQuery 20 β”‚ 269.04 / 284.91 Β±12.79 / 300.23 ms β”‚              276.34 / 289.09 Β±7.57 / 298.14 ms β”‚ no change β”‚
   β”‚ QQuery 21 β”‚ 692.00 / 706.55 Β±11.39 / 726.55 ms β”‚              685.35 / 694.67 Β±6.43 / 704.61 ms β”‚ no change β”‚
   β”‚ QQuery 22 β”‚     64.23 / 66.01 Β±1.40 / 67.85 ms β”‚                 63.23 / 67.70 Β±4.90 / 77.06 ms β”‚ no change β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
   ┃ Benchmark Summary                                             ┃           ┃
   ┑━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
   β”‚ Total Time (HEAD)                                             β”‚ 6527.90ms β”‚
   β”‚ Total Time (smj-derive-sort-options-and-target-partition-1)   β”‚ 6516.58ms β”‚
   β”‚ Average Time (HEAD)                                           β”‚  296.72ms β”‚
   β”‚ Average Time (smj-derive-sort-options-and-target-partition-1) β”‚  296.21ms β”‚
   β”‚ Queries Faster                                                β”‚         0 β”‚
   β”‚ Queries Slower                                                β”‚         0 β”‚
   β”‚ Queries with No Change                                        β”‚        22 β”‚
   β”‚ Queries with Failure                                          β”‚         0 β”‚
   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
   ```
   
   </p>
   </details>
   
   <details><summary>Resource Usage</summary>
   
   **tpch10 β€” base (merge-base)**
   | Metric | Value |
   |--------|-------|
   | Wall time | 35.0s |
   | Peak memory | 2.9 GiB |
   | Avg memory | 1.2 GiB |
   | CPU user | 330.7s |
   | CPU sys | 19.8s |
   | Peak spill | 0 B |
   
   **tpch10 β€” branch**
   | Metric | Value |
   |--------|-------|
   | Wall time | 35.0s |
   | Peak memory | 3.5 GiB |
   | Avg memory | 1.3 GiB |
   | CPU user | 329.9s |
   | CPU sys | 19.4s |
   | Peak spill | 0 B |
   
   </details>
   
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub)" <gi...@apache.org>.
bert-beyondloops commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6211971420

   @quwin Thank you for the detailed benchmark analysis!
   
   My main goal was to optimize single-target-partition SortMergeJoin plans where both inputs already share the same sort order on the join key. Previously, this case wasn’t handled because the sort options didn’t match the hardcoded ones.
   
   The intended use case is joining large numbers of rows from both inputs in a streaming fashion, without having to sort them first.
   
   I’m glad to see that other use cases, such as multi-target partitions, benefit from this PR as well.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "quwin (via GitHub)" <gi...@apache.org>.
quwin commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4267375061


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+
+statement ok
+DROP TABLE l;
+
+statement ok
+DROP TABLE r;
+
+statement ok
+DROP TABLE r_asc;
+
+statement ok
+reset datafusion.execution.target_partitions;

Review Comment:
   Could we restore the test runner’s initial value of 4 explicitly here and in `sort_merge_join.slt`?
   `RESET` restores the machine default instead. After correcting the snapshots, both files fail with target_partitions: 4 -> 8 on my machine.
   ```suggestion
   SET datafusion.execution.target_partitions = 4;
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,105 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right input is sorted differently: the join keeps the default options and sorts the inputs
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)ProjectionExec: expr=[k1@0 as k1, k2@1 as k2, v@2 as v, w@5 as w]
+02)--SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)]
+03)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
+05)----SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]

Review Comment:
   This fallback case has the same projection mismatch.
   ```suggestion
   01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
   02)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   04)--SortExec: expr=[k1@0 ASC, k2@1 ASC], preserve_partitioning=[false]
   05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 ASC, k2@1 ASC]
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "comphead (via GitHub) via github" <gi...@datafusion.apache.org>.
comphead commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6858530257

   run benchmark tpch tpcds


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6858580017

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6858530257)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6858530257-4957-6w79s 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "adriangbot (via GitHub) via github" <gi...@datafusion.apache.org>.
adriangbot commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6859739053

   πŸ€– Benchmark running (GKE) | [trigger](https://github.com/apache/datafusion/pull/26261#issuecomment-6859664529)
   **Instance:** `c4a-highmem-16` (12 vCPU / 65 GiB) | `Linux bench-c6859664529-4974-nrdjj 6.12.105+ #1 SMP Sat Sep 19 07:55:25 UTC 2026 aarch64 GNU/Linux`
   <details><summary>CPU Details (lscpu)</summary>
   
   ```
   Architecture:                            aarch64
   CPU op-mode(s):                          64-bit
   Byte Order:                              Little Endian
   CPU(s):                                  16
   On-line CPU(s) list:                     0-15
   Vendor ID:                               ARM
   Model name:                              Neoverse-V2
   Model:                                   1
   Thread(s) per core:                      1
   Core(s) per cluster:                     16
   Socket(s):                               -
   Cluster(s):                              1
   Stepping:                                r0p1
   BogoMIPS:                                2000.00
   Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
   L1d cache:                               1 MiB (16 instances)
   L1i cache:                               1 MiB (16 instances)
   L2 cache:                                32 MiB (16 instances)
   L3 cache:                                80 MiB (1 instance)
   NUMA node(s):                            1
   NUMA node0 CPU(s):                       0-15
   Vulnerability Gather data sampling:      Not affected
   Vulnerability Indirect target selection: Not affected
   Vulnerability Itlb multihit:             Not affected
   Vulnerability L1tf:                      Not affected
   Vulnerability Mds:                       Not affected
   Vulnerability Meltdown:                  Not affected
   Vulnerability Mmio stale data:           Not affected
   Vulnerability Reg file data sampling:    Not affected
   Vulnerability Retbleed:                  Not affected
   Vulnerability Spec rstack overflow:      Not affected
   Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
   Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
   Vulnerability Spectre v2:                Mitigation; CSV2, BHB
   Vulnerability Srbds:                     Not affected
   Vulnerability Tsa:                       Not affected
   Vulnerability Tsx async abort:           Not affected
   Vulnerability Vmscape:                   Not affected
   ```
   
   </details>
   
   Comparing smj-derive-sort-options-and-target-partition-1 (8255cfc4420e13782eff575ceeaccabf4580e031) to 3f837ad (merge-base) [diff](https://github.com/apache/datafusion/compare/3f837ad0db6f0a9e80bdff0864db5b6ad980ce1a..8255cfc4420e13782eff575ceeaccabf4580e031)
   
   <details><summary>Run configuration</summary>
   
   ```yaml
   run benchmark tpcds
   env:
     PREFER_HASH_JOIN: "false"
   ```
   
   </details>
   
   Results will be posted here when complete
   
   ---
   [File an issue](https://github.com/adriangb/datafusion-benchmarking/issues) against this benchmark runner


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "comphead (via GitHub) via github" <gi...@datafusion.apache.org>.
comphead commented on code in PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#discussion_r4566541459


##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)

Review Comment:
   The fixtures have no NULL keys and every query is an inner join, so the new `NULLS LAST` and `DESC` paths are only checked on matched non-null rows. The SMJ unit tests use non-default options only as `DESC NULLS LAST` and never for an outer join, and the join fuzz uses the default options. Could we add a NULL key row to both files and one `FULL JOIN` query? I ran a local comparison of SMJ against hash join for all join types, with `ASC NULLS LAST`, `DESC NULLS FIRST` and `DESC NULLS LAST` inputs and NULL keys, and found no mismatch. So this is about coverage, not a bug.



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -1902,14 +1902,18 @@ impl DefaultPhysicalPlanner {
                     // Use ordinary sort-merge join when hash join is not preferred.
                     // Null-aware joins not handled above fall back to CollectLeft
                     // HashJoin because their semantics are not supported by sort-merge join.
-                    let join_on_len = join_on.len();
+                    // The join keys are an unordered set of equalities, so they may be
+                    // reshuffled (together with their sort options) to match the input
+                    // orderings and avoid sorting.

Review Comment:
   `sort_merge_join_keys` already documents this, so the call-site comment can go.
   ```suggestion
   ```



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right table declares a different ordering than the file holds, so only the left ordering is adopted
+# and the right input is sorted
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET

Review Comment:
   `r_asc` declares an ordering that the file does not have, and the test works only because the right side is sorted again. A right file written without `ORDER BY` (so no `sorting_columns` to infer an ordering from) would cover the one-sided case without the false declaration, and would let us check the result too.



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3728,6 +3732,117 @@ enum PlanAsyncExpr {
     Async(AsyncMapper, PlannedExprResult),
 }
 
+/// Chooses the join key order and per-key [`SortOptions`] of a [`SortMergeJoinExec`], so that as
+/// few inputs as possible need a `SortExec`.
+///
+/// The join keys are an unordered set of equalities, so they can be permuted freely. Candidates
+/// are derived from each known output ordering of each input: its entries that (after normalizing
+/// through the equivalence group) are join keys become the leading keys, in that order and with
+/// those options, and the remaining keys follow with the default options. Each candidate is
+/// scored by how many inputs `ordering_satisfy` confirms are already sorted on it, so constants,
+/// equal columns and any key order are handled. The default (original key order, default options)
+/// wins ties, so plans that already match are unchanged.
+///
+/// Cost: nothing is done if neither input has an ordering; otherwise one candidate per known
+/// ordering, each verified with two `ordering_satisfy` calls.
+fn sort_merge_join_keys(

Review Comment:
   Could this reuse `EquivalenceProperties::find_longest_permutation`? Windows and aggregates already use it for the same problem. It returns the longest permutation of a key list that an input is sorted on, with the options taken from the ordering, and it handles equivalent columns and constants. That removes the per-ordering loop, the manual `normalize_expr`, the `used` mask and the two `left_side`/`driven_by_left` selections, and it leaves at most two candidates instead of one per ordering:
   
   ```rust
   /// Chooses the join key order and per-key [`SortOptions`] of a [`SortMergeJoinExec`].
   ///
   /// The keys are an unordered set of equalities, so each input proposes the longest
   /// permutation of its keys that it is already sorted on. A proposal replaces the default
   /// (original order, `ASC NULLS FIRST`) only if strictly more inputs are sorted on it.
   fn sort_merge_join_keys(
       left: &Arc<dyn ExecutionPlan>,
       right: &Arc<dyn ExecutionPlan>,
       on: &join_utils::JoinOn,
   ) -> Result<(join_utils::JoinOn, Vec<SortOptions>)> {
       let (left_keys, right_keys): (Vec<_>, Vec<_>) = on.iter().cloned().unzip();
       let inputs = [
           (left.equivalence_properties(), left_keys),
           (right.equivalence_properties(), right_keys),
       ];
       // Number of inputs already sorted on the keys at `order` with `options`.
       let sorted_inputs = |order: &[usize], options: &[SortOptions]| {
           inputs.iter().try_fold(0, |sorted, (eq, keys)| {
               let sort_exprs = order.iter().zip(options).map(|(&i, &options)| {
                   PhysicalSortExpr::new(Arc::clone(&keys[i]), options)
               });
               Ok::<_, DataFusionError>(
                   sorted + usize::from(eq.ordering_satisfy(sort_exprs)?),
               )
           })
       };
   
       let mut best_order: Vec<usize> = (0..on.len()).collect();
       let mut best_options = vec![SortOptions::default(); on.len()];
       let mut best_sorted = sorted_inputs(&best_order, &best_options)?;
       for (eq, keys) in &inputs {
           if best_sorted == inputs.len() || eq.oeq_class().is_empty() {
               continue;
           }
           let (sort_exprs, mut order) = eq.find_longest_permutation(keys)?;
           if order.is_empty() {
               continue;
           }
           let mut options: Vec<_> = sort_exprs.iter().map(|e| e.options).collect();
           // Keys this input is not sorted on follow in their original order.
           let unsorted: Vec<_> = (0..on.len()).filter(|i| !order.contains(i)).collect();
           options.resize(on.len(), SortOptions::default());
           order.extend(unsorted);
           let sorted = sorted_inputs(&order, &options)?;
           if sorted > best_sorted {
               (best_order, best_options, best_sorted) = (order, options, sorted);
           }
       }
       let on = best_order.into_iter().map(|i| on[i].clone()).collect();
       Ok((on, best_options))
   }
   ```
   
   I tried this on top of the branch. Both slt files pass unchanged except the constant `k2` case, which now plans `on=[(k2@1, k2@1), (k1@0, k1@0)]`, still without a `SortExec`.



##########
datafusion/core/src/physical_planner.rs:
##########
@@ -3728,6 +3732,117 @@ enum PlanAsyncExpr {
     Async(AsyncMapper, PlannedExprResult),
 }
 
+/// Chooses the join key order and per-key [`SortOptions`] of a [`SortMergeJoinExec`], so that as
+/// few inputs as possible need a `SortExec`.
+///
+/// The join keys are an unordered set of equalities, so they can be permuted freely. Candidates
+/// are derived from each known output ordering of each input: its entries that (after normalizing
+/// through the equivalence group) are join keys become the leading keys, in that order and with
+/// those options, and the remaining keys follow with the default options. Each candidate is
+/// scored by how many inputs `ordering_satisfy` confirms are already sorted on it, so constants,
+/// equal columns and any key order are handled. The default (original key order, default options)
+/// wins ties, so plans that already match are unchanged.
+///
+/// Cost: nothing is done if neither input has an ordering; otherwise one candidate per known
+/// ordering, each verified with two `ordering_satisfy` calls.
+fn sort_merge_join_keys(
+    left: &Arc<dyn ExecutionPlan>,
+    right: &Arc<dyn ExecutionPlan>,
+    on: &join_utils::JoinOn,
+) -> Result<(join_utils::JoinOn, Vec<SortOptions>)> {
+    let left_eq = left.equivalence_properties();
+    let right_eq = right.equivalence_properties();
+
+    // The baseline candidate is the join keys in the order they were provided, all with the
+    // default sort options. Other candidates replace it only if they strictly reduce the number
+    // of inputs that still need sorting.
+    let mut best = (on.clone(), vec![SortOptions::default(); on.len()]);
+    if left_eq.oeq_class().is_empty() && right_eq.oeq_class().is_empty() {
+        return Ok(best);
+    }
+
+    // Scores a candidate (join keys and their sort options) by the number of inputs that are
+    // already sorted on it and therefore need no `SortExec`: 0 (both inputs need a sort),
+    // 1 (one input is already sorted) or 2 (neither input needs a sort).
+    let score = |candidate: &(join_utils::JoinOn, Vec<SortOptions>)| -> Result<usize> {
+        let satisfied = |eq: &datafusion_physical_expr::EquivalenceProperties,
+                         left_side: bool| {
+            eq.ordering_satisfy(candidate.0.iter().zip(&candidate.1).map(
+                |((l, r), options)| {
+                    PhysicalSortExpr::new(
+                        Arc::clone(if left_side { l } else { r }),
+                        *options,
+                    )
+                },
+            ))
+        };
+        Ok(usize::from(satisfied(left_eq, true)?)
+            + usize::from(satisfied(right_eq, false)?))
+    };
+    let mut best_score = score(&best)?;
+
+    // Derive candidates from the orderings of both inputs, left first and then right. The left
+    // orderings may not lead to any improvement (e.g. the left input is unsorted or sorted on
+    // other columns), but the right input's orderings can still give a better score, so a
+    // candidate that only the right input is already sorted on is still found. Stop early once
+    // a candidate scores 2, as no input needs sorting anymore.
+    for (eq, driven_by_left) in [(left_eq, true), (right_eq, false)] {
+        if best_score == 2 {
+            break;
+        }
+        // The join key of the driving side, normalized through its equivalence group, so that
+        // equal columns match the entries of the ordering below.
+        let keys: Vec<_> = on
+            .iter()
+            .map(|(l, r)| {
+                eq.eq_group().normalize_expr(Arc::clone(if driven_by_left {
+                    l
+                } else {
+                    r
+                }))
+            })
+            .collect();
+        for ordering in eq.oeq_class().iter() {
+            // `used[j]` is true once join key `j` was found in this lex ordering and moved to the
+            // front of the candidate. Each key is used at most once, so an expression that
+            // appears twice in the ordering only matches one key.
+            let mut used = vec![false; on.len()];
+            let mut candidate =
+                (Vec::with_capacity(on.len()), Vec::with_capacity(on.len()));
+            // Walk the ordering in order: every sort expression that is a (not yet used) join
+            // key adds that key to the candidate, with the sort options of the ordering.
+            // Sort expressions that are not join keys are skipped; `ordering_satisfy` in
+            // `score` decides whether the skipped ones (e.g. constants) are harmless.
+            for sort_expr in ordering.iter() {
+                let expr = eq.eq_group().normalize_expr(Arc::clone(&sort_expr.expr));
+                if let Some(j) = (0..on.len()).find(|&j| !used[j] && keys[j].eq(&expr)) {
+                    used[j] = true;
+                    candidate.0.push(on[j].clone());
+                    candidate.1.push(sort_expr.options);
+                }
+            }
+            if candidate.0.is_empty() {
+                continue;
+            }
+            // The join keys that were not found in the ordering follow, in their provided order
+            // and with the default sort options.
+            for (key, _) in on.iter().zip(&used).filter(|(_, used)| !**used) {
+                candidate.0.push(key.clone());
+                candidate.1.push(SortOptions::default());
+            }
+            let candidate_score = score(&candidate)?;
+            if candidate_score > best_score {

Review Comment:
   When the inputs are sorted on only some of the keys, every candidate scores 0, so the default wins and the sorted prefix is dropped. For example, `l.k = r.k AND l.v % 2 = r.w % 2` over inputs sorted `k DESC NULLS LAST` plans `SortExec: expr=[k@0 ASC, l.v % Int64(2)@2 ASC]` on both sides. For bounded inputs that costs about the same. For unbounded inputs, `replace_with_partial_sort` can only use a `PartialSortExec` if the prefix is kept. Would breaking ties by the length of the sorted prefix be worth it here, or as a follow-up?



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right table declares a different ordering than the file holds, so only the left ordering is adopted
+# and the right input is sorted
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--SortExec: expr=[k1@0 DESC NULLS LAST, k2@1 ASC], preserve_partitioning=[false]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 DESC NULLS LAST, k2@1 ASC], reverse_row_groups=true
+
+# the join keys are listed in a different order than the input ordering: the keys are reordered
+# to follow the ordering, so no SortExec is needed
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k2 = r.k2, l.k1 = r.k1
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+2 b 20 200
+3 c 30 300
+
+# only the leading ordering column is a join key: its options are adopted and no SortExec is needed

Review Comment:
   This takes the same path as the first query, with one key instead of two. Could it be replaced by the NULL key / `FULL JOIN` case?



##########
datafusion/sqllogictest/test_files/sort_merge_join_sort_options.slt:
##########
@@ -0,0 +1,158 @@
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership
+# SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements
+
+##########
+## The sort options of a SortMergeJoin follow the ordering both inputs declare on the join keys,
+## so no SortExec is needed on top of inputs that are already sorted that way.
+##########
+
+statement ok
+set datafusion.optimizer.prefer_hash_join = false;
+
+statement ok
+set datafusion.execution.target_partitions = 2;
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 30), (2, 'b', 20), (1, 'a', 10)) AS t(k1, k2, v) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+COPY (SELECT * FROM (VALUES (3, 'c', 300), (2, 'b', 200), (4, 'd', 400)) AS t(k1, k2, w) ORDER BY k1 DESC NULLS LAST, k2 ASC NULLS FIRST)
+TO 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+# both inputs sorted `k1 DESC NULLS LAST, k2 ASC NULLS FIRST`: the join adopts these per key options
+statement ok
+CREATE EXTERNAL TABLE l (k1 INT, k2 VARCHAR, v INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/left.parquet';
+
+statement ok
+CREATE EXTERNAL TABLE r (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+2 b 20 200
+3 c 30 300
+
+# the right table declares a different ordering than the file holds, so only the left ordering is adopted
+# and the right input is sorted
+statement ok
+CREATE EXTERNAL TABLE r_asc (k1 INT, k2 VARCHAR, w INT) STORED AS PARQUET
+WITH ORDER (k1 ASC NULLS LAST, k2 ASC NULLS FIRST)
+LOCATION 'test_files/scratch/sort_merge_join_sort_options/right.parquet';
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r_asc r ON l.k1 = r.k1 AND l.k2 = r.k2
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k1 = r.k1, l.k2 = r.k2
+03)----TableScan: l projection=[k1, k2, v]
+04)----SubqueryAlias: r
+05)------TableScan: r_asc projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--SortExec: expr=[k1@0 DESC NULLS LAST, k2@1 ASC], preserve_partitioning=[false]
+04)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], file_type=parquet, sort_order_for_reorder=[k1@0 DESC NULLS LAST, k2@1 ASC], reverse_row_groups=true
+
+# the join keys are listed in a different order than the input ordering: the keys are reordered
+# to follow the ordering, so no SortExec is needed
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k2 = r.k2, l.k1 = r.k1
+03)----TableScan: l projection=[k1, k2, v]
+04)----TableScan: r projection=[k1, k2, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1
+----
+2 b 20 200
+3 c 30 300
+
+# only the leading ordering column is a join key: its options are adopted and no SortExec is needed
+query TT
+EXPLAIN SELECT l.k1, v, w FROM l JOIN r ON l.k1 = r.k1
+----
+logical_plan
+01)Projection: l.k1, l.v, r.w
+02)--Inner Join: l.k1 = r.k1
+03)----TableScan: l projection=[k1, v]
+04)----TableScan: r projection=[k1, w]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0)], projection=[k1@0, v@1, w@3]
+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, v], output_ordering=[k1@0 DESC NULLS LAST], file_type=parquet
+03)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, w], output_ordering=[k1@0 DESC NULLS LAST], file_type=parquet
+
+query III rowsort
+SELECT l.k1, v, w FROM l JOIN r ON l.k1 = r.k1
+----
+2 20 200
+3 30 300
+
+# k2 is constant after the filter, so it does not need to be in the ordering: the join adopts the
+# options of the k1 ordering and no SortExec is needed (a single partition avoids a repartition)
+statement ok
+set datafusion.execution.target_partitions = 1;
+
+query TT
+EXPLAIN SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1 WHERE l.k2 = 'b'
+----
+logical_plan
+01)Projection: l.k1, l.k2, l.v, r.w
+02)--Inner Join: l.k2 = r.k2, l.k1 = r.k1
+03)----Filter: l.k2 = Utf8View("b")
+04)------TableScan: l projection=[k1, k2, v], partial_filters=[l.k2 = Utf8View("b")]
+05)----Filter: r.k2 = Utf8View("b")
+06)------TableScan: r projection=[k1, k2, w], partial_filters=[r.k2 = Utf8View("b")]
+physical_plan
+01)SortMergeJoinExec: join_type=Inner, on=[(k1@0, k1@0), (k2@1, k2@1)], projection=[k1@0, k2@1, v@2, w@5]
+02)--FilterExec: k2@1 = b
+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/left.parquet]]}, projection=[k1, k2, v], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet, predicate=k2@1 = b, pruning_predicate=k2_null_count@2 != row_count@3 AND k2_min@0 <= b AND b <= k2_max@1, required_guarantees=[k2 in (b)]
+04)--FilterExec: k2@1 = b
+05)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/sort_merge_join_sort_options/right.parquet]]}, projection=[k1, k2, w], output_ordering=[k1@0 DESC NULLS LAST, k2@1 ASC], file_type=parquet, predicate=k2@1 = b, pruning_predicate=k2_null_count@2 != row_count@3 AND k2_min@0 <= b AND b <= k2_max@1, required_guarantees=[k2 in (b)]
+
+query ITII rowsort
+SELECT l.k1, l.k2, v, w FROM l JOIN r ON l.k2 = r.k2 AND l.k1 = r.k1 WHERE l.k2 = 'b'
+----
+2 b 20 200
+
+statement ok
+set datafusion.execution.target_partitions = 2;

Review Comment:
   This setting is overwritten before any other query runs.
   ```suggestion
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] feat/perf: derive SortMergeJoin sort options from input ordering and allow SMJ for a single partition [datafusion]

Posted by "bert-beyondloops (via GitHub) via github" <gi...@datafusion.apache.org>.
bert-beyondloops commented on PR #26261:
URL: https://github.com/apache/datafusion/pull/26261#issuecomment-6913823182

   @comphead  Thanks so much for the thorough review, I really appreciate it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org