You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/06/04 12:50:23 UTC

[PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]

adriangb opened a new pull request, #22760:
URL: https://github.com/apache/datafusion/pull/22760

   ## Which issue does this PR close?
   
   - Relates to https://github.com/apache/datafusion/issues/11900
   
   ## Rationale for this change
   
   This splits the test and benchmark scaffolding out of #21621 so the
   `PushDownTopKThroughJoin` optimizer rule itself can be reviewed in
   isolation, with a small, focused diff.
   
   The benchmark and SLT files here do not depend on the rule. They are
   committed first so that:
   
   1. The benchmark can measure the rule's effect against a baseline that
      does not register it.
   2. The follow-up rule PR's diff shows exactly which plans change, since
      the EXPLAIN plans here capture the current (pre-rule) behavior.
   
   ## What changes are included in this PR?
   
   - A `push_down_topk` benchmark (`dfbench push-down-topk`) that runs
     `ORDER BY <cols> LIMIT N` queries over outer joins against TPC-H
     `customer`/`orders`/`nation`, plus its query files under
     `benchmarks/queries/push_down_topk/`.
   - `push_down_topk_through_join.slt` covering the scenarios the rule
     handles: preserved-side sort keys, ineligible join types
     (inner/full/semi/anti), `ON`-clause filters, projection and
     `SubqueryAlias` resolution, existing child sorts, ties, multi-level
     joins, `OFFSET`, and volatile expressions.
   
   The EXPLAIN plans assert current behavior (TopK not yet pushed through
   the join). The follow-up PR that adds the rule updates those plans in
   place; the query-result checks hold regardless of whether the rule is
   enabled.
   
   The new optimizer rule, the `push_down_limit.rs` changes, and the
   `optimizer_rule_reference.md` update from #21621 are intentionally left
   for the follow-up PR.
   
   ## Are these changes tested?
   
   Yes — this PR is the tests. `push_down_topk_through_join.slt` passes
   against `main`, and the benchmark binary compiles and runs.
   
   ## Are there any user-facing changes?
   
   No. No API changes; only new benchmark and test files plus benchmark CLI
   wiring.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22760:
URL: https://github.com/apache/datafusion/pull/22760#discussion_r3356696533


##########
benchmarks/src/push_down_topk.rs:
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark for `push_down_topk_through_join`.

Review Comment:
   I initially copied over verbatim but I will rework them into the new framework.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]

Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #22760:
URL: https://github.com/apache/datafusion/pull/22760


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]

Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22760:
URL: https://github.com/apache/datafusion/pull/22760#discussion_r3356659203


##########
benchmarks/src/push_down_topk.rs:
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark for `push_down_topk_through_join`.

Review Comment:
   So we really need a whole special executor for this benchmark? Can we use the new benchmark runner stuff that @Omega359  is working on?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org