You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "adriangb (via GitHub)" <gi...@apache.org> on 2026/06/04 12:50:23 UTC
[PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]
adriangb opened a new pull request, #22760:
URL: https://github.com/apache/datafusion/pull/22760
## Which issue does this PR close?
- Relates to https://github.com/apache/datafusion/issues/11900
## Rationale for this change
This splits the test and benchmark scaffolding out of #21621 so the
`PushDownTopKThroughJoin` optimizer rule itself can be reviewed in
isolation, with a small, focused diff.
The benchmark and SLT files here do not depend on the rule. They are
committed first so that:
1. The benchmark can measure the rule's effect against a baseline that
does not register it.
2. The follow-up rule PR's diff shows exactly which plans change, since
the EXPLAIN plans here capture the current (pre-rule) behavior.
## What changes are included in this PR?
- A `push_down_topk` benchmark (`dfbench push-down-topk`) that runs
`ORDER BY <cols> LIMIT N` queries over outer joins against TPC-H
`customer`/`orders`/`nation`, plus its query files under
`benchmarks/queries/push_down_topk/`.
- `push_down_topk_through_join.slt` covering the scenarios the rule
handles: preserved-side sort keys, ineligible join types
(inner/full/semi/anti), `ON`-clause filters, projection and
`SubqueryAlias` resolution, existing child sorts, ties, multi-level
joins, `OFFSET`, and volatile expressions.
The EXPLAIN plans assert current behavior (TopK not yet pushed through
the join). The follow-up PR that adds the rule updates those plans in
place; the query-result checks hold regardless of whether the rule is
enabled.
The new optimizer rule, the `push_down_limit.rs` changes, and the
`optimizer_rule_reference.md` update from #21621 are intentionally left
for the follow-up PR.
## Are these changes tested?
Yes — this PR is the tests. `push_down_topk_through_join.slt` passes
against `main`, and the benchmark binary compiles and runs.
## Are there any user-facing changes?
No. No API changes; only new benchmark and test files plus benchmark CLI
wiring.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb commented on code in PR #22760:
URL: https://github.com/apache/datafusion/pull/22760#discussion_r3356696533
##########
benchmarks/src/push_down_topk.rs:
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark for `push_down_topk_through_join`.
Review Comment:
I initially copied over verbatim but I will rework them into the new framework.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]
Posted by "adriangb (via GitHub)" <gi...@apache.org>.
adriangb merged PR #22760:
URL: https://github.com/apache/datafusion/pull/22760
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] test: benchmarks and SLT tests for push-down TopK through join [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #22760:
URL: https://github.com/apache/datafusion/pull/22760#discussion_r3356659203
##########
benchmarks/src/push_down_topk.rs:
##########
@@ -0,0 +1,264 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Benchmark for `push_down_topk_through_join`.
Review Comment:
So we really need a whole special executor for this benchmark? Can we use the new benchmark runner stuff that @Omega359 is working on?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org