You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2026/08/30 14:18:29 UTC
[PR] perf: push left filters through ASOF joins [datafusion]
Xuanwo opened a new pull request, #24801:
URL: https://github.com/apache/datafusion/pull/24801
## Which issue does this PR close?
- Part of #318.
- Umbrella PR: #23738.
- Depends on #23829.
## Rationale for this change
An ASOF join emits each left row exactly once and does not change left-side
values. A deterministic predicate that references only the left input can
therefore run before matching. Right-only and mixed predicates must remain
above the join because unmatched right fields are NULL-padded, and volatile
predicates must keep their original evaluation point.
Pushing the safe subset reduces the number of rows entering the ASOF join
without changing query results.
## What changes are included in this PR?
- Push deterministic left-only conjuncts below logical ASOF joins.
- Keep right-only, mixed, and volatile predicates above the join.
- Document the cardinality and NULL-padding assumptions at the optimizer rule.
- Add focused optimizer snapshots for all four predicate categories.
## Are these changes tested?
- `cargo fmt --all`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test --profile ci -p datafusion-optimizer asof_join_ --all-features`
- The extended workspace test command from the contributor guide
## Are there any user-facing changes?
No result semantics or public APIs change. Eligible ASOF joins may execute with
fewer left-side rows after logical optimization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#discussion_r4063810722
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -256,6 +256,64 @@ physical_plan
04)--SortExec: expr=[grp@0 ASC, ts@1 ASC], preserve_partitioning=[false]
05)----DataSourceExec: partitions=1, partition_sizes=[1]
+# Deterministic predicates that reference only the left input can run before
Review Comment:
Makes a lot sense!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #24801:
URL: https://github.com/apache/datafusion/pull/24801
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#discussion_r4063284771
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1141,6 +1141,32 @@ impl OptimizerRule for PushDownFilter {
result.map_data(|plan| Ok(with_filters(keep_predicates, plan)))
}
LogicalPlan::Join(join) => push_down_join(join, Some(filter.predicate)),
+ LogicalPlan::AsOfJoin(mut join) => {
Review Comment:
Follow-up is fine: a pushed left-only predicate on an equality key (`ON l.grp = r.grp`, `WHERE l.grp = 'A'`) can be mirrored to the right input as `r.grp = 'A'`. Right rows in other groups can never match a surviving left row, so null padding is unaffected. It is the same idea as `infer_join_predicates` for regular joins. Q08 has no key, so a keyed benchmark would be needed to show the gain.
I reproduced Q08 on Apple silicon (release build, interleaved runs, 7 iterations): 121.5–127.3 ms → 18.0 ms median, about 6.9x.
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -256,6 +256,64 @@ physical_plan
04)--SortExec: expr=[grp@0 ASC, ts@1 ASC], preserve_partitioning=[false]
05)----DataSourceExec: partitions=1, partition_sizes=[1]
+# Deterministic predicates that reference only the left input can run before
Review Comment:
Both new tests are `EXPLAIN`-only. Nothing executes a query with the left filter below the join next to a null-padded right predicate, so a bug in which conjunct lands where would only change plan text. Please add result queries. I ran these on this branch and they pass:
```sql
query IT
SELECT l.id, r.val
FROM asof_left l
ASOF JOIN asof_right r
MATCH_CONDITION (l.ts >= r.ts)
ON l.grp = r.grp
WHERE l.id > 1 AND r.val <> 'a2' AND l.id <> length(r.val)
ORDER BY l.id;
----
3 a6
4 b1
5 b6
query IT
SELECT l.id, r.val
FROM asof_left l
ASOF JOIN asof_right r
MATCH_CONDITION (l.ts >= r.ts)
ON l.grp = r.grp
WHERE l.grp IS NULL OR l.ts IS NULL OR l.id = 1
ORDER BY l.id;
----
1 NULL
6 NULL
7 NULL
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#issuecomment-5750743716
Hi @2010YOUY01 and @jayzhan211, this ASOF filter pushdown follow-up is ready for review. Would you mind taking a look when you have time? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#issuecomment-5469313105
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.0.0 (current)
Built [ 57.369s] (current)
Parsing datafusion v55.0.0 (current)
Parsed [ 0.037s] (current)
Building datafusion v55.0.0 (baseline)
Built [ 56.759s] (baseline)
Parsing datafusion v55.0.0 (baseline)
Parsed [ 0.038s] (baseline)
Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.864s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 117.216s] datafusion
Building datafusion-expr v55.0.0 (current)
Built [ 28.357s] (current)
Parsing datafusion-expr v55.0.0 (current)
Parsed [ 0.081s] (current)
Building datafusion-expr v55.0.0 (baseline)
Built [ 28.744s] (baseline)
Parsing datafusion-expr v55.0.0 (baseline)
Parsed [ 0.082s] (baseline)
Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.928s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/enum_variant_added.ron
Failed in:
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 60.464s] datafusion-expr
Building datafusion-optimizer v55.0.0 (current)
Built [ 26.596s] (current)
Parsing datafusion-optimizer v55.0.0 (current)
Parsed [ 0.033s] (current)
Building datafusion-optimizer v55.0.0 (baseline)
Built [ 26.526s] (baseline)
Parsing datafusion-optimizer v55.0.0 (baseline)
Parsed [ 0.034s] (baseline)
Checking datafusion-optimizer v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.225s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 54.456s] datafusion-optimizer
Building datafusion-physical-plan v55.0.0 (current)
Built [ 37.425s] (current)
Parsing datafusion-physical-plan v55.0.0 (current)
Parsed [ 0.156s] (current)
Building datafusion-physical-plan v55.0.0 (baseline)
Built [ 37.128s] (baseline)
Parsing datafusion-physical-plan v55.0.0 (baseline)
Parsed [ 0.158s] (baseline)
Checking datafusion-physical-plan v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.058s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 77.527s] datafusion-physical-plan
Building datafusion-proto v55.0.0 (current)
Built [ 52.623s] (current)
Parsing datafusion-proto v55.0.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-proto v55.0.0 (baseline)
Built [ 52.982s] (baseline)
Parsing datafusion-proto v55.0.0 (baseline)
Parsed [ 0.019s] (baseline)
Checking datafusion-proto v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.167s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 107.219s] datafusion-proto
Building datafusion-sql v55.0.0 (current)
Built [ 41.434s] (current)
Parsing datafusion-sql v55.0.0 (current)
Parsed [ 0.033s] (current)
Building datafusion-sql v55.0.0 (baseline)
Built [ 41.590s] (baseline)
Parsing datafusion-sql v55.0.0 (baseline)
Parsed [ 0.035s] (baseline)
Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.330s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 84.896s] datafusion-sql
Building datafusion-substrait v55.0.0 (current)
Built [ 296.134s] (current)
Parsing datafusion-substrait v55.0.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-substrait v55.0.0 (baseline)
Built [ 296.511s] (baseline)
Parsing datafusion-substrait v55.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-substrait v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.331s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 595.250s] datafusion-substrait
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#discussion_r4063812458
##########
datafusion/optimizer/src/push_down_filter.rs:
##########
@@ -1141,6 +1141,32 @@ impl OptimizerRule for PushDownFilter {
result.map_data(|plan| Ok(with_filters(keep_predicates, plan)))
}
LogicalPlan::Join(join) => push_down_join(join, Some(filter.predicate)),
+ LogicalPlan::AsOfJoin(mut join) => {
Review Comment:
Nice, will create a follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] perf: push left filters through ASOF joins [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #24801:
URL: https://github.com/apache/datafusion/pull/24801#issuecomment-5469342848
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/24801?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `44.95614%` with `251 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 81.48%. Comparing base ([`c4910e0`](https://app.codecov.io/gh/apache/datafusion/commit/c4910e0764f3b02cc27d347bfc117f85a4edb46d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`3a5e377`](https://app.codecov.io/gh/apache/datafusion/commit/3a5e377e3886f5c42519db9f7352bcd9ff97566d?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 20 commits behind head on main.
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/24801?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/expr/src/logical\_plan/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | 50.25% | [81 Missing and 16 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 0.00% | [41 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/tree\_node.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Ftree_node.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vdHJlZV9ub2RlLnJz) | 9.30% | [38 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/optimizer/src/analyzer/type\_coercion.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fanalyzer%2Ftype_coercion.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL2FuYWx5emVyL3R5cGVfY29lcmNpb24ucnM=) | 0.00% | [28 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/builder.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fbuilder.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | 60.71% | [17 Missing and 5 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/optimizer/src/push\_down\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fpush_down_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3B1c2hfZG93bl9maWx0ZXIucnM=) | 84.41% | [5 Missing and 7 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/display.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fdisplay.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vZGlzcGxheS5ycw==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/optimizer/src/optimizer.rs](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Foptimizer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL29wdGltaXplci5ycw==) | 50.00% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/24801?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #24801 +/- ##
==========================================
+ Coverage 81.47% 81.48% +0.01%
==========================================
Files 1122 1123 +1
Lines 403629 406555 +2926
Branches 403629 406555 +2926
==========================================
+ Hits 328866 331301 +2435
- Misses 55510 55856 +346
- Partials 19253 19398 +145
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/24801?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org