You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kristofferjoha (via GitHub)" <gi...@apache.org> on 2026/10/22 14:58:48 UTC
[PR] feat: support partitioned execution for ASOF joins [datafusion]
Kristofferjoha opened a new pull request, #26592:
URL: https://github.com/apache/datafusion/pull/26592
## Which issue does this PR close?
Closes #26478.
## Rationale for this change
ASOF joins currently collect the complete right input into a non-spillable reservation. Queries fail when that input exceeds the memory pool, even when upstream sorts can spill. Each left partition also scans the collected right input independently.
## What changes are included in this PR?
- Add `AsOfJoinMode::Partitioned`, requiring both inputs to be co-partitioned on equality keys and sorted within each partition. Each left partition streams its corresponding right partition.
- Account for retained streaming buffers in the memory pool, including candidate matches and pending output rows, without counting shared buffers twice.
- Preserve the mode through projection, child replacement, and protobuf/JSON serialization. Older serialized plans default to broadcast.
- Add the opt-in `datafusion.optimizer.repartition_asof_joins` option, defaulting to `false`. Joins without equality keys continue to use broadcast execution.
The join itself does not spill: upstream sorts handle spilling, and the join still needs memory for its active working set. Skewed keys can reduce parallelism.
Existing ASOF SQL benchmarks compared with local `main` (`6c1727fcd`), using four partitions and 12 measured samples per case after warm-up:
| Case | Main median | Partitioned median |
| --- | ---: | ---: |
| Q02: balanced equality keys | 183.55 ms | 108.14 ms |
| Q03: wide payload | 63.86 ms | 46.93 ms |
| Q06: skewed equality keys | 175.70 ms | 335.76 ms |
| Q11: large right input | 2687.45 ms | 1045.61 ms |
These timings used no memory limit. Broadcast timings on this branch showed no substantial regression. The skew result supports keeping partitioned execution opt-in.
## What is the testing strategy for this PR?
- SQL logic tests cover all four match operators, composite and computed equality keys, NULLs, empty inputs, ties, and small batches.
- SQL integration tests check strategy selection, input distribution and ordering, plan display, and query results.
- A memory-limit integration test verifies that partitioned execution succeeds with spilled sort output under a budget that rejects broadcast collection.
- Unit tests cover partition selection, mode preservation, buffer accounting, bounded streaming, and reservation cleanup on completion, cancellation, and error.
- Protobuf and JSON tests cover mode round trips, compatibility with missing mode fields, and invalid serialized modes.
Local validation: the required extended workspace suite passed (12,814 Rust tests and all 538 SQL test files). After consolidating redundant unit tests, all 21 ASOF unit tests passed. Formatting, all-targets/all-features Clippy with warnings denied, generated documentation, HTML/API documentation, and dependency structure checks passed.
The full lint script's security audit reports RUSTSEC-2026-0382 (`rustls-webpki`) and RUSTSEC-2026-0383 (`webpki-root-certs`). Both findings were reproduced using the unchanged lockfile on local `main`.
## Are there any user-facing changes?
Yes. Enable partitioned execution for ASOF joins with equality keys using:
```sql
SET datafusion.optimizer.repartition_asof_joins = true;
```
The default execution strategy remains broadcast. The SQL guide and generated configuration reference document the option.
This contribution used AI assistance. Keeping this PR in draft while I finish reviewing and walking through the implementation before requesting maintainer review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support partitioned execution for ASOF joins [datafusion]
Posted by "Kristofferjoha (via GitHub)" <gi...@apache.org>.
Kristofferjoha commented on PR #26592:
URL: https://github.com/apache/datafusion/pull/26592#issuecomment-6349724421
> Thank you for opening this pull request!
>
> Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
>
> Details
> Thank you for opening this pull request!
>
> Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
>
> Details
This is my first contribution, so i am unsure as to how to handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support partitioned execution for ASOF joins [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26592:
URL: https://github.com/apache/datafusion/pull/26592#issuecomment-6349168724
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.1.0 (current)
Built [ 65.791s] (current)
Parsing datafusion v55.1.0 (current)
Parsed [ 0.032s] (current)
Building datafusion v55.1.0 (baseline)
Built [ 59.527s] (baseline)
Parsing datafusion v55.1.0 (baseline)
Parsed [ 0.032s] (baseline)
Checking datafusion v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.811s] 229 checks: 229 pass, 31 skip
Summary no semver update required
Finished [ 127.835s] datafusion
Building datafusion-common v55.1.0 (current)
Built [ 34.976s] (current)
Parsing datafusion-common v55.1.0 (current)
Parsed [ 0.070s] (current)
Building datafusion-common v55.1.0 (baseline)
Built [ 35.014s] (baseline)
Parsing datafusion-common v55.1.0 (baseline)
Parsed [ 0.069s] (baseline)
Checking datafusion-common v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.917s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field OptimizerOptions.repartition_asof_joins in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:1620
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 72.184s] datafusion-common
Building datafusion-physical-plan v55.1.0 (current)
Built [ 40.956s] (current)
Parsing datafusion-physical-plan v55.1.0 (current)
Parsed [ 0.211s] (current)
Building datafusion-physical-plan v55.1.0 (baseline)
Built [ 41.040s] (baseline)
Parsing datafusion-physical-plan v55.1.0 (baseline)
Parsed [ 0.208s] (baseline)
Checking datafusion-physical-plan v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.855s] 229 checks: 229 pass, 31 skip
Summary no semver update required
Finished [ 84.675s] datafusion-physical-plan
Building datafusion-proto v55.1.0 (current)
Built [ 55.563s] (current)
Parsing datafusion-proto v55.1.0 (current)
Parsed [ 0.012s] (current)
Building datafusion-proto v55.1.0 (baseline)
Built [ 55.980s] (baseline)
Parsing datafusion-proto v55.1.0 (baseline)
Parsed [ 0.013s] (baseline)
Checking datafusion-proto v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 0.143s] 229 checks: 229 pass, 31 skip
Summary no semver update required
Finished [ 113.132s] datafusion-proto
Building datafusion-proto-models v55.1.0 (current)
Built [ 26.899s] (current)
Parsing datafusion-proto-models v55.1.0 (current)
Parsed [ 0.152s] (current)
Building datafusion-proto-models v55.1.0 (baseline)
Built [ 26.697s] (baseline)
Parsing datafusion-proto-models v55.1.0 (baseline)
Parsed [ 0.151s] (baseline)
Checking datafusion-proto-models v55.1.0 -> v55.1.0 (no change; assume patch)
Checked [ 2.491s] 229 checks: 228 pass, 1 fail, 0 warn, 31 skip
--- failure constructible_struct_adds_field: struct exhaustively constructible through public API adds field ---
Description:
A pub struct that could be exhaustively constructed with a literal using only public API has a new pub field, breaking existing exhaustive literals.
ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.51.0/src/lints/constructible_struct_adds_field.ron
Failed in:
field AsOfJoinExecNode.mode in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2801
field AsOfJoinExecNode.mode in /home/runner/work/datafusion/datafusion/datafusion/proto-models/src/generated/prost.rs:2801
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 57.715s] datafusion-proto-models
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org