You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Xuanwo (via GitHub)" <gi...@apache.org> on 2026/07/23 05:45:48 UTC
[PR] feat: support ASOF JOIN SQL [datafusion]
Xuanwo opened a new pull request, #23830:
URL: https://github.com/apache/datafusion/pull/23830
## Which issue does this PR close?
- Part of #318.
- Umbrella PR: #23738.
- Depends on #23829 (and transitively #23828).
## Rationale for this change
This is the SQL frontend layer of the ASOF JOIN stack. It adds syntax and
unparsing only after the physical and logical contracts have been separated
into their own reviewable PRs.
GitHub shows the cumulative stack because fork branches cannot be selected as
upstream PR bases. The isolated diff for this layer is:
https://github.com/Xuanwo/datafusion/compare/xuanwo/asof-logical...xuanwo/asof-sql
## What changes are included in this PR?
- Plan `ASOF JOIN ... MATCH_CONDITION (...)` with optional `ON` or `USING`
equality keys.
- Reject unsupported match shapes and non-equality `ON` predicates.
- Unparse ASOF joins while preserving right-side candidate preselection and
nested join scope.
- Document the supported SQL syntax and semantics.
- Add SQL integration and sqllogictest coverage for all four match directions,
coercion, equality-free joins, USING, invalid contracts, EXPLAIN, boundedness,
and optimized-plan round trips.
## Are these changes tested?
Yes:
- `cargo fmt --all`
- `./ci/scripts/doc_prettier_check.sh --write --allow-dirty`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `cargo test -p datafusion --test core_integration sql::joins::asof_join --all-features`
- `cargo test -p datafusion-sqllogictest --test sqllogictests -- asof_join`
- Extended workspace tests from the contributor guide
## Are there any user-facing changes?
Users can express Snowflake-style ASOF joins in SQL with
`MATCH_CONDITION`, optional equality keys, and `<`, `<=`, `>`, or `>=` match
directions. The user guide documents the contract and examples.
This PR should be reviewed and merged after #23829. After its dependencies
squash-merge, this branch will be refreshed from `main` without rewriting its
published history.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943225380
##########
docs/source/user-guide/sql/select.md:
##########
@@ -400,6 +401,45 @@ SELECT * FROM x LEFT JOIN x AS y ON x.column_1 = y.column_2;
+----------+----------+----------+----------+
```
+### ASOF JOIN
+
+An `ASOF JOIN` matches each left row with at most one right row according to an
+ordered comparison. It preserves every left row and fills the right columns
+with `NULL` when no right row matches.
Review Comment:
Added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5535390480
cc @2010YOUY01, this PR is good to go! 🚀
Also CC @jayzhan211: would you like to follow this stack's review? Happy to ping you if you're interested. 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3938854644
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -2235,26 +2423,28 @@ impl Unparser<'_> {
}
}
- fn qualified_passthrough_join_projection_to_nested_relation(
+ fn join_input_to_nested_relation(
&self,
plan: &LogicalPlan,
query: &mut Option<QueryBuilder>,
) -> Result<Option<RelationBuilder>> {
- let LogicalPlan::Projection(projection) = plan else {
- return Ok(None);
+ let join_plan = match plan {
+ LogicalPlan::Join(_) => plan,
+ LogicalPlan::Projection(projection)
Review Comment:
Do we need asof join
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
+ &self,
+ join: &AsOfJoin,
+ query: &mut Option<QueryBuilder>,
+ select: &mut SelectBuilder,
+ relation: &mut RelationBuilder,
+ ) -> Result<()> {
+ let already_projected = select.already_projected();
+ let left_plan =
+ Self::unwrap_qualified_passthrough_join_projection(Arc::clone(&join.left));
+ let inline_left_join = matches!(left_plan.as_ref(), LogicalPlan::Join(_));
Review Comment:
Do we need asof join here
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
+ &self,
+ join: &AsOfJoin,
+ query: &mut Option<QueryBuilder>,
+ select: &mut SelectBuilder,
+ relation: &mut RelationBuilder,
+ ) -> Result<()> {
+ let already_projected = select.already_projected();
+ let left_plan =
+ Self::unwrap_qualified_passthrough_join_projection(Arc::clone(&join.left));
+ let inline_left_join = matches!(left_plan.as_ref(), LogicalPlan::Join(_));
Review Comment:
Same with `unwrap_qualified_passthrough_join_projection`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5557627830
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v55.0.0 (current)
Built [ 59.523s] (current)
Parsing datafusion v55.0.0 (current)
Parsed [ 0.036s] (current)
Building datafusion v55.0.0 (baseline)
Built [ 57.835s] (baseline)
Parsing datafusion v55.0.0 (baseline)
Parsed [ 0.035s] (baseline)
Checking datafusion v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.636s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 119.745s] datafusion
Building datafusion-expr v55.0.0 (current)
Built [ 29.044s] (current)
Parsing datafusion-expr v55.0.0 (current)
Parsed [ 0.078s] (current)
Building datafusion-expr v55.0.0 (baseline)
Built [ 29.270s] (baseline)
Parsing datafusion-expr v55.0.0 (baseline)
Parsed [ 0.084s] (baseline)
Checking datafusion-expr v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 1.170s] 223 checks: 222 pass, 1 fail, 0 warn, 31 skip
--- failure inherent_method_missing: pub method removed or renamed ---
Description:
A publicly-visible method or associated fn is no longer available under its prior name. It may have been renamed or removed entirely.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#item-remove
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.50.0/src/lints/inherent_method_missing.ron
Failed in:
LogicalPlanBuilder::asof_join, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/13b4144fa032bf81b7ddcf1c2dc2dc618eedbec2/datafusion/expr/src/logical_plan/builder.rs:1012
LogicalPlanBuilder::asof_join, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/13b4144fa032bf81b7ddcf1c2dc2dc618eedbec2/datafusion/expr/src/logical_plan/builder.rs:1012
LogicalPlanBuilder::asof_join, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/13b4144fa032bf81b7ddcf1c2dc2dc618eedbec2/datafusion/expr/src/logical_plan/builder.rs:1012
LogicalPlanBuilder::asof_join, previously in file /home/runner/work/datafusion/datafusion/target/semver-checks/git-apache_main/13b4144fa032bf81b7ddcf1c2dc2dc618eedbec2/datafusion/expr/src/logical_plan/builder.rs:1012
Summary semver requires new major version: 1 major and 0 minor checks failed
Finished [ 60.589s] datafusion-expr
Building datafusion-sql v55.0.0 (current)
Built [ 42.871s] (current)
Parsing datafusion-sql v55.0.0 (current)
Parsed [ 0.032s] (current)
Building datafusion-sql v55.0.0 (baseline)
Built [ 42.697s] (baseline)
Parsing datafusion-sql v55.0.0 (baseline)
Parsed [ 0.032s] (baseline)
Checking datafusion-sql v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.227s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 86.857s] datafusion-sql
Building datafusion-sqllogictest v55.0.0 (current)
Built [ 100.600s] (current)
Parsing datafusion-sqllogictest v55.0.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-sqllogictest v55.0.0 (baseline)
Built [ 101.892s] (baseline)
Parsing datafusion-sqllogictest v55.0.0 (baseline)
Parsed [ 0.023s] (baseline)
Checking datafusion-sqllogictest v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.097s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 205.196s] datafusion-sqllogictest
Building datafusion-substrait v55.0.0 (current)
Built [ 317.646s] (current)
Parsing datafusion-substrait v55.0.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-substrait v55.0.0 (baseline)
Built [ 317.616s] (baseline)
Parsing datafusion-substrait v55.0.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-substrait v55.0.0 -> v55.0.0 (no change; assume patch)
Checked [ 0.227s] 223 checks: 223 pass, 31 skip
Summary no semver update required
Finished [ 637.471s] datafusion-substrait
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5557567127
Thank you @jayzhan211 @2010YOUY01 for the review!
I have updated the builder API and added direct unparser coverage, including nested ASOF inputs. I kept the setup-dependent Rust tests here and will handle broader SLT consolidation separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5055105584
<!-- semver-check-comment -->
Thank you for opening this pull request!
Reviewer note: [cargo-semver-checks](https://github.com/obi1kenobi/cargo-semver-checks) reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).
<details>
<summary>Details</summary>
```
Cloning apache/main
Building datafusion v54.1.0 (current)
Built [ 109.209s] (current)
Parsing datafusion v54.1.0 (current)
Parsed [ 0.035s] (current)
Building datafusion v54.1.0 (baseline)
Built [ 105.529s] (baseline)
Parsing datafusion v54.1.0 (baseline)
Parsed [ 0.035s] (baseline)
Checking datafusion v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.596s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 217.477s] datafusion
Building datafusion-expr v54.1.0 (current)
Built [ 27.724s] (current)
Parsing datafusion-expr v54.1.0 (current)
Parsed [ 0.075s] (current)
Building datafusion-expr v54.1.0 (baseline)
Built [ 27.096s] (baseline)
Parsing datafusion-expr v54.1.0 (baseline)
Parsed [ 0.076s] (baseline)
Checking datafusion-expr v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 1.269s] 223 checks: 221 pass, 1 fail, 1 warn, 30 skip
--- failure enum_variant_added: enum variant added on exhaustive enum ---
Description:
A publicly-visible enum without #[non_exhaustive] has a new variant.
ref: https://doc.rust-lang.org/cargo/reference/semver.html#enum-variant-new
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/enum_variant_added.ron
Failed in:
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:245
variant LogicalPlan:AsOfJoin in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:245
--- warning partial_ord_enum_variants_reordered: enum variants reordered in #[derive(PartialOrd)] enum ---
Description:
A public enum that derives PartialOrd had its variants reordered. #[derive(PartialOrd)] uses the enum variant order to set the enum's ordering behavior, so this change may break downstream code that relies on the previous order.
ref: https://doc.rust-lang.org/std/cmp/trait.PartialOrd.html#derivable
impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.49.0/src/lints/partial_ord_enum_variants_reordered.ron
Failed in:
LogicalPlan::Repartition moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:249
LogicalPlan::Union moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:253
LogicalPlan::TableScan moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:256
LogicalPlan::EmptyRelation moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:260
LogicalPlan::Subquery moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:263
LogicalPlan::SubqueryAlias moved from position 12 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:265
LogicalPlan::Limit moved from position 13 to 14, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:267
LogicalPlan::Statement moved from position 14 to 15, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:269
LogicalPlan::Values moved from position 15 to 16, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:274
LogicalPlan::Explain moved from position 16 to 17, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:277
LogicalPlan::Analyze moved from position 17 to 18, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:281
LogicalPlan::Extension moved from position 18 to 19, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:284
LogicalPlan::Distinct moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:287
LogicalPlan::Dml moved from position 20 to 21, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:289
LogicalPlan::Ddl moved from position 21 to 22, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:291
LogicalPlan::Copy moved from position 22 to 23, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:293
LogicalPlan::DescribeTable moved from position 23 to 24, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:296
LogicalPlan::Unnest moved from position 24 to 25, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:299
LogicalPlan::RecursiveQuery moved from position 25 to 26, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
LogicalPlan::Repartition moved from position 7 to 8, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:249
LogicalPlan::Union moved from position 8 to 9, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:253
LogicalPlan::TableScan moved from position 9 to 10, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:256
LogicalPlan::EmptyRelation moved from position 10 to 11, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:260
LogicalPlan::Subquery moved from position 11 to 12, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:263
LogicalPlan::SubqueryAlias moved from position 12 to 13, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:265
LogicalPlan::Limit moved from position 13 to 14, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:267
LogicalPlan::Statement moved from position 14 to 15, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:269
LogicalPlan::Values moved from position 15 to 16, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:274
LogicalPlan::Explain moved from position 16 to 17, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:277
LogicalPlan::Analyze moved from position 17 to 18, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:281
LogicalPlan::Extension moved from position 18 to 19, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:284
LogicalPlan::Distinct moved from position 19 to 20, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:287
LogicalPlan::Dml moved from position 20 to 21, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:289
LogicalPlan::Ddl moved from position 21 to 22, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:291
LogicalPlan::Copy moved from position 22 to 23, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:293
LogicalPlan::DescribeTable moved from position 23 to 24, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:296
LogicalPlan::Unnest moved from position 24 to 25, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:299
LogicalPlan::RecursiveQuery moved from position 25 to 26, in /home/runner/work/datafusion/datafusion/datafusion/expr/src/logical_plan/plan.rs:301
Summary semver requires new major version: 1 major and 0 minor checks failed
Warning produced 1 major and 0 minor level warnings
Finished [ 57.647s] datafusion-expr
Building datafusion-optimizer v54.1.0 (current)
Built [ 27.596s] (current)
Parsing datafusion-optimizer v54.1.0 (current)
Parsed [ 0.030s] (current)
Building datafusion-optimizer v54.1.0 (baseline)
Built [ 27.809s] (baseline)
Parsing datafusion-optimizer v54.1.0 (baseline)
Parsed [ 0.031s] (baseline)
Checking datafusion-optimizer v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.161s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 56.478s] datafusion-optimizer
Building datafusion-physical-plan v54.1.0 (current)
Built [ 37.883s] (current)
Parsing datafusion-physical-plan v54.1.0 (current)
Parsed [ 0.145s] (current)
Building datafusion-physical-plan v54.1.0 (baseline)
Built [ 37.328s] (baseline)
Parsing datafusion-physical-plan v54.1.0 (baseline)
Parsed [ 0.143s] (baseline)
Checking datafusion-physical-plan v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.593s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 78.832s] datafusion-physical-plan
Building datafusion-proto v54.1.0 (current)
Built [ 60.147s] (current)
Parsing datafusion-proto v54.1.0 (current)
Parsed [ 0.018s] (current)
Building datafusion-proto v54.1.0 (baseline)
Built [ 59.149s] (baseline)
Parsing datafusion-proto v54.1.0 (baseline)
Parsed [ 0.020s] (baseline)
Checking datafusion-proto v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.234s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 120.946s] datafusion-proto
Building datafusion-sql v54.1.0 (current)
Built [ 41.266s] (current)
Parsing datafusion-sql v54.1.0 (current)
Parsed [ 0.030s] (current)
Building datafusion-sql v54.1.0 (baseline)
Built [ 41.161s] (baseline)
Parsing datafusion-sql v54.1.0 (baseline)
Parsed [ 0.031s] (baseline)
Checking datafusion-sql v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.248s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 83.865s] datafusion-sql
Building datafusion-sqllogictest v54.1.0 (current)
Built [ 185.213s] (current)
Parsing datafusion-sqllogictest v54.1.0 (current)
Parsed [ 0.021s] (current)
Building datafusion-sqllogictest v54.1.0 (baseline)
Built [ 189.859s] (baseline)
Parsing datafusion-sqllogictest v54.1.0 (baseline)
Parsed [ 0.023s] (baseline)
Checking datafusion-sqllogictest v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.094s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 378.652s] datafusion-sqllogictest
Building datafusion-substrait v54.1.0 (current)
Built [ 380.569s] (current)
Parsing datafusion-substrait v54.1.0 (current)
Parsed [ 0.019s] (current)
Building datafusion-substrait v54.1.0 (baseline)
Built [ 376.551s] (baseline)
Parsing datafusion-substrait v54.1.0 (baseline)
Parsed [ 0.018s] (baseline)
Checking datafusion-substrait v54.1.0 -> v54.1.0 (no change; assume patch)
Checked [ 0.232s] 223 checks: 223 pass, 30 skip
Summary no semver update required
Finished [ 759.703s] datafusion-substrait
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3944178901
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE asof_left(id INT, grp TEXT, ts TIMESTAMP) AS VALUES
+ (1, 'A', TIMESTAMP '2024-01-01 09:00:01'),
+ (2, 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (3, 'A', TIMESTAMP '2024-01-01 09:00:07'),
+ (4, 'B', TIMESTAMP '2024-01-01 09:00:02'),
+ (5, 'B', TIMESTAMP '2024-01-01 09:00:08'),
+ (6, NULL, TIMESTAMP '2024-01-01 09:00:03'),
+ (7, 'A', NULL);
+
+statement ok
+CREATE TABLE asof_right(grp TEXT, ts TIMESTAMP, val TEXT) AS VALUES
+ ('A', TIMESTAMP '2024-01-01 09:00:02', 'a2'),
+ ('A', TIMESTAMP '2024-01-01 09:00:04', 'a4'),
+ ('A', TIMESTAMP '2024-01-01 09:00:06', 'a6'),
+ ('B', TIMESTAMP '2024-01-01 09:00:01', 'b1'),
+ ('B', TIMESTAMP '2024-01-01 09:00:06', 'b6'),
+ (NULL, TIMESTAMP '2024-01-01 09:00:02', 'null-group'),
+ ('A', NULL, 'null-ts');
+
+# Inclusive predecessor per equality group. This also verifies unmatched left
+# rows and NULL behavior for equality keys and ordered expressions.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict predecessor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts > r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:02 a2
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Inclusive successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts <= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts < r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:06 a6
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# USING exposes one unqualified equality key.
+query TIPT
+SELECT grp, l.id, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+A 1 NULL NULL
+A 2 2024-01-01T09:00:04 a4
+A 3 2024-01-01T09:00:06 a6
+B 4 2024-01-01T09:00:01 b1
+B 5 2024-01-01T09:00:06 b6
+NULL 6 NULL NULL
+A 7 NULL NULL
+
+# Both qualified equality keys remain addressable.
+query ITT
+SELECT l.id, l.grp, r.grp
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+1 A NULL
+2 A A
+3 A A
+4 B B
+5 B B
+6 NULL NULL
+7 A NULL
+
+# Equality keys are optional.
+query IT
+SELECT l.id, r.label
+FROM (VALUES (1, 1), (2, 5), (3, CAST(NULL AS INT))) AS l(id, ts)
+ASOF JOIN (VALUES (2, 'r2'), (4, 'r4')) AS r(ts, label)
+MATCH_CONDITION (l.ts >= r.ts)
+ORDER BY l.id;
+----
+1 NULL
+2 r4
+3 NULL
+
+# Multiple equality keys form one candidate group.
+query IT
+SELECT l.id, r.val
+FROM (VALUES
+ (1, 'X', 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (2, 'Y', 'A', TIMESTAMP '2024-01-01 09:00:04')
+) AS l(id, venue, grp, ts)
+ASOF JOIN (VALUES
+ ('X', 'A', TIMESTAMP '2024-01-01 09:00:02', 'x-a2'),
+ ('Y', 'A', TIMESTAMP '2024-01-01 09:00:03', 'y-a3'),
+ ('X', 'B', TIMESTAMP '2024-01-01 09:00:04', 'x-b4')
+) AS r(venue, grp, ts, val)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.venue = r.venue AND l.grp = r.grp
+ORDER BY l.id;
+----
+1 x-a2
+2 y-a3
+
+# Candidate selection sees the right input after subquery filtering.
+query IT
+SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN (SELECT * FROM asof_right WHERE val <> 'a6') r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+WHERE l.id IN (3, 5)
+ORDER BY l.id;
+----
+3 a4
+5 b6
+
+# Equality and match operands use the planner's common coercion types.
+query II
+SELECT l.id, r.payload
+FROM (VALUES (1, CAST(5 AS SMALLINT), CAST(10 AS INT))) AS l(id, grp, ts)
+ASOF JOIN (
+ VALUES (CAST(5 AS BIGINT), CAST(9 AS BIGINT), 90)
+) AS r(grp, ts, payload)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+1 90
+
+query TT
+EXPLAIN SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+logical_plan
+01)Projection: l.id, r.val
+02)--AsOf Join: match=[l.ts >= r.ts], constraint=On, on=[l.grp = r.grp]
+03)----SubqueryAlias: l
+04)------TableScan: asof_left projection=[id, grp, ts]
+05)----SubqueryAlias: r
+06)------TableScan: asof_right projection=[grp, ts, val]
+physical_plan
+01)ProjectionExec: expr=[id@0 as id, val@5 as val]
+02)--AsOfJoinExec: on=[(grp = grp)], match=[ts >= ts]
+03)----SortExec: expr=[grp@1 ASC, ts@2 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[grp@0 ASC, ts@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]
+
+query error ASOF MATCH_CONDITION requires <, <=, >, or >=
+SELECT *
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts = r.ts)
+ON l.grp = r.grp;
+
+query error ASOF MATCH_CONDITION left operand must reference only the left input
Review Comment:
I see, that's the snowflake behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943220573
##########
datafusion/core/tests/sql/joins.rs:
##########
@@ -297,3 +299,412 @@ async fn unparse_cross_join() -> Result<()> {
Ok(())
}
+
+fn register_asof_test_tables(ctx: &SessionContext) -> Result<()> {
Review Comment:
Agreed for the SQL-level cases. I kept the Rust tests that need multi-partition plans, unbounded inputs, or plan-to-SQL roundtrips. I’ll consolidate the remaining overlap in a follow-up.
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1007,8 +1009,12 @@ impl LogicalPlanBuilder {
)
}
- /// Apply a left-preserving ASOF join using equality expressions and one
- /// ordered match condition.
+ /// Apply a left-preserving ASOF join using pre-separated equality
+ /// expressions and a structured ordered match condition.
+ ///
+ /// This is a low-level API. Most callers should use
+ /// [`asof_join_on`](Self::asof_join_on), which validates and separates
+ /// ordinary predicate expressions.
pub fn asof_join(
Review Comment:
removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 merged PR #23830:
URL: https://github.com/apache/datafusion/pull/23830
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943224902
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
+ &self,
Review Comment:
There are unparser test suite in `datafusion/sql/tests/cases/plan_to_sql.rs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943217335
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
+ &self,
+ join: &AsOfJoin,
+ query: &mut Option<QueryBuilder>,
+ select: &mut SelectBuilder,
+ relation: &mut RelationBuilder,
+ ) -> Result<()> {
+ let already_projected = select.already_projected();
+ let left_plan =
+ Self::unwrap_qualified_passthrough_join_projection(Arc::clone(&join.left));
+ let inline_left_join = matches!(left_plan.as_ref(), LogicalPlan::Join(_));
Review Comment:
Yes, good catch. I added AsOfJoin here.
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -2235,26 +2423,28 @@ impl Unparser<'_> {
}
}
- fn qualified_passthrough_join_projection_to_nested_relation(
+ fn join_input_to_nested_relation(
&self,
plan: &LogicalPlan,
query: &mut Option<QueryBuilder>,
) -> Result<Option<RelationBuilder>> {
- let LogicalPlan::Projection(projection) = plan else {
- return Ok(None);
+ let join_plan = match plan {
+ LogicalPlan::Join(_) => plan,
+ LogicalPlan::Projection(projection)
Review Comment:
updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943221389
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1018,6 +1024,48 @@ impl LogicalPlanBuilder {
self.asof_join_with_constraint(right, on, match_condition, JoinConstraint::On)
}
+ /// Apply a left-preserving ASOF join using ordinary `ON` and
+ /// `MATCH_CONDITION` expressions.
+ ///
+ /// Equality predicates may be combined with `AND`. This method validates
+ /// and separates their left and right operands before constructing the
+ /// structured ASOF logical plan.
Review Comment:
Updated 💌
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943223622
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
Review Comment:
Will treat this change as a follow up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5548118863
> cc @2010YOUY01, this PR is good to go! 🚀
>
> Also CC @jayzhan211: would you like to follow this stack's review? Happy to ping you if you're interested. 😄
Of course!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3942757156
##########
datafusion/core/tests/sql/joins.rs:
##########
@@ -297,3 +299,412 @@ async fn unparse_cross_join() -> Result<()> {
Ok(())
}
+
+fn register_asof_test_tables(ctx: &SessionContext) -> Result<()> {
Review Comment:
Can we unify those tests into sqllogictests, this way I think it's easier to maintain. Unless there are some setups that is not possible to do in `slt`, we might want extra coverage here.
In order to move those tests checking partition properties, I think we can assert the end behavior (final plan shape), rather than internal properties.
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1007,8 +1009,12 @@ impl LogicalPlanBuilder {
)
}
- /// Apply a left-preserving ASOF join using equality expressions and one
- /// ordered match condition.
+ /// Apply a left-preserving ASOF join using pre-separated equality
+ /// expressions and a structured ordered match condition.
+ ///
+ /// This is a low-level API. Most callers should use
+ /// [`asof_join_on`](Self::asof_join_on), which validates and separates
+ /// ordinary predicate expressions.
pub fn asof_join(
Review Comment:
It seems we can remove this function, since it directly calls a helper.
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
Review Comment:
I don't fully understand the unparser now, but my AI tool suggest it is good to go, with some cleanup advices:
### AI review
> The unparser is heavy relative to the join it mirrors. About 120 lines of branching on "already projected" versus not, and "needs a derived subquery" versus not, for each side. The regular join unparser has the same structural problem, so this is inherited rather than introduced. A cleaner alternative would be a small shared helper that takes a join input and returns both the relation and its projection items, used by both join kinds. That refactor is bigger than this PR should carry, but it would be a natural follow-up given the branch already touched the shared nesting helper.
##########
datafusion/sql/src/relation/join.rs:
##########
@@ -98,10 +98,77 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
JoinOperator::CrossJoin(JoinConstraint::None) => {
self.parse_cross_join(left, right)
}
+ JoinOperator::AsOf {
+ match_condition,
+ constraint,
+ } => self.parse_asof_join(
+ left,
+ right,
+ match_condition,
+ constraint,
+ planner_context,
+ ),
other => not_impl_err!("Unsupported JOIN operator {other:?}"),
}
}
+ fn parse_asof_join(
+ &self,
+ left: LogicalPlan,
+ right: LogicalPlan,
+ sql_match_condition: sqlparser::ast::Expr,
+ constraint: JoinConstraint,
+ planner_context: &mut PlannerContext,
+ ) -> Result<LogicalPlan> {
+ let join_schema = left.schema().join(right.schema())?;
+ let match_condition =
+ self.sql_to_expr(sql_match_condition, &join_schema, planner_context)?;
+
+ match constraint {
+ JoinConstraint::On(sql_on) => {
+ let on = self.sql_to_expr(sql_on, &join_schema, planner_context)?;
+ LogicalPlanBuilder::from(left)
+ .asof_join_on(right, [on], match_condition)?
+ .build()
+ }
+ JoinConstraint::Using(object_names) => {
+ let keys = object_names
+ .into_iter()
+ .map(|object_name| {
+ let ObjectName(mut object_names) = object_name;
+ if object_names.len() != 1 {
+ return not_impl_err!(
+ "Invalid identifier in ASOF USING clause. Expected single identifier, got {}",
+ ObjectName(object_names)
+ );
+ }
+ let id = object_names.swap_remove(0);
+ id.as_ident()
+ .ok_or_else(|| {
+ plan_datafusion_err!(
+ "Expected identifier in ASOF USING clause"
+ )
+ })
+ .map(|ident| {
+ Column::from_name(
+ self.ident_normalizer.normalize(ident.clone()),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ LogicalPlanBuilder::from(left)
+ .asof_join_using(right, keys, AsOfMatch::try_from(match_condition)?)?
+ .build()
+ }
+ JoinConstraint::None => LogicalPlanBuilder::from(left)
+ .asof_join_on(right, [], match_condition)?
+ .build(),
+ JoinConstraint::Natural => {
+ not_impl_err!("NATURAL ASOF JOIN is not supported")
Review Comment:
It seem to be invalid syntax, then it should be `plan_err` instead
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -1917,6 +1917,126 @@ impl Unparser<'_> {
}
}
+ // Keep ASOF-specific locals out of the recursive plan unparser's stack frame.
+ #[inline(never)]
+ fn asof_join_to_sql(
+ &self,
Review Comment:
btw how do we test unparser now, is there any test infra for roundtrip tests?
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE asof_left(id INT, grp TEXT, ts TIMESTAMP) AS VALUES
+ (1, 'A', TIMESTAMP '2024-01-01 09:00:01'),
+ (2, 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (3, 'A', TIMESTAMP '2024-01-01 09:00:07'),
+ (4, 'B', TIMESTAMP '2024-01-01 09:00:02'),
+ (5, 'B', TIMESTAMP '2024-01-01 09:00:08'),
+ (6, NULL, TIMESTAMP '2024-01-01 09:00:03'),
+ (7, 'A', NULL);
+
+statement ok
+CREATE TABLE asof_right(grp TEXT, ts TIMESTAMP, val TEXT) AS VALUES
+ ('A', TIMESTAMP '2024-01-01 09:00:02', 'a2'),
+ ('A', TIMESTAMP '2024-01-01 09:00:04', 'a4'),
+ ('A', TIMESTAMP '2024-01-01 09:00:06', 'a6'),
+ ('B', TIMESTAMP '2024-01-01 09:00:01', 'b1'),
+ ('B', TIMESTAMP '2024-01-01 09:00:06', 'b6'),
+ (NULL, TIMESTAMP '2024-01-01 09:00:02', 'null-group'),
+ ('A', NULL, 'null-ts');
+
+# Inclusive predecessor per equality group. This also verifies unmatched left
+# rows and NULL behavior for equality keys and ordered expressions.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict predecessor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts > r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:02 a2
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Inclusive successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts <= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts < r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:06 a6
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# USING exposes one unqualified equality key.
+query TIPT
+SELECT grp, l.id, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+A 1 NULL NULL
+A 2 2024-01-01T09:00:04 a4
+A 3 2024-01-01T09:00:06 a6
+B 4 2024-01-01T09:00:01 b1
+B 5 2024-01-01T09:00:06 b6
+NULL 6 NULL NULL
+A 7 NULL NULL
+
+# Both qualified equality keys remain addressable.
+query ITT
+SELECT l.id, l.grp, r.grp
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+1 A NULL
+2 A A
+3 A A
+4 B B
+5 B B
+6 NULL NULL
+7 A NULL
+
+# Equality keys are optional.
+query IT
+SELECT l.id, r.label
+FROM (VALUES (1, 1), (2, 5), (3, CAST(NULL AS INT))) AS l(id, ts)
+ASOF JOIN (VALUES (2, 'r2'), (4, 'r4')) AS r(ts, label)
+MATCH_CONDITION (l.ts >= r.ts)
+ORDER BY l.id;
+----
+1 NULL
+2 r4
+3 NULL
+
+# Multiple equality keys form one candidate group.
+query IT
+SELECT l.id, r.val
+FROM (VALUES
+ (1, 'X', 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (2, 'Y', 'A', TIMESTAMP '2024-01-01 09:00:04')
+) AS l(id, venue, grp, ts)
+ASOF JOIN (VALUES
+ ('X', 'A', TIMESTAMP '2024-01-01 09:00:02', 'x-a2'),
+ ('Y', 'A', TIMESTAMP '2024-01-01 09:00:03', 'y-a3'),
+ ('X', 'B', TIMESTAMP '2024-01-01 09:00:04', 'x-b4')
+) AS r(venue, grp, ts, val)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.venue = r.venue AND l.grp = r.grp
+ORDER BY l.id;
+----
+1 x-a2
+2 y-a3
+
+# Candidate selection sees the right input after subquery filtering.
+query IT
+SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN (SELECT * FROM asof_right WHERE val <> 'a6') r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+WHERE l.id IN (3, 5)
+ORDER BY l.id;
+----
+3 a4
+5 b6
+
+# Equality and match operands use the planner's common coercion types.
+query II
+SELECT l.id, r.payload
+FROM (VALUES (1, CAST(5 AS SMALLINT), CAST(10 AS INT))) AS l(id, grp, ts)
+ASOF JOIN (
+ VALUES (CAST(5 AS BIGINT), CAST(9 AS BIGINT), 90)
+) AS r(grp, ts, payload)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+1 90
+
+query TT
+EXPLAIN SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+logical_plan
+01)Projection: l.id, r.val
+02)--AsOf Join: match=[l.ts >= r.ts], constraint=On, on=[l.grp = r.grp]
+03)----SubqueryAlias: l
+04)------TableScan: asof_left projection=[id, grp, ts]
+05)----SubqueryAlias: r
+06)------TableScan: asof_right projection=[grp, ts, val]
+physical_plan
+01)ProjectionExec: expr=[id@0 as id, val@5 as val]
+02)--AsOfJoinExec: on=[(grp = grp)], match=[ts >= ts]
+03)----SortExec: expr=[grp@1 ASC, ts@2 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[grp@0 ASC, ts@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]
+
+query error ASOF MATCH_CONDITION requires <, <=, >, or >=
+SELECT *
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts = r.ts)
+ON l.grp = r.grp;
+
+query error ASOF MATCH_CONDITION left operand must reference only the left input
Review Comment:
Should this query get normalized internally, and make it valid? 🤔
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1018,6 +1024,48 @@ impl LogicalPlanBuilder {
self.asof_join_with_constraint(right, on, match_condition, JoinConstraint::On)
}
+ /// Apply a left-preserving ASOF join using ordinary `ON` and
+ /// `MATCH_CONDITION` expressions.
+ ///
+ /// Equality predicates may be combined with `AND`. This method validates
+ /// and separates their left and right operands before constructing the
+ /// structured ASOF logical plan.
+ pub fn asof_join_on(
+ self,
+ right: LogicalPlan,
+ on_exprs: impl IntoIterator<Item = Expr>,
Review Comment:
I think it's better to make it a single `Expr`, otherwise callers have to figure out 'vec of expr means ANDed conjunction', I think it's an extra complexity.
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1018,6 +1024,48 @@ impl LogicalPlanBuilder {
self.asof_join_with_constraint(right, on, match_condition, JoinConstraint::On)
}
+ /// Apply a left-preserving ASOF join using ordinary `ON` and
+ /// `MATCH_CONDITION` expressions.
+ ///
+ /// Equality predicates may be combined with `AND`. This method validates
+ /// and separates their left and right operands before constructing the
+ /// structured ASOF logical plan.
Review Comment:
I think now the comments explains the internal mechanism, but it should be user-facing: like specifying the `on_expr` and `match_condition`
e.g. match_condition must be in the form `l_expr cmp r_expr`, `l_expr` is an expression only referencing left relation columns, ...
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
Review Comment:
```suggestion
match_condition: Expr,
```
to make it consistent with `asof_join_on`
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE asof_left(id INT, grp TEXT, ts TIMESTAMP) AS VALUES
+ (1, 'A', TIMESTAMP '2024-01-01 09:00:01'),
+ (2, 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (3, 'A', TIMESTAMP '2024-01-01 09:00:07'),
+ (4, 'B', TIMESTAMP '2024-01-01 09:00:02'),
+ (5, 'B', TIMESTAMP '2024-01-01 09:00:08'),
+ (6, NULL, TIMESTAMP '2024-01-01 09:00:03'),
+ (7, 'A', NULL);
+
+statement ok
+CREATE TABLE asof_right(grp TEXT, ts TIMESTAMP, val TEXT) AS VALUES
+ ('A', TIMESTAMP '2024-01-01 09:00:02', 'a2'),
+ ('A', TIMESTAMP '2024-01-01 09:00:04', 'a4'),
+ ('A', TIMESTAMP '2024-01-01 09:00:06', 'a6'),
+ ('B', TIMESTAMP '2024-01-01 09:00:01', 'b1'),
+ ('B', TIMESTAMP '2024-01-01 09:00:06', 'b6'),
+ (NULL, TIMESTAMP '2024-01-01 09:00:02', 'null-group'),
+ ('A', NULL, 'null-ts');
+
+# Inclusive predecessor per equality group. This also verifies unmatched left
+# rows and NULL behavior for equality keys and ordered expressions.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict predecessor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts > r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:02 a2
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Inclusive successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts <= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts < r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:06 a6
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# USING exposes one unqualified equality key.
+query TIPT
+SELECT grp, l.id, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+A 1 NULL NULL
+A 2 2024-01-01T09:00:04 a4
+A 3 2024-01-01T09:00:06 a6
+B 4 2024-01-01T09:00:01 b1
+B 5 2024-01-01T09:00:06 b6
+NULL 6 NULL NULL
+A 7 NULL NULL
+
+# Both qualified equality keys remain addressable.
+query ITT
+SELECT l.id, l.grp, r.grp
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+1 A NULL
+2 A A
+3 A A
+4 B B
+5 B B
+6 NULL NULL
+7 A NULL
+
+# Equality keys are optional.
+query IT
+SELECT l.id, r.label
+FROM (VALUES (1, 1), (2, 5), (3, CAST(NULL AS INT))) AS l(id, ts)
+ASOF JOIN (VALUES (2, 'r2'), (4, 'r4')) AS r(ts, label)
+MATCH_CONDITION (l.ts >= r.ts)
+ORDER BY l.id;
+----
+1 NULL
+2 r4
+3 NULL
+
+# Multiple equality keys form one candidate group.
+query IT
+SELECT l.id, r.val
+FROM (VALUES
+ (1, 'X', 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (2, 'Y', 'A', TIMESTAMP '2024-01-01 09:00:04')
+) AS l(id, venue, grp, ts)
+ASOF JOIN (VALUES
+ ('X', 'A', TIMESTAMP '2024-01-01 09:00:02', 'x-a2'),
+ ('Y', 'A', TIMESTAMP '2024-01-01 09:00:03', 'y-a3'),
+ ('X', 'B', TIMESTAMP '2024-01-01 09:00:04', 'x-b4')
+) AS r(venue, grp, ts, val)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.venue = r.venue AND l.grp = r.grp
+ORDER BY l.id;
+----
+1 x-a2
+2 y-a3
+
+# Candidate selection sees the right input after subquery filtering.
+query IT
+SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN (SELECT * FROM asof_right WHERE val <> 'a6') r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+WHERE l.id IN (3, 5)
+ORDER BY l.id;
+----
+3 a4
+5 b6
+
+# Equality and match operands use the planner's common coercion types.
Review Comment:
I suggest to add an explain to this testcase, to ensure the coercion is on the right side
##########
docs/source/user-guide/sql/select.md:
##########
@@ -400,6 +401,45 @@ SELECT * FROM x LEFT JOIN x AS y ON x.column_1 = y.column_2;
+----------+----------+----------+----------+
```
+### ASOF JOIN
+
+An `ASOF JOIN` matches each left row with at most one right row according to an
+ordered comparison. It preserves every left row and fills the right columns
+with `NULL` when no right row matches.
Review Comment:
We can also mention DataFusion is following snowflake syntax, and add a reference link.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943226384
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE asof_left(id INT, grp TEXT, ts TIMESTAMP) AS VALUES
+ (1, 'A', TIMESTAMP '2024-01-01 09:00:01'),
+ (2, 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (3, 'A', TIMESTAMP '2024-01-01 09:00:07'),
+ (4, 'B', TIMESTAMP '2024-01-01 09:00:02'),
+ (5, 'B', TIMESTAMP '2024-01-01 09:00:08'),
+ (6, NULL, TIMESTAMP '2024-01-01 09:00:03'),
+ (7, 'A', NULL);
+
+statement ok
+CREATE TABLE asof_right(grp TEXT, ts TIMESTAMP, val TEXT) AS VALUES
+ ('A', TIMESTAMP '2024-01-01 09:00:02', 'a2'),
+ ('A', TIMESTAMP '2024-01-01 09:00:04', 'a4'),
+ ('A', TIMESTAMP '2024-01-01 09:00:06', 'a6'),
+ ('B', TIMESTAMP '2024-01-01 09:00:01', 'b1'),
+ ('B', TIMESTAMP '2024-01-01 09:00:06', 'b6'),
+ (NULL, TIMESTAMP '2024-01-01 09:00:02', 'null-group'),
+ ('A', NULL, 'null-ts');
+
+# Inclusive predecessor per equality group. This also verifies unmatched left
+# rows and NULL behavior for equality keys and ordered expressions.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict predecessor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts > r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:02 a2
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Inclusive successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts <= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts < r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:06 a6
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# USING exposes one unqualified equality key.
+query TIPT
+SELECT grp, l.id, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+A 1 NULL NULL
+A 2 2024-01-01T09:00:04 a4
+A 3 2024-01-01T09:00:06 a6
+B 4 2024-01-01T09:00:01 b1
+B 5 2024-01-01T09:00:06 b6
+NULL 6 NULL NULL
+A 7 NULL NULL
+
+# Both qualified equality keys remain addressable.
+query ITT
+SELECT l.id, l.grp, r.grp
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+1 A NULL
+2 A A
+3 A A
+4 B B
+5 B B
+6 NULL NULL
+7 A NULL
+
+# Equality keys are optional.
+query IT
+SELECT l.id, r.label
+FROM (VALUES (1, 1), (2, 5), (3, CAST(NULL AS INT))) AS l(id, ts)
+ASOF JOIN (VALUES (2, 'r2'), (4, 'r4')) AS r(ts, label)
+MATCH_CONDITION (l.ts >= r.ts)
+ORDER BY l.id;
+----
+1 NULL
+2 r4
+3 NULL
+
+# Multiple equality keys form one candidate group.
+query IT
+SELECT l.id, r.val
+FROM (VALUES
+ (1, 'X', 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (2, 'Y', 'A', TIMESTAMP '2024-01-01 09:00:04')
+) AS l(id, venue, grp, ts)
+ASOF JOIN (VALUES
+ ('X', 'A', TIMESTAMP '2024-01-01 09:00:02', 'x-a2'),
+ ('Y', 'A', TIMESTAMP '2024-01-01 09:00:03', 'y-a3'),
+ ('X', 'B', TIMESTAMP '2024-01-01 09:00:04', 'x-b4')
+) AS r(venue, grp, ts, val)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.venue = r.venue AND l.grp = r.grp
+ORDER BY l.id;
+----
+1 x-a2
+2 y-a3
+
+# Candidate selection sees the right input after subquery filtering.
+query IT
+SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN (SELECT * FROM asof_right WHERE val <> 'a6') r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+WHERE l.id IN (3, 5)
+ORDER BY l.id;
+----
+3 a4
+5 b6
+
+# Equality and match operands use the planner's common coercion types.
+query II
+SELECT l.id, r.payload
+FROM (VALUES (1, CAST(5 AS SMALLINT), CAST(10 AS INT))) AS l(id, grp, ts)
+ASOF JOIN (
+ VALUES (CAST(5 AS BIGINT), CAST(9 AS BIGINT), 90)
+) AS r(grp, ts, payload)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+1 90
+
+query TT
+EXPLAIN SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp;
+----
+logical_plan
+01)Projection: l.id, r.val
+02)--AsOf Join: match=[l.ts >= r.ts], constraint=On, on=[l.grp = r.grp]
+03)----SubqueryAlias: l
+04)------TableScan: asof_left projection=[id, grp, ts]
+05)----SubqueryAlias: r
+06)------TableScan: asof_right projection=[grp, ts, val]
+physical_plan
+01)ProjectionExec: expr=[id@0 as id, val@5 as val]
+02)--AsOfJoinExec: on=[(grp = grp)], match=[ts >= ts]
+03)----SortExec: expr=[grp@1 ASC, ts@2 ASC], preserve_partitioning=[false]
+04)------DataSourceExec: partitions=1, partition_sizes=[1]
+05)----SortExec: expr=[grp@0 ASC, ts@1 ASC], preserve_partitioning=[false]
+06)------DataSourceExec: partitions=1, partition_sizes=[1]
+
+query error ASOF MATCH_CONDITION requires <, <=, >, or >=
+SELECT *
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts = r.ts)
+ON l.grp = r.grp;
+
+query error ASOF MATCH_CONDITION left operand must reference only the left input
Review Comment:
I’d keep this invalid. [Snowflake defines MATCH_CONDITION operand order semantically](https://docs.snowflake.com/en/sql-reference/constructs/asof-join), so the left input must remain on the left. ON equality operands are still accepted in either order.
##########
datafusion/sqllogictest/test_files/asof_join.slt:
##########
@@ -0,0 +1,245 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+statement ok
+CREATE TABLE asof_left(id INT, grp TEXT, ts TIMESTAMP) AS VALUES
+ (1, 'A', TIMESTAMP '2024-01-01 09:00:01'),
+ (2, 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (3, 'A', TIMESTAMP '2024-01-01 09:00:07'),
+ (4, 'B', TIMESTAMP '2024-01-01 09:00:02'),
+ (5, 'B', TIMESTAMP '2024-01-01 09:00:08'),
+ (6, NULL, TIMESTAMP '2024-01-01 09:00:03'),
+ (7, 'A', NULL);
+
+statement ok
+CREATE TABLE asof_right(grp TEXT, ts TIMESTAMP, val TEXT) AS VALUES
+ ('A', TIMESTAMP '2024-01-01 09:00:02', 'a2'),
+ ('A', TIMESTAMP '2024-01-01 09:00:04', 'a4'),
+ ('A', TIMESTAMP '2024-01-01 09:00:06', 'a6'),
+ ('B', TIMESTAMP '2024-01-01 09:00:01', 'b1'),
+ ('B', TIMESTAMP '2024-01-01 09:00:06', 'b6'),
+ (NULL, TIMESTAMP '2024-01-01 09:00:02', 'null-group'),
+ ('A', NULL, 'null-ts');
+
+# Inclusive predecessor per equality group. This also verifies unmatched left
+# rows and NULL behavior for equality keys and ordered expressions.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict predecessor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts > r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 NULL NULL
+2 2024-01-01T09:00:04 2024-01-01T09:00:02 a2
+3 2024-01-01T09:00:07 2024-01-01T09:00:06 a6
+4 2024-01-01T09:00:02 2024-01-01T09:00:01 b1
+5 2024-01-01T09:00:08 2024-01-01T09:00:06 b6
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Inclusive successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts <= r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:04 a4
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# Strict successor per equality group.
+query IPPT
+SELECT l.id, l.ts, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts < r.ts)
+ON l.grp = r.grp
+ORDER BY l.id;
+----
+1 2024-01-01T09:00:01 2024-01-01T09:00:02 a2
+2 2024-01-01T09:00:04 2024-01-01T09:00:06 a6
+3 2024-01-01T09:00:07 NULL NULL
+4 2024-01-01T09:00:02 2024-01-01T09:00:06 b6
+5 2024-01-01T09:00:08 NULL NULL
+6 2024-01-01T09:00:03 NULL NULL
+7 NULL NULL NULL
+
+# USING exposes one unqualified equality key.
+query TIPT
+SELECT grp, l.id, r.ts, r.val
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+A 1 NULL NULL
+A 2 2024-01-01T09:00:04 a4
+A 3 2024-01-01T09:00:06 a6
+B 4 2024-01-01T09:00:01 b1
+B 5 2024-01-01T09:00:06 b6
+NULL 6 NULL NULL
+A 7 NULL NULL
+
+# Both qualified equality keys remain addressable.
+query ITT
+SELECT l.id, l.grp, r.grp
+FROM asof_left l
+ASOF JOIN asof_right r
+MATCH_CONDITION (l.ts >= r.ts)
+USING (grp)
+ORDER BY l.id;
+----
+1 A NULL
+2 A A
+3 A A
+4 B B
+5 B B
+6 NULL NULL
+7 A NULL
+
+# Equality keys are optional.
+query IT
+SELECT l.id, r.label
+FROM (VALUES (1, 1), (2, 5), (3, CAST(NULL AS INT))) AS l(id, ts)
+ASOF JOIN (VALUES (2, 'r2'), (4, 'r4')) AS r(ts, label)
+MATCH_CONDITION (l.ts >= r.ts)
+ORDER BY l.id;
+----
+1 NULL
+2 r4
+3 NULL
+
+# Multiple equality keys form one candidate group.
+query IT
+SELECT l.id, r.val
+FROM (VALUES
+ (1, 'X', 'A', TIMESTAMP '2024-01-01 09:00:04'),
+ (2, 'Y', 'A', TIMESTAMP '2024-01-01 09:00:04')
+) AS l(id, venue, grp, ts)
+ASOF JOIN (VALUES
+ ('X', 'A', TIMESTAMP '2024-01-01 09:00:02', 'x-a2'),
+ ('Y', 'A', TIMESTAMP '2024-01-01 09:00:03', 'y-a3'),
+ ('X', 'B', TIMESTAMP '2024-01-01 09:00:04', 'x-b4')
+) AS r(venue, grp, ts, val)
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.venue = r.venue AND l.grp = r.grp
+ORDER BY l.id;
+----
+1 x-a2
+2 y-a3
+
+# Candidate selection sees the right input after subquery filtering.
+query IT
+SELECT l.id, r.val
+FROM asof_left l
+ASOF JOIN (SELECT * FROM asof_right WHERE val <> 'a6') r
+MATCH_CONDITION (l.ts >= r.ts)
+ON l.grp = r.grp
+WHERE l.id IN (3, 5)
+ORDER BY l.id;
+----
+3 a4
+5 b6
+
+# Equality and match operands use the planner's common coercion types.
Review Comment:
added.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "2010YOUY01 (via GitHub)" <gi...@apache.org>.
2010YOUY01 commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5559755671
> Thank you @jayzhan211 @2010YOUY01 for the review!
>
> I have updated the builder API and added direct unparser coverage, including nested ASOF inputs. I kept the setup-dependent Rust tests here and will handle broader SLT consolidation separately.
Sounds like a good plan!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943222616
##########
datafusion/sql/src/relation/join.rs:
##########
@@ -98,10 +98,77 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
JoinOperator::CrossJoin(JoinConstraint::None) => {
self.parse_cross_join(left, right)
}
+ JoinOperator::AsOf {
+ match_condition,
+ constraint,
+ } => self.parse_asof_join(
+ left,
+ right,
+ match_condition,
+ constraint,
+ planner_context,
+ ),
other => not_impl_err!("Unsupported JOIN operator {other:?}"),
}
}
+ fn parse_asof_join(
+ &self,
+ left: LogicalPlan,
+ right: LogicalPlan,
+ sql_match_condition: sqlparser::ast::Expr,
+ constraint: JoinConstraint,
+ planner_context: &mut PlannerContext,
+ ) -> Result<LogicalPlan> {
+ let join_schema = left.schema().join(right.schema())?;
+ let match_condition =
+ self.sql_to_expr(sql_match_condition, &join_schema, planner_context)?;
+
+ match constraint {
+ JoinConstraint::On(sql_on) => {
+ let on = self.sql_to_expr(sql_on, &join_schema, planner_context)?;
+ LogicalPlanBuilder::from(left)
+ .asof_join_on(right, [on], match_condition)?
+ .build()
+ }
+ JoinConstraint::Using(object_names) => {
+ let keys = object_names
+ .into_iter()
+ .map(|object_name| {
+ let ObjectName(mut object_names) = object_name;
+ if object_names.len() != 1 {
+ return not_impl_err!(
+ "Invalid identifier in ASOF USING clause. Expected single identifier, got {}",
+ ObjectName(object_names)
+ );
+ }
+ let id = object_names.swap_remove(0);
+ id.as_ident()
+ .ok_or_else(|| {
+ plan_datafusion_err!(
+ "Expected identifier in ASOF USING clause"
+ )
+ })
+ .map(|ident| {
+ Column::from_name(
+ self.ident_normalizer.normalize(ident.clone()),
+ )
+ })
+ })
+ .collect::<Result<Vec<_>>>()?;
+ LogicalPlanBuilder::from(left)
+ .asof_join_using(right, keys, AsOfMatch::try_from(match_condition)?)?
+ .build()
+ }
+ JoinConstraint::None => LogicalPlanBuilder::from(left)
+ .asof_join_on(right, [], match_condition)?
+ .build(),
+ JoinConstraint::Natural => {
+ not_impl_err!("NATURAL ASOF JOIN is not supported")
Review Comment:
changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943221860
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
@@ -1018,6 +1024,48 @@ impl LogicalPlanBuilder {
self.asof_join_with_constraint(right, on, match_condition, JoinConstraint::On)
}
+ /// Apply a left-preserving ASOF join using ordinary `ON` and
+ /// `MATCH_CONDITION` expressions.
+ ///
+ /// Equality predicates may be combined with `AND`. This method validates
+ /// and separates their left and right operands before constructing the
+ /// structured ASOF logical plan.
+ pub fn asof_join_on(
+ self,
+ right: LogicalPlan,
+ on_exprs: impl IntoIterator<Item = Expr>,
Review Comment:
Agreed and updated.
##########
datafusion/expr/src/logical_plan/builder.rs:
##########
Review Comment:
Updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on code in PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#discussion_r3943508612
##########
datafusion/core/tests/sql/joins.rs:
##########
@@ -297,3 +299,412 @@ async fn unparse_cross_join() -> Result<()> {
Ok(())
}
+
+fn register_asof_test_tables(ctx: &SessionContext) -> Result<()> {
Review Comment:
See #24964
##########
datafusion/core/tests/sql/joins.rs:
##########
@@ -297,3 +299,412 @@ async fn unparse_cross_join() -> Result<()> {
Ok(())
}
+
+fn register_asof_test_tables(ctx: &SessionContext) -> Result<()> {
Review Comment:
Follow-up is in #24964.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: support ASOF JOIN SQL [datafusion]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23830:
URL: https://github.com/apache/datafusion/pull/23830#issuecomment-5055090163
## [Codecov](https://app.codecov.io/gh/apache/datafusion/pull/23830?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `79.33255%` with `353 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 80.72%. Comparing base ([`ba8796e`](https://app.codecov.io/gh/apache/datafusion/commit/ba8796e460bd60b8239c5f2f51f83d464efc177b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`fe3b182`](https://app.codecov.io/gh/apache/datafusion/commit/fe3b1827c9154d2a136299f7f4955c06bf500aba?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/datafusion/pull/23830?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [datafusion/physical-plan/src/joins/asof\_join.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fphysical-plan%2Fsrc%2Fjoins%2Fasof_join.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9waHlzaWNhbC1wbGFuL3NyYy9qb2lucy9hc29mX2pvaW4ucnM=) | 87.28% | [73 Missing and 55 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vcGxhbi5ycw==) | 55.08% | [63 Missing and 12 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/sql/src/unparser/plan.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Funparser%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3VucGFyc2VyL3BsYW4ucnM=) | 66.12% | [49 Missing and 14 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/optimizer/src/push\_down\_filter.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Fpush_down_filter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL3B1c2hfZG93bl9maWx0ZXIucnM=) | 0.00% | [24 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/builder.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fbuilder.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vYnVpbGRlci5ycw==) | 81.70% | [3 Missing and 12 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/sql/src/relation/join.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fsql%2Fsrc%2Frelation%2Fjoin.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9zcWwvc3JjL3JlbGF0aW9uL2pvaW4ucnM=) | 83.33% | [7 Missing and 8 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/core/src/physical\_planner.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fcore%2Fsrc%2Fphysical_planner.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9jb3JlL3NyYy9waHlzaWNhbF9wbGFubmVyLnJz) | 81.81% | [6 Missing and 4 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/display.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Fdisplay.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vZGlzcGxheS5ycw==) | 0.00% | [10 Missing :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [datafusion/expr/src/logical\_plan/tree\_node.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Fexpr%2Fsrc%2Flogical_plan%2Ftree_node.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9leHByL3NyYy9sb2dpY2FsX3BsYW4vdHJlZV9ub2RlLnJz) | 93.02% | [0 Missing and 3 partials :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...tafusion/optimizer/src/optimize\_projections/mod.rs](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&filepath=datafusion%2Foptimizer%2Fsrc%2Foptimize_projections%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-ZGF0YWZ1c2lvbi9vcHRpbWl6ZXIvc3JjL29wdGltaXplX3Byb2plY3Rpb25zL21vZC5ycw==) | 57.14% | [2 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [3 more](https://app.codecov.io/gh/apache/datafusion/pull/23830?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #23830 +/- ##
==========================================
- Coverage 80.73% 80.72% -0.01%
==========================================
Files 1089 1090 +1
Lines 368948 370643 +1695
Branches 368948 370643 +1695
==========================================
+ Hits 297857 299195 +1338
- Misses 53334 53578 +244
- Partials 17757 17870 +113
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/datafusion/pull/23830?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org