You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "gmhelmold (via GitHub)" <gi...@apache.org> on 2026/07/11 20:10:32 UTC
[PR] fix: use build_join_schema for CrossJoinExec output schema metadata [datafusion]
gmhelmold opened a new pull request, #23485:
URL: https://github.com/apache/datafusion/pull/23485
## Which issue does this PR close?
Closes #23434.
## Rationale for this change
An aggregate over a pure cross join of two tables that carry the *same* schema-metadata key with different values fails at planning time:
```
Internal error: Physical input schema should be the same as the one converted from
logical input schema. Differences:
- schema metadata differs: (physical) {"pandas": "RIGHT-SCHEMA"}
vs (logical) {"pandas": "LEFT-SCHEMA"}
```
The logical plan builds a cross join as `Join { join_type: Inner, .. }` (`LogicalPlanBuilder::cross_join`), so its schema comes from `build_join_schema`, which merges input metadata **left-wins**. `CrossJoinExec::new`, however, merged metadata inline with an `extend`, which is **right-wins**. The two schemas therefore disagree on any conflicting metadata key, and the physical-vs-logical schema check rejects the plan.
`CrossJoinExec` is the only join operator that still hand-rolled its output schema. `HashJoinExec`, `NestedLoopJoinExec`, `SortMergeJoinExec`, `SymmetricHashJoinExec` and the piecewise merge join all already build their schema via `build_join_schema`. This aligns cross join with that precedent, in the same family as the union-metadata consistency fix #21127.
## What changes are included in this PR?
- `CrossJoinExec::new` now derives its output schema from `build_join_schema(&left.schema(), &right.schema(), &JoinType::Inner)` instead of the inline field-chain + right-wins metadata `extend`. The field list (left-then-right, no induced nullability — Inner sets `force_nullable = false`) is unchanged; only the metadata-merge order changes to left-wins, matching the logical schema.
- A unit test (`test_cross_join_metadata`) mirroring `joins::utils::test_join_metadata`.
- An end-to-end regression test (`cross_join_with_conflicting_schema_metadata`) reproducing the exact failure from #23434.
## Are these changes tested?
Yes. Both tests fail on `main` with the errors quoted above and pass with the fix. The e2e's inputs (2-row left, 1-row right) also drive the `JoinSelection` swap path, exercising `swap_inputs`.
## Are there any user-facing changes?
Yes — a bug fix. The output schema **metadata** of a cross join now follows left-wins on conflicting keys (consistent with the logical plan and with every other join operator). Field names, types, nullability and column order are unchanged. Aggregations over cross joins of metadata-carrying tables that previously errored now plan and execute.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: use build_join_schema for CrossJoinExec output schema metadata [datafusion]
Posted by "fangchenli (via GitHub)" <gi...@apache.org>.
fangchenli commented on PR #23485:
URL: https://github.com/apache/datafusion/pull/23485#issuecomment-5080614798
duplicated of #23442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: use build_join_schema for CrossJoinExec output schema metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #23485:
URL: https://github.com/apache/datafusion/pull/23485#issuecomment-5805956443
Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: use build_join_schema for CrossJoinExec output schema metadata [datafusion]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #23485: fix: use build_join_schema for CrossJoinExec output schema metadata
URL: https://github.com/apache/datafusion/pull/23485
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org