You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kontinuation (via GitHub)" <gi...@apache.org> on 2025/09/11 16:37:31 UTC
[PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Kontinuation opened a new pull request, #17524:
URL: https://github.com/apache/datafusion/pull/17524
## Which issue does this PR close?
- Closes #17422.
## Rationale for this change
As reported by #17422, extension metadata was dropped in some queries involving subqueries. This happens when a column references another column in a correlated subquery, which creates an `OuterReferenceColumn` in DataFusion's logical plan.
`OuterReferenceColumn` is defined to contain a data type and a referenced column. As we know, outer reference column refers to a field in another table, so there's no way to retrieve the field metadata of referenced column given the schema of the current table. To overcome this problem, we define `OuterReferenceColumn` to contain the entire referenced field as a `FieldRef`. We also updated the implementation of `to_field` to return the referenced field, which contains the original metadata of referenced field.
## What changes are included in this PR?
Changed `OuterReferenceColumn` to contain the referenced outer field as a `FieldRef`.
## Are these changes tested?
* Added a unit test
* Added the repro in the issue to `user_defined_scalar_functions` test
## Are there any user-facing changes?
Yes. This changes part of the public API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #17524:
URL: https://github.com/apache/datafusion/pull/17524
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17524:
URL: https://github.com/apache/datafusion/pull/17524#discussion_r2352753625
##########
datafusion/expr/src/expr_fn.rs:
##########
@@ -71,7 +71,15 @@ pub fn col(ident: impl Into<Column>) -> Expr {
/// Create an out reference column which hold a reference that has been resolved to a field
/// outside of the current plan.
pub fn out_ref_col(dt: DataType, ident: impl Into<Column>) -> Expr {
- Expr::OuterReferenceColumn(dt, ident.into())
+ // Construct a synthetic field (no metadata). Prefer using `out_ref_field` when possible
+ let column = ident.into();
Review Comment:
Rather than removing it, how about we mark it "deprecated" per https://datafusion.apache.org/contributor-guide/api-health.html#deprecation-guidelines?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #17524:
URL: https://github.com/apache/datafusion/pull/17524#issuecomment-3303991034
let's try it this way, and if we find other bugs related to using the wrong `out_ref_col` we can reasses
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #17524:
URL: https://github.com/apache/datafusion/pull/17524#issuecomment-3286052629
CC @paleolimbot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "Kontinuation (via GitHub)" <gi...@apache.org>.
Kontinuation commented on PR #17524:
URL: https://github.com/apache/datafusion/pull/17524#issuecomment-3301076013
I have marked `out_ref_col` as deprecated. I also found that the new API `out_ref_field` is quite cumbersome, so I replaced it with `out_ref_col_with_metadata`. This will also make transitioning from the deprecated API to the new API more easier (only need to pass in one empty HashMap).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: Change `OuterReferenceColumn` to contain the entire outer field to prevent metadata loss [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #17524:
URL: https://github.com/apache/datafusion/pull/17524#discussion_r2352750552
##########
datafusion/expr/src/expr.rs:
##########
@@ -3951,7 +3951,7 @@ mod test {
// If this test fails when you change `Expr`, please try
// `Box`ing the fields to make `Expr` smaller
// See https://github.com/apache/datafusion/issues/16199 for details
- assert_eq!(size_of::<Expr>(), 128);
+ assert!(size_of::<Expr>() <= 128);
Review Comment:
I think it would be nicer to include the new, smaller size, in order to prevent regressions (from the smaller 112 bytes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org