You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "eejbyfeldt (via GitHub)" <gi...@apache.org> on 2024/09/24 18:51:30 UTC
[PR] Disallow duplicated qualified field names [datafusion]
eejbyfeldt opened a new pull request, #12608:
URL: https://github.com/apache/datafusion/pull/12608
## Which issue does this PR close?
Closes #11464
Closes #11408
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
## Rationale for this change
In https://github.com/apache/datafusion/pull/6091 we relaxed and allowed duplicated qualified fields in order to support duplicated group bys. This leads to some confusion and bugs as we do not allow for unqualified duplicated names. This makes the necessary changes to still support duplicated group by while still disallowing duplicated qualified names.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
Revert allowing duplicated qualified names from #6091
Deduplicate group by exprs when creating output schema for LogicalPlan::Aggregate.
Remove test case `with_column_self_join` as this test no fails already at the join, I don't think it possible to modify to fulfill it old purpose.
The old logic in `plan_table_with_joins` would create schemas with duplicated fields as it would end up with multiple copies for the first table when there are multiple joins. I am not 100% this is the right fix, but it passes all existing test cases.
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
Existing test cases.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
Since this adds new restrictions it possible that it will break downstream users depending on the old behavior.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "eejbyfeldt (via GitHub)" <gi...@apache.org>.
eejbyfeldt commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1774864320
##########
datafusion/sql/src/planner.rs:
##########
@@ -197,9 +197,9 @@ impl PlannerContext {
/// extends the FROM schema, returning the existing one, if any
pub fn extend_outer_from_schema(&mut self, schema: &DFSchemaRef) -> Result<()> {
- self.outer_from_schema = match self.outer_from_schema.as_ref() {
- Some(from_schema) => Some(Arc::new(from_schema.join(schema)?)),
- None => Some(Arc::clone(schema)),
+ match self.outer_from_schema.as_mut() {
+ Some(from_schema) => Arc::make_mut(from_schema).merge(schema),
+ None => self.outer_from_schema = Some(Arc::clone(schema)),
Review Comment:
@aalexandrov Can you help verify that this is correct?
I am not fully following your comment here: https://github.com/apache/datafusion/pull/11456#discussion_r1677146980 At least in the current code merge will still have both values for `j1_id` since they will have differnt qualifiers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#issuecomment-2374911993
we need probably to check how it works if provided the schema and catalog
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1774988724
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -3337,52 +3337,6 @@ mod tests {
Ok(())
}
- // Table 't1' self join
- // Supplementary test of issue: https://github.com/apache/datafusion/issues/7790
Review Comment:
why is this test removed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "eejbyfeldt (via GitHub)" <gi...@apache.org>.
eejbyfeldt commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1775168912
##########
datafusion/core/src/dataframe/mod.rs:
##########
@@ -3337,52 +3337,6 @@ mod tests {
Ok(())
}
- // Table 't1' self join
- // Supplementary test of issue: https://github.com/apache/datafusion/issues/7790
Review Comment:
Should have explained that more in the description. Before it tests that calling `with_column` after a self join without alias will fail in a certain way. After this change it will fail already at the self join. So there is no way to make it test what it testsed before and I belive the self join already has test coverage elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1778541621
##########
datafusion/sql/src/planner.rs:
##########
@@ -197,9 +197,9 @@ impl PlannerContext {
/// extends the FROM schema, returning the existing one, if any
pub fn extend_outer_from_schema(&mut self, schema: &DFSchemaRef) -> Result<()> {
- self.outer_from_schema = match self.outer_from_schema.as_ref() {
- Some(from_schema) => Some(Arc::new(from_schema.join(schema)?)),
- None => Some(Arc::clone(schema)),
+ match self.outer_from_schema.as_mut() {
+ Some(from_schema) => Arc::make_mut(from_schema).merge(schema),
Review Comment:
TIL `Arc::make_mut` 📓
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1775609041
##########
datafusion/expr/src/utils.rs:
##########
@@ -260,7 +252,11 @@ pub fn grouping_set_to_exprlist(group_expr: &[Expr]) -> Result<Vec<&Expr>> {
}
Ok(grouping_set.distinct_expr())
} else {
- Ok(group_expr.iter().collect())
+ Ok(group_expr
+ .iter()
Review Comment:
wondering if can get rid of double collection iteration?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "jonahgao (via GitHub)" <gi...@apache.org>.
jonahgao commented on PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#issuecomment-2372763476
I think this is the right way; DuckDB has a [RemoveDuplicateGroups](https://github.com/duckdb/duckdb/blob/12e9777cf6283f44710b2610ba3d3735a1208751/src/include/duckdb/optimizer/remove_duplicate_groups.hpp#L20C7-L20C28) that does the same thing.
There might be a minor issue with volatile expressions, such as `group by random(), random()`, but I think we can disregard it for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Disallow duplicated qualified field names [datafusion]
Posted by "eejbyfeldt (via GitHub)" <gi...@apache.org>.
eejbyfeldt commented on code in PR #12608:
URL: https://github.com/apache/datafusion/pull/12608#discussion_r1774864320
##########
datafusion/sql/src/planner.rs:
##########
@@ -197,9 +197,9 @@ impl PlannerContext {
/// extends the FROM schema, returning the existing one, if any
pub fn extend_outer_from_schema(&mut self, schema: &DFSchemaRef) -> Result<()> {
- self.outer_from_schema = match self.outer_from_schema.as_ref() {
- Some(from_schema) => Some(Arc::new(from_schema.join(schema)?)),
- None => Some(Arc::clone(schema)),
+ match self.outer_from_schema.as_mut() {
+ Some(from_schema) => Arc::make_mut(from_schema).merge(schema),
+ None => self.outer_from_schema = Some(Arc::clone(schema)),
Review Comment:
@aalexandrov Can you help verify that this is correct?
I am not fully following your comment here: https://github.com/apache/datafusion/pull/11456#discussion_r1677146980 At least in the current code merge will still have both values for `j1_id` since they will have different qualifiers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org