You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mkleen (via GitHub)" <gi...@apache.org> on 2026/01/21 17:07:23 UTC
[PR] Plann recursive subquery cleanup [datafusion]
mkleen opened a new pull request, #19930:
URL: https://github.com/apache/datafusion/pull/19930
## Which issue does this PR close?
Cleaned-up version of https://github.com/apache/datafusion/pull/18806 with:
- Removed outer_queries_schema from PlannerContext
- Planning logic only (optimizer modifications removed)
- sql logic test which rely on the optimizer are moved to sql_integration.rs
## Rationale for this change
See https://github.com/apache/datafusion/pull/18806
## What changes are included in this PR?
See https://github.com/apache/datafusion/pull/18806
## Are these changes tested?
Yes
## Are there any user-facing changes?
`outer_queries_schemas` is removed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "apavlo (via GitHub)" <gi...@apache.org>.
apavlo commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3880567307
@duongcongtoai + @mkleen This is a huge deal. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2777724487
##########
datafusion/sql/src/expr/identifier.rs:
##########
@@ -172,36 +173,44 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
not_impl_err!("compound identifier: {ids:?}")
} else {
// Check the outer_query_schema and try to find a match
- if let Some(outer) = planner_context.outer_query_schema() {
- let search_result = search_dfschema(&ids, outer);
- match search_result {
- // Found matching field with spare identifier(s) for nested field(s) in structure
- Some((field, qualifier, nested_names))
- if !nested_names.is_empty() =>
- {
- // TODO: remove when can support nested identifiers for OuterReferenceColumn
- not_impl_err!(
- "Nested identifiers are not yet supported for OuterReferenceColumn {}",
- Column::from((qualifier, field))
- .quoted_flat_name()
- )
- }
- // Found matching field with no spare identifier(s)
- Some((field, qualifier, _nested_names)) => {
- // Found an exact match on a qualified name in the outer plan schema, so this is an outer reference column
- Ok(Expr::OuterReferenceColumn(
- Arc::clone(field),
- Column::from((qualifier, field)),
- ))
- }
- // Found no matching field, will return a default
- None => {
- let s = &ids[0..ids.len()];
- // safe unwrap as s can never be empty or exceed the bounds
- let (relation, column_name) =
- form_identifier(s).unwrap();
- Ok(Expr::Column(Column::new(relation, column_name)))
- }
+ let outer_schemas = planner_context.outer_queries_schemas();
+ let mut maybe_result = None;
+ if !outer_schemas.is_empty() {
Review Comment:
This is gone and cleaned it up a little bit more.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3858537214
@alamb Thank you for the review. I will add your feedback soon!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3851658315
@duongcongtoai Thank you for your feedback!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713571920
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
+ FROM lineitem l (a, b, c, d, e)";
let plan = logical_plan(sql).unwrap();
assert_snapshot!(
plan,
@r"
- Projection: l.a, l.b, l.c
+ Projection: l.a, l.b, l.c, l.d, l.e
SubqueryAlias: l
- Projection: lineitem.l_item_id AS a, lineitem.l_description AS b, lineitem.price AS c
+ Projection: lineitem.l_orderkey AS a, lineitem.l_item_id AS b, lineitem.l_description AS c, lineitem.l_extendedprice AS d, lineitem.price AS e
Review Comment:
This is because the lineitems is extended now.
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
+ FROM lineitem l (a, b, c, d, e)";
let plan = logical_plan(sql).unwrap();
assert_snapshot!(
plan,
@r"
- Projection: l.a, l.b, l.c
+ Projection: l.a, l.b, l.c, l.d, l.e
SubqueryAlias: l
- Projection: lineitem.l_item_id AS a, lineitem.l_description AS b, lineitem.price AS c
+ Projection: lineitem.l_orderkey AS a, lineitem.l_item_id AS b, lineitem.l_description AS c, lineitem.l_extendedprice AS d, lineitem.price AS e
Review Comment:
This is because the lineitems table is extended now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2771506807
##########
datafusion/sql/src/planner.rs:
##########
@@ -297,19 +300,26 @@ impl PlannerContext {
self
}
- // Return a reference to the outer query's schema
- pub fn outer_query_schema(&self) -> Option<&DFSchema> {
- self.outer_query_schema.as_ref().map(|s| s.as_ref())
+ /// Return the stack of outer relations' schemas, the outer most
+ /// relation are at the first entry
+ pub fn outer_queries_schemas(&self) -> Vec<DFSchemaRef> {
Review Comment:
is there a reason we have to convert this to a `Vec`? As in why not return a reference like
```rust
pub fn outer_queries_schemas(&self) -> &[DFSchemaRef] {
```
That would avoid copying / allocating
##########
datafusion/sql/src/planner.rs:
##########
@@ -261,8 +261,11 @@ pub struct PlannerContext {
/// Map of CTE name to logical plan of the WITH clause.
/// Use `Arc<LogicalPlan>` to allow cheap cloning
ctes: HashMap<String, Arc<LogicalPlan>>,
- /// The query schema of the outer query plan, used to resolve the columns in subquery
- outer_query_schema: Option<DFSchemaRef>,
+
+ /// The queries schemas of outer query relations, used to resolve the outer referenced
Review Comment:
perfect
##########
datafusion/sql/src/planner.rs:
##########
@@ -297,19 +300,26 @@ impl PlannerContext {
self
}
- // Return a reference to the outer query's schema
- pub fn outer_query_schema(&self) -> Option<&DFSchema> {
- self.outer_query_schema.as_ref().map(|s| s.as_ref())
+ /// Return the stack of outer relations' schemas, the outer most
+ /// relation are at the first entry
+ pub fn outer_queries_schemas(&self) -> Vec<DFSchemaRef> {
+ self.outer_queries_schemas_stack.to_vec()
}
/// Sets the outer query schema, returning the existing one, if
/// any
- pub fn set_outer_query_schema(
- &mut self,
- mut schema: Option<DFSchemaRef>,
- ) -> Option<DFSchemaRef> {
- std::mem::swap(&mut self.outer_query_schema, &mut schema);
- schema
+ pub fn append_outer_query_schema(&mut self, schema: DFSchemaRef) {
+ self.outer_queries_schemas_stack.push(schema);
+ }
+
+ /// The schema of the adjacent outer relation
+ pub fn latest_outer_query_schema(&mut self) -> Option<DFSchemaRef> {
Review Comment:
Likewise here I recommend returning the reference instead
```rust
pub fn latest_outer_query_schema(&mut self) -> Option<&DFSchemaRef> {
```
That way the callsite can decide if it needs a clone or not - this API forces a clone always (which admittedly is just an Arc::clone, but it would be nice to avoid if possible)
##########
datafusion/sql/src/expr/identifier.rs:
##########
@@ -172,36 +173,44 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
not_impl_err!("compound identifier: {ids:?}")
} else {
// Check the outer_query_schema and try to find a match
- if let Some(outer) = planner_context.outer_query_schema() {
- let search_result = search_dfschema(&ids, outer);
- match search_result {
- // Found matching field with spare identifier(s) for nested field(s) in structure
- Some((field, qualifier, nested_names))
- if !nested_names.is_empty() =>
- {
- // TODO: remove when can support nested identifiers for OuterReferenceColumn
- not_impl_err!(
- "Nested identifiers are not yet supported for OuterReferenceColumn {}",
- Column::from((qualifier, field))
- .quoted_flat_name()
- )
- }
- // Found matching field with no spare identifier(s)
- Some((field, qualifier, _nested_names)) => {
- // Found an exact match on a qualified name in the outer plan schema, so this is an outer reference column
- Ok(Expr::OuterReferenceColumn(
- Arc::clone(field),
- Column::from((qualifier, field)),
- ))
- }
- // Found no matching field, will return a default
- None => {
- let s = &ids[0..ids.len()];
- // safe unwrap as s can never be empty or exceed the bounds
- let (relation, column_name) =
- form_identifier(s).unwrap();
- Ok(Expr::Column(Column::new(relation, column_name)))
- }
+ let outer_schemas = planner_context.outer_queries_schemas();
+ let mut maybe_result = None;
+ if !outer_schemas.is_empty() {
Review Comment:
Do we really need this check? I think you could reduce the indent level (and not change the meaning) by just leaving the loop (which will not run if the list is empty)
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
Review Comment:
I agree we should have some sqllogictest coverage to make sure everything is hooked up right
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3867321182
I took the liberty of pushing a few more tests, and some small API cleanups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3865155312
@alamb @duongcongtoai Could you please do one more review?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713580143
##########
datafusion/sql/tests/common/mod.rs:
##########
@@ -161,12 +161,26 @@ impl ContextProvider for MockContextProvider {
])),
"orders" => Ok(Schema::new(vec![
Field::new("order_id", DataType::UInt32, false),
+ Field::new("o_orderkey", DataType::UInt32, false),
Review Comment:
These extension were necessary to get the tests to work.
##########
datafusion/sql/tests/common/mod.rs:
##########
@@ -161,12 +161,26 @@ impl ContextProvider for MockContextProvider {
])),
"orders" => Ok(Schema::new(vec![
Field::new("order_id", DataType::UInt32, false),
+ Field::new("o_orderkey", DataType::UInt32, false),
Review Comment:
These extensions were necessary to get the tests to work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "duongcongtoai (via GitHub)" <gi...@apache.org>.
duongcongtoai commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3808721227
nice, i'll take a look once i have time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713577678
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -1017,7 +1017,7 @@ fn table_with_column_alias_number_cols() {
assert_snapshot!(
err.strip_backtrace(),
- @"Error during planning: Source table contains 3 columns but only 2 names given as column alias"
+ @"Error during planning: Source table contains 5 columns but only 2 names given as column alias"
Review Comment:
This is because the lineitem table is extended now.
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
+ FROM lineitem l (a, b, c, d, e)";
let plan = logical_plan(sql).unwrap();
assert_snapshot!(
plan,
@r"
- Projection: l.a, l.b, l.c
+ Projection: l.a, l.b, l.c, l.d, l.e
SubqueryAlias: l
- Projection: lineitem.l_item_id AS a, lineitem.l_description AS b, lineitem.price AS c
+ Projection: lineitem.l_orderkey AS a, lineitem.l_item_id AS b, lineitem.l_description AS c, lineitem.l_extendedprice AS d, lineitem.price AS e
Review Comment:
This is because the lineitem table is extended now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "duongcongtoai (via GitHub)" <gi...@apache.org>.
duongcongtoai commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2766657985
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
Review Comment:
would be nice to have a sqllogictest for this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "duongcongtoai (via GitHub)" <gi...@apache.org>.
duongcongtoai commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2766666967
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
Review Comment:
ah i see you replaced them with integration test instead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #19930:
URL: https://github.com/apache/datafusion/pull/19930
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2777720721
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
Review Comment:
I added two sqllogictests back which pass without the optimizer changes, the other two remain in sql_integration.rs because they depend on the optimizer change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2779263373
##########
datafusion/sql/src/expr/identifier.rs:
##########
@@ -76,15 +76,16 @@ impl<S: ContextProvider> SqlToRel<'_, S> {
}
// Check the outer query schema
- if let Some(outer) = planner_context.outer_query_schema()
- && let Ok((qualifier, field)) =
+ for outer in planner_context.outer_queries_schemas() {
Review Comment:
Upon a second look at this code, I think this is resolving the names in the wrong order (starts at outermost query, but really should start at the innermost) -- something like `outer in planner_context.outer_queries_schemas().iter().rev()`
I'll try and come up with a reproducer for this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713583317
##########
datafusion/sql/src/planner.rs:
##########
@@ -282,7 +285,7 @@ impl PlannerContext {
Self {
prepare_param_data_types: Arc::new(vec![]),
ctes: HashMap::new(),
- outer_query_schema: None,
Review Comment:
`outer_query_schema` is removed now, everything is handled over the `outer_queries_schemas_stack`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713574192
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -4801,7 +4801,11 @@ fn test_using_join_wildcard_schema() {
// Only columns from one join side should be present
let expected_fields = vec![
"o1.order_id".to_string(),
+ "o1.o_orderkey".to_string(),
+ "o1.o_custkey".to_string(),
+ "o1.o_orderstatus".to_string(),
"o1.customer_id".to_string(),
+ "o1.o_totalprice".to_string(),
Review Comment:
This is because the order table is extended now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3798484669
@alamb @duongcongtoai Could you review this when you have a moment so we can get it merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2777720721
##########
datafusion/sql/tests/sql_integration.rs:
##########
@@ -995,15 +995,15 @@ fn select_nested_with_filters() {
#[test]
fn table_with_column_alias() {
- let sql = "SELECT a, b, c
- FROM lineitem l (a, b, c)";
+ let sql = "SELECT a, b, c, d, e
Review Comment:
I added two sqllogictest test back which pass without the optimizer changes, the other two remain in sql_integration.rs because they depend on the optimizer change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3860394855
Thanks @mkleen -- sorry for the delay in reviewing this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations Cleanup [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2713583317
##########
datafusion/sql/src/planner.rs:
##########
@@ -282,7 +285,7 @@ impl PlannerContext {
Self {
prepare_param_data_types: Arc::new(vec![]),
ctes: HashMap::new(),
- outer_query_schema: None,
Review Comment:
`outer_query_schema` is removed now, everything is handled over the `outer_queries_schemas_stack`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "mkleen (via GitHub)" <gi...@apache.org>.
mkleen commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2777720872
##########
datafusion/expr/src/logical_plan/invariants.rs:
##########
@@ -208,14 +208,16 @@ pub fn check_subquery_expr(
if group_expr.contains(expr) && !aggr_expr.contains(expr) {
// TODO revisit this validation logic
plan_err!(
- "Correlated scalar subquery in the GROUP BY clause must also be in the aggregate expressions"
+ "Correlated scalar subquery in the GROUP BY clause must \
Review Comment:
This is from fmt.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "duongcongtoai (via GitHub)" <gi...@apache.org>.
duongcongtoai commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2766670487
##########
datafusion/expr/src/logical_plan/invariants.rs:
##########
@@ -208,14 +208,16 @@ pub fn check_subquery_expr(
if group_expr.contains(expr) && !aggr_expr.contains(expr) {
// TODO revisit this validation logic
plan_err!(
- "Correlated scalar subquery in the GROUP BY clause must also be in the aggregate expressions"
+ "Correlated scalar subquery in the GROUP BY clause must \
Review Comment:
nit: this change is unrelated (it's in my original PR)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#discussion_r2779330727
##########
datafusion/sql/tests/common/mod.rs:
##########
@@ -161,12 +161,26 @@ impl ContextProvider for MockContextProvider {
])),
"orders" => Ok(Schema::new(vec![
Field::new("order_id", DataType::UInt32, false),
+ Field::new("o_orderkey", DataType::UInt32, false),
+ Field::new("o_custkey", DataType::UInt32, false),
+ Field::new("o_orderstatus", DataType::Utf8, false),
Field::new("customer_id", DataType::UInt32, false),
+ Field::new("o_totalprice", DataType::Decimal32(15, 2), false),
Field::new("o_item_id", DataType::Utf8, false),
Field::new("qty", DataType::Int32, false),
Field::new("price", DataType::Float64, false),
Field::new("delivered", DataType::Boolean, false),
])),
+ "customer" => Ok(Schema::new(vec![
+ Field::new("c_custkey", DataType::UInt32, false),
+ Field::new("c_name", DataType::Utf8, false),
+ Field::new("c_address", DataType::Utf8, false),
+ Field::new("c_nationkey", DataType::UInt32, false),
+ Field::new("c_phone", DataType::Decimal32(15, 2), false),
Review Comment:
c_phone is utf8 -- https://github.com/apache/datafusion/blob/badaa8400d3a5bf4f27496dd377a2c3a2286908f/benchmarks/src/tpch/mod.rs#L86
##########
datafusion/sql/tests/common/mod.rs:
##########
@@ -161,12 +161,26 @@ impl ContextProvider for MockContextProvider {
])),
"orders" => Ok(Schema::new(vec![
Field::new("order_id", DataType::UInt32, false),
+ Field::new("o_orderkey", DataType::UInt32, false),
+ Field::new("o_custkey", DataType::UInt32, false),
+ Field::new("o_orderstatus", DataType::Utf8, false),
Field::new("customer_id", DataType::UInt32, false),
+ Field::new("o_totalprice", DataType::Decimal32(15, 2), false),
Review Comment:
Decimal32 is not consistent with the other schemas which use Decimal 128 https://github.com/apache/datafusion/blob/main/benchmarks/src/tpch/mod.rs#L81-L110
I'll fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] feat: Support planning subqueries with OuterReferenceColumn belongs to non-adjacent outer relations [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #19930:
URL: https://github.com/apache/datafusion/pull/19930#issuecomment-3871853409
Thanks again @duongcongtoai and @mkleen for all your help 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org