You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "y-f-u (via GitHub)" <gi...@apache.org> on 2024/07/17 06:12:27 UTC
[PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
y-f-u opened a new pull request, #11505:
URL: https://github.com/apache/datafusion/pull/11505
## Which issue does this PR close?
Unparser creates invalid sqls for LogicPlan that generated from sql with derived table with columns.
## Rationale for this change
See "Which issue does this PR close", also check below comment.
Copy/pasted from the code comments.
```
// A roundtrip example for table alias with columns
//
// query: SELECT id FROM (SELECT j1_id from j1) AS c (id)
//
// LogicPlan:
// Projection: c.id
// SubqueryAlias: c
// Projection: j1.j1_id AS id
// Projection: j1.j1_id
// TableScan: j1
//
// Before introducing this logic, the unparsed query would be `SELECT c.id FROM (SELECT j1.j1_id AS
// id FROM (SELECT j1.j1_id FROM j1)) AS c`.
// The query is invalid as `j1.j1_id` is not a valid identifier in the derived table
// `(SELECT j1.j1_id FROM j1)`
```
## What changes are included in this PR?
Introduce a method that probe the SubqueryAlias plan to see if it contains a double layer of projections. If so, extract column alias from the outer layer projection into table alias and use the inner layer projection for relation construction.
## Are these changes tested?
Yes
## Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
Posted by "y-f-u (via GitHub)" <gi...@apache.org>.
y-f-u commented on code in PR #11505:
URL: https://github.com/apache/datafusion/pull/11505#discussion_r1683736161
##########
datafusion/sql/tests/cases/plan_to_sql.rs:
##########
@@ -240,6 +240,35 @@ fn roundtrip_statement_with_dialect() -> Result<()> {
parser_dialect: Box::new(GenericDialect {}),
unparser_dialect: Box::new(UnparserDefaultDialect {}),
},
+ // more tests around subquery/derived table roundtrip
+ TestStatementWithDialect {
+ sql: "SELECT string_count FROM (
+ SELECT
Review Comment:
Thanks for the suggestion. This query is actually not working as "CAST" will be presented in the inner projection and the projections matching actually breaks.
Maybe another PR to fix this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11505:
URL: https://github.com/apache/datafusion/pull/11505#discussion_r1684920814
##########
datafusion/sql/tests/cases/plan_to_sql.rs:
##########
@@ -240,6 +240,35 @@ fn roundtrip_statement_with_dialect() -> Result<()> {
parser_dialect: Box::new(GenericDialect {}),
unparser_dialect: Box::new(UnparserDefaultDialect {}),
},
+ // more tests around subquery/derived table roundtrip
+ TestStatementWithDialect {
+ sql: "SELECT string_count FROM (
+ SELECT
Review Comment:
It might also be worth filing an issue to track the defect
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #11505:
URL: https://github.com/apache/datafusion/pull/11505#discussion_r1683380691
##########
datafusion/sql/src/unparser/plan.rs:
##########
@@ -611,6 +607,67 @@ impl Unparser<'_> {
}
}
+// This logic is to work out the columns and inner query for SubqueryAlias plan for both types of
+// subquery
+// - `(SELECT column_a as a from table) AS A`
+// - `(SELECT column_a from table) AS A (a)`
+//
+// A roundtrip example for table alias with columns
+//
+// query: SELECT id FROM (SELECT j1_id from j1) AS c (id)
+//
+// LogicPlan:
+// Projection: c.id
+// SubqueryAlias: c
+// Projection: j1.j1_id AS id
+// Projection: j1.j1_id
+// TableScan: j1
+//
+// Before introducing this logic, the unparsed query would be `SELECT c.id FROM (SELECT j1.j1_id AS
+// id FROM (SELECT j1.j1_id FROM j1)) AS c`.
+// The query is invalid as `j1.j1_id` is not a valid identifier in the derived table
+// `(SELECT j1.j1_id FROM j1)`
+//
+// With this logic, the unparsed query will be:
+// `SELECT c.id FROM (SELECT j1.j1_id FROM j1) AS c (id)`
+//
+// Caveat: this won't handle the case like `select * from (select 1, 2) AS a (b, c)`
+// as the parser gives a wrong plan which has mismatch `Int(1)` types: Literal and
+// Column in the Projections. Once the parser side is fixed, this logic should work
Review Comment:
Thank you for the comments
##########
datafusion/sql/tests/cases/plan_to_sql.rs:
##########
@@ -240,6 +240,35 @@ fn roundtrip_statement_with_dialect() -> Result<()> {
parser_dialect: Box::new(GenericDialect {}),
unparser_dialect: Box::new(UnparserDefaultDialect {}),
},
+ // more tests around subquery/derived table roundtrip
+ TestStatementWithDialect {
+ sql: "SELECT string_count FROM (
+ SELECT
Review Comment:
I recommend you add a query that has an additional test that selects more than one column and an expression
something like
```sql
select string_count * 100, id FROM ...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #11505:
URL: https://github.com/apache/datafusion/pull/11505#issuecomment-2240024758
Thanks again @y-f-u
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] fix: unparser generates wrong sql for derived table with columns [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #11505:
URL: https://github.com/apache/datafusion/pull/11505
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org