You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/04/01 12:50:38 UTC
[PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
alamb opened a new pull request, #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897
## Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/9839
## Rationale for this change
The names of the columns created by callling `struct` changed in https://github.com/apache/arrow-datafusion/pull/9743
### Before https://github.com/apache/arrow-datafusion/pull/9743
```sql
select struct(a, b) from t;
+-----------------+
| struct(t.a,t.b) |
+-----------------+
| {c0: 1, c1: 2} |
| {c0: 3, c1: 4} |
+-----------------+
```
### After https://github.com/apache/arrow-datafusion/pull/9743
(note how the column name is different)
```sql
select struct(a as field_a, b) from t;
+--------------------------------------------------+
| named_struct(Utf8("field_a"),t.a,Utf8("c1"),t.b) |
+--------------------------------------------------+
| {field_a: 1, c1: 2} |
| {field_a: 3, c1: 4} |
+--------------------------------------------------+
```
This change caused https://github.com/apache/arrow-datafusion/issues/9891 as well as removed test coverage for `struct`
## What changes are included in this PR?
1. Use `struct` when there are no named fields
2. reverts the test change in https://github.com/apache/arrow-datafusion/pull/9894
## Are these changes tested?
Yes, by existing tests
## Are there any user-facing changes?
The column names are now the same as they were prior to https://github.com/apache/arrow-datafusion/pull/9894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "yyy1000 (via GitHub)" <gi...@apache.org>.
yyy1000 commented on code in PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897#discussion_r1546642518
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -603,6 +604,25 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
if !fields.is_empty() {
return not_impl_err!("Struct fields are not supported yet");
}
+
+ if values
+ .iter()
+ .any(|value| matches!(value, SQLExpr::Named { .. }))
+ {
+ self.create_named_struct(values, input_schema, planner_context)
Review Comment:
I think the values in `create_named_struct` will be `SQLExpr::Named` in this PR, so maybe we can also make a change here?
https://github.com/apache/arrow-datafusion/blob/d8d521ac8b90002fa0ba1f91456051a9775ae193/datafusion/sql/src/expr/mod.rs#L610-L631
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897#issuecomment-2033120874
Thanks again for the reviews 🙏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "yyy1000 (via GitHub)" <gi...@apache.org>.
yyy1000 commented on code in PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897#discussion_r1546808308
##########
datafusion/sql/src/expr/mod.rs:
##########
@@ -603,6 +604,25 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
if !fields.is_empty() {
return not_impl_err!("Struct fields are not supported yet");
}
+
+ if values
+ .iter()
+ .any(|value| matches!(value, SQLExpr::Named { .. }))
+ {
+ self.create_named_struct(values, input_schema, planner_context)
Review Comment:
Oh, ignore this. One case is only some values are Named so no change is needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897#discussion_r1546289165
##########
datafusion/sqllogictest/test_files/expr.slt:
##########
@@ -2288,39 +2288,39 @@ select struct(time,load1,load2,host) from t1;
# can have an aggregate function with an inner coalesce
query TR
-select t2.info['c3'] as host, sum(coalesce(t2.info)['c1']) from (select struct(time,load1,load2,host) as info from t1) t2 where t2.info['c3'] IS NOT NULL group by t2.info['c3'] order by host;
+select t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] as host, sum(coalesce(t2."struct(t1.time,t1.load1,t1.load2,t1.host)")['c1']) from (select struct(time,load1,load2,host) from t1) t2 where t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] IS NOT NULL group by t2."struct(t1.time,t1.load1,t1.load2,t1.host)"['c3'] order by host;
Review Comment:
This reverts the change from https://github.com/apache/arrow-datafusion/pull/9894
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Use `struct` instead of `named_struct` when there are no aliases [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #9897:
URL: https://github.com/apache/arrow-datafusion/pull/9897#issuecomment-2032762771
> thanks @alamb 1 improvement I can see here is we over iterating through values. I mean 1 iteration is do identify named/not_named and when creating struct/named_struct its another one. prob it can be optimized
I agree it does two iterations over the value. I think the current approach basically requires two iterations because if we are doing `named_struct` we need to add names for each column that don't have a name.
So in other words, `struct(1, 2, 3 as other_col)` needs to become `struct(1 as c1, 2 as c2, 3 as other_col)` but we don't know until we see `3 as other_col` that we need to add in `c1` and `c2`) 🤔
I would like to spend my optimization budget on bigger things initially if that is ok (specifically https://github.com/apache/arrow-datafusion/issues/9637)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org