You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/07 22:53:19 UTC
[GitHub] [arrow-datafusion] maxburke opened a new pull request #1943: Upstream implementation of typed query variables
maxburke opened a new pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943
Closes #1942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on a change in pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 commented on a change in pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#discussion_r821243638
##########
File path: datafusion-expr/src/expr.rs
##########
@@ -399,7 +399,7 @@ impl fmt::Debug for Expr {
match self {
Expr::Alias(expr, alias) => write!(f, "{:?} AS {}", expr, alias),
Expr::Column(c) => write!(f, "{}", c),
- Expr::ScalarVariable(var_names) => write!(f, "{}", var_names.join(".")),
+ Expr::ScalarVariable(_, var_names) => write!(f, "{}", var_names.join(".")),
Review comment:
It'll be better to print the data type, such as
`Expr::ScalarVariable(data_type, var_names) => write!(f, "{}, data type: {}", var_names.join("."), data_type),`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] maxburke commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
maxburke commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061345206
> Thanks @maxburke -- Could you please add some tests?
I thought about that but it looks like the existing tests for the user variables already cover it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1062122219
Thank you @maxburke . I agree with @xudong963 that all this PR really needs are some tests and it would be good to go 👍
Maybe you can adapt / update the ones in https://github.com/apache/arrow-datafusion/blob/master/datafusion/src/execution/context.rs#L1286-L1313 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a change in pull request #1943: Allow different types of query variables (`@@var`) rather than just string
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#discussion_r825287647
##########
File path: datafusion/src/execution/context.rs
##########
@@ -1300,14 +1317,15 @@ mod tests {
ctx.register_table("dual", provider)?;
let results =
- plan_and_collect(&mut ctx, "SELECT @@version, @name FROM dual").await?;
+ plan_and_collect(&mut ctx, "SELECT @@version, @name, @integer + 1 FROM dual")
Review comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #1943: Allow different types of query variables (`@@var`) rather than just string
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061344803
BTW, ballista indirectly uses `ScalarVariable`, so you need to process it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] maxburke commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
maxburke commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061423134
> How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved
Could you elaborate more on what kinds of issues you are expecting?
The crux of this change is that, previously, the types of all `StaticVariable` type expressions was hardcoded to be `DataType::Utf8`; what this change does is add a way of having the variable provider specify the actual type of the value. It does not do any type conversions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061499463
> > How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved
>
> Could you elaborate more on what kinds of issues you are expecting?
>
> The crux of this change is that, previously, the types of all `StaticVariable` type expressions was hardcoded to be `DataType::Utf8`; what this change does is add a way of having the variable provider specify the actual type of the value. It does not do any type conversions.
Sorry for the confusion. I went through the original code and the changes made to this ticket. I found the changes will eventually generate `Expr::ScalarVariable` with data type in the logical plan phase. But In the physical phase, the ticket doesn't do anything with the data type. So I'm confused what problem does this ticket actually solve?
I would appreciate it if you could give an example usage about the ticket to help me understand and learn ❤️, thanks @maxburke
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] maxburke commented on a change in pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
maxburke commented on a change in pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#discussion_r821272783
##########
File path: datafusion-expr/src/expr.rs
##########
@@ -399,7 +399,7 @@ impl fmt::Debug for Expr {
match self {
Expr::Alias(expr, alias) => write!(f, "{:?} AS {}", expr, alias),
Expr::Column(c) => write!(f, "{}", c),
- Expr::ScalarVariable(var_names) => write!(f, "{}", var_names.join(".")),
+ Expr::ScalarVariable(_, var_names) => write!(f, "{}", var_names.join(".")),
Review comment:
I avoided this because it looks like the Debug implementation for Expr tries to print SQL-compatible output, but, for example, "@v0, data type: DataType::Utf8" wouldn't be SQL-like
##########
File path: datafusion-expr/src/expr.rs
##########
@@ -399,7 +399,7 @@ impl fmt::Debug for Expr {
match self {
Expr::Alias(expr, alias) => write!(f, "{:?} AS {}", expr, alias),
Expr::Column(c) => write!(f, "{}", c),
- Expr::ScalarVariable(var_names) => write!(f, "{}", var_names.join(".")),
+ Expr::ScalarVariable(_, var_names) => write!(f, "{}", var_names.join(".")),
Review comment:
I avoided this because it looks like the Debug implementation for Expr tries to print SQL-compatible output, but, for example, "@v0, data type: DataType::Utf8" wouldn't be SQL-like
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on a change in pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 commented on a change in pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#discussion_r821290635
##########
File path: datafusion-expr/src/expr.rs
##########
@@ -399,7 +399,7 @@ impl fmt::Debug for Expr {
match self {
Expr::Alias(expr, alias) => write!(f, "{:?} AS {}", expr, alias),
Expr::Column(c) => write!(f, "{}", c),
- Expr::ScalarVariable(var_names) => write!(f, "{}", var_names.join(".")),
+ Expr::ScalarVariable(_, var_names) => write!(f, "{}", var_names.join(".")),
Review comment:
> it looks like the Debug implementation for Expr tries to print SQL-compatible output
It's a good point, makes sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 edited a comment on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 edited a comment on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061376989
> > Thanks @maxburke -- Could you please add some tests?
>
> I thought about that but it looks like the existing tests for the user variables already cover it?
Are you referring to here?
https://github.com/apache/arrow-datafusion/blob/d468177687a596721c0a2af61f395f14c41942cd/datafusion/src/execution/context.rs#L1327
~How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] maxburke commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
maxburke commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1064397375
Test added / extended!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] xudong963 commented on pull request #1943: Upstream implementation of typed query variables
Posted by GitBox <gi...@apache.org>.
xudong963 commented on pull request #1943:
URL: https://github.com/apache/arrow-datafusion/pull/1943#issuecomment-1061376989
> > Thanks @maxburke -- Could you please add some tests?
>
> I thought about that but it looks like the existing tests for the user variables already cover it?
Are you referring to here?
https://github.com/apache/arrow-datafusion/blob/d468177687a596721c0a2af61f395f14c41942cd/datafusion/src/execution/context.rs#L1327
How about trying to test the other type and check the type? I have a vague feeling that there will be some problems of type conversion involved
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org