You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/27 18:15:36 UTC
[GitHub] [arrow-datafusion] alamb opened a new issue #1184: Null constants can not be used in most expressions
alamb opened a new issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I want to be able to use null constants in my datafusion queries (e.g. `NULL < 5`). While at first this seems a silly usecase, we end up with such predicates during automated rewrites in IOx (where sometimes we replace column references with null constants -- See more details on https://github.com/influxdata/influxdb_iox/issues/883)
While they are allowed in the SQL or expr language (e.g. `Expr::Literal(ScalarValue::Utf8(None)))` they typically generate some runtime error before execution
**Describe the solution you'd like**
I want to be able to write expressions like `NULL < 5` in all cases where I could write `col < 5` and have DataFusion evaluate to the correct thing.
The proper fix for this ticket is likely first class support for `DataType::Null` and then proper coercion logic in the codebase as required. See the discussion by @Dandandan and @Jimexist on #1179 at https://github.com/apache/arrow-datafusion/issues/1179#issuecomment-952997737
**Task List***
- [ ] Add cast support in arrow-rs to/from NULL to other types
- [ ] https://github.com/apache/arrow-datafusion/issues/1179
- [ ] Handle null literals in function calls
- [ ] Handle null literals in CASE statements
- [ ] Handle null literals in IN LISTS
- [ ] TODO review remaining expression types
**Additional context**
This is a tracking ticket for a variety of null incorrectness issues I found while working on https://github.com/apache/arrow-datafusion/issues/1179
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-953986386
@alamb have you compared the two different proposals that @Dandandan and I put?
1. to have one untyped enum case `Null`
2. to have one typed enum case `Null(DataType)` and remove the `Option` wrapping for all other enum cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-953871849
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] WinkerDu edited a comment on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
WinkerDu edited a comment on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-1086871247
@alamb Maybe I can pick up some tickets as appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] WinkerDu commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
WinkerDu commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-1086871247
Maybe I can pick up some tickets as appropriate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-954788593
I have a draft up at https://github.com/apache/arrow-datafusion/pull/1199 please take an initial look @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Jimexist commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
Jimexist commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-954018205
I can take a stab at approach 2 and see how much effort it would take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #1184: Null constants can not be used in most expressions
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #1184:
URL: https://github.com/apache/arrow-datafusion/issues/1184#issuecomment-954029887
> @alamb have you compared the two different proposals that @Dandandan and I put?
> to have one untyped enum case Null
> to have one typed enum case Null(DataType) and remove the Option wrapping for all other enum cases
@Jimexist I have not fully worked through the implications of either approach.
One thing I suspect will occur is that the code in GroupByHash (and elsewhere) really wants to know what "type" the ScalarValue is even if it is null -- because rust is strongly typed.
A third option might be to leave the representation of `ScalarValue` as it is and instead return `DataType::Null` if it represents a null value.
Another challenge is that there really is a difference between a `NullArray` and an `Int8Array` of all nulls (for example). And both are valid concepts
🤔
Thus I conclude that trying out an approach with some sort of technical spike sounds like a great idea to me 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org