You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2023/11/04 10:26:28 UTC
[PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
alamb opened a new pull request, #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052
## Which issue does this PR close?
N/A
## Rationale for this change
The question came up on ASF Slack about what `Expr::IsNotFalse(e)` meant, and since it is subtle I had to look it up in the code, and thus I wanted to save myself (and future people) the trouble of doing that again
## What changes are included in this PR?
Update documentation about the various operator.
## Are these changes tested?
doctests
## Are there any user-facing changes?
Just Docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#discussion_r1382561939
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
I think saying these expressions result in NULL is accurate. There is indeed some subtlety related to using expressions that evaluate to `NULL` (booleans) for filters -- where the value does not pass the filter if it evaluates to `false` or `NULL` (it only passes if it is true). 🤯
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
I think saying these expressions result in NULL is accurate. There is indeed some subtlety related to using expressions that evaluate to `NULL` (booleans) for filters -- where the value does not pass the filter if it evaluates to `false` or `NULL` (it only passes if it is true). 🤯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#discussion_r1382436731
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
This might be a little bit misleading, prob we need to add what is unknown ....
https://en.wikipedia.org/wiki/Boolean_data_type#SQL
```
NULL BOOLEAN and UNKNOWN "may be used interchangeably to mean exactly the same thing"
```
so here the user can think that unknown the same as null, but its boolean null, not sure if it matters tbh
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#issuecomment-1794692558
Thanks again @comphead
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb merged PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#discussion_r1383763493
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
Here's a fun example of that subtlety: https://www.db-fiddle.com/f/xbHAYw59p4tgtR1L39p84x/0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "comphead (via GitHub)" <gi...@apache.org>.
comphead commented on code in PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#discussion_r1382436291
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
Review Comment:
👍
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
Review Comment:
👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#discussion_r1383798718
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
`NOT IN` (aka what are the semantics of an Anti join with NULL). 🤯 indeed
##########
datafusion/expr/src/expr.rs:
##########
@@ -99,21 +99,21 @@ pub enum Expr {
SimilarTo(Like),
/// Negation of an expression. The expression's type must be a boolean to make sense.
Not(Box<Expr>),
- /// Whether an expression is not Null. This expression is never null.
+ /// True if argument is not NULL, false otherwise. This expression itself is never NULL.
IsNotNull(Box<Expr>),
- /// Whether an expression is Null. This expression is never null.
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
IsNull(Box<Expr>),
- /// Whether an expression is True. Boolean operation
+ /// True if argument is true, false otherwise. This expression itself is never NULL.
IsTrue(Box<Expr>),
- /// Whether an expression is False. Boolean operation
+ /// True if argument is false, false otherwise. This expression itself is never NULL.
IsFalse(Box<Expr>),
- /// Whether an expression is Unknown. Boolean operation
+ /// True if argument is NULL, false otherwise. This expression itself is never NULL.
Review Comment:
`NOT IN` (aka what are the semantics of an Anti join with NULL). 🤯 indeed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Minor: improve documentation for IsNotNull, DISTINCT, etc [arrow-datafusion]
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on PR #8052:
URL: https://github.com/apache/arrow-datafusion/pull/8052#issuecomment-1795738666
This is great. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org