You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jonahgao (via GitHub)" <gi...@apache.org> on 2024/07/22 14:11:56 UTC
[I] Propagation of ordered `SortProperties` should consider `nulls_first` [datafusion]
jonahgao opened a new issue, #11596:
URL: https://github.com/apache/datafusion/issues/11596
### Describe the bug
As pointed by @2010YOUY01 in https://github.com/apache/datafusion/pull/11571#discussion_r1685702554
For binary expressions or multi-input functions , if their args are ordered but the `nulls_first` option in `SortOptions` differs, they are still very likely to be unordered.
### To Reproduce
Need to examine the code.
- The methods of `SortProperties`, such as [add](https://github.com/apache/datafusion/blob/81d06f2e103385fe744fb909563d4fb4c4b13d49/datafusion/expr/src/sort_properties.rs#L48), `sub`, seem to not consider nulls_first.
- Similar to [log.rs](https://github.com/apache/datafusion/blob/81d06f2e103385fe744fb909563d4fb4c4b13d49/datafusion/functions/src/math/log.rs#L84) (should be fixed in PR 11571), the implementation of `ScalarUDFImpl::output_ordering` also needs to be reviewed.
### Expected behavior
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Propagation of ordered `SortProperties` should consider `nulls_first` [datafusion]
Posted by "Nagato-Yuzuru (via GitHub)" <gi...@apache.org>.
Nagato-Yuzuru commented on issue #11596:
URL: https://github.com/apache/datafusion/issues/11596#issuecomment-5232044222
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Propagation of ordered `SortProperties` should consider `nulls_first` [datafusion]
Posted by "Nagato-Yuzuru (via GitHub)" <gi...@apache.org>.
Nagato-Yuzuru commented on issue #11596:
URL: https://github.com/apache/datafusion/issues/11596#issuecomment-5232788766
I discovered `and_or` is also incorrect here. DataFusion's three-valued bool `nulls_first` consistency check is not sound.
NULL does not always propagate. `null AND false = false`.
```
DataFusion CLI v54.1.0
> WITH v(x) AS (VALUES (true), (false), (NULL))
SELECT a.x AS a,
b.x AS b,
a.x AND b.x AS and_,
a.x OR b.x AS or_
FROM v a, v b
ORDER BY 1 NULLS LAST, 2 NULLS LAST;
+-------+-------+-------+-------+
| a | b | and_ | or_ |
+-------+-------+-------+-------+
| false | false | false | false |
| false | true | false | true |
| false | NULL | false | NULL |
| true | false | false | true |
| true | true | true | true |
| true | NULL | NULL | true |
| NULL | false | false | NULL |
| NULL | true | NULL | true |
| NULL | NULL | NULL | NULL |
+-------+-------+-------+-------+
9 row(s) fetched.
```
Counterexample:
`a = [null, null, false, true]`, `b = [false, true, true, true]`, both ASC NULLS FIRST. `a AND b = [false, null, false, true]` not sorted.
Will fix in the same PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Propagation of ordered `SortProperties` should consider `nulls_first` [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #11596: Propagation of ordered `SortProperties` should consider `nulls_first`
URL: https://github.com/apache/datafusion/issues/11596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org