You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "edmondop (via GitHub)" <gi...@apache.org> on 2024/07/28 12:57:28 UTC
[PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
edmondop opened a new pull request, #11696:
URL: https://github.com/apache/datafusion/pull/11696
This PR tries to extract some work necessary for #10943. Min and Max are special functions that are used for statistics and optimizations, and they need a special treatment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1694856368
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Here is the code that we need the boolean, maybe name it with `is_descending: Option<bool>`?
```rust
let (field, desc) = aggr.get_minmax_desc()?;
if desc != order.options.descending {
return None;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1694856368
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Here is the code that we need the boolean, maybe name it with `is_descending: Option<bool>`?
```rust
let (field, desc) = aggr.get_minmax_desc()?;
if desc != order.options.descending {
return None;
}
```
And, we can get fields separately
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "Rafferty97 (via GitHub)" <gi...@apache.org>.
Rafferty97 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1694406954
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Can I suggest using an `enum` like `enum MinMax { Min, Max }` instead of a `bool`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #11696:
URL: https://github.com/apache/datafusion/pull/11696
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "edmondop (via GitHub)" <gi...@apache.org>.
edmondop commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696769348
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Thanks. The goal of this PR was to simplify #10943 by implementing here the necessary method for udaf, do you think I should proceed in the refactoring now or can we do as a separate improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#issuecomment-2258238514
Thanks @edmondop
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "edmondop (via GitHub)" <gi...@apache.org>.
edmondop commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696124455
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Fixed with [`eb0ebe0` (#11696)](https://github.com/apache/datafusion/pull/11696/commits/eb0ebe04fc92baec527ffd945b8801f72ffecf48) although I am not sure I understood what "is descending" mean for a function?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696231200
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
let field = aggr.field()
let is_sorted_descending = aggr.is_sorted_descending()
And, I think getting field and descending separately is better than `get_minmax_desc`, so if we can get `field` independently if we don;t care about descending
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
```rust
let field = aggr.field()
let is_sorted_descending = aggr.is_sorted_descending()
```
And, I think getting field and descending separately is better than `get_minmax_desc`, so if we can get `field` independently if we don;t care about descending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696231200
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
```rust
let field = aggr.field()
let is_sorted_descending = aggr.is_sorted_descending()
```
And, I think getting field and descending separately is better than `get_minmax_desc`, so if we can get `field` only if we don't care about descending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696229340
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
It is the same idea like descending in `SortOptions`
```rust
pub struct SortOptions {
/// Whether to sort in descending order
pub descending: bool,
```
Would `is_sorted_descending: Option<bool>` be much more better?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Initial changes to support using udaf min/max for statistics and opti… [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #11696:
URL: https://github.com/apache/datafusion/pull/11696#discussion_r1696791166
##########
datafusion/expr/src/udaf.rs:
##########
@@ -249,6 +249,14 @@ impl AggregateUDF {
pub fn simplify(&self) -> Option<AggregateFunctionSimplification> {
self.inner.simplify()
}
+
+ /// Returns true if the function is max, false if the function is min
+ /// None in all other cases, used in certain optimizations or
+ /// or aggregate
+ ///
+ pub fn get_minmax_desc(&self) -> Option<bool> {
Review Comment:
Not necessary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org