You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "jayzhan211 (via GitHub)" <gi...@apache.org> on 2024/06/09 00:43:56 UTC

[I] Convert `approx_median` to UDAF [datafusion]

jayzhan211 opened a new issue, #10838:
URL: https://github.com/apache/datafusion/issues/10838

   ### Is your feature request related to a problem or challenge?
   
   Similar to https://github.com/apache/datafusion/pull/10713 https://github.com/apache/datafusion/pull/10836 https://github.com/apache/datafusion/pull/10834 and others in https://github.com/apache/datafusion/issues/8708
   
   ### Describe the solution you'd like
   
   _No response_
   
   ### Describe alternatives you've considered
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156610120

   > We can add `self.args` to `AccumulatorArgs` and compute `percentile` and `tdigest max size` in `accumulator`.
   >
   
   Thanks for the suggestion. I can make a PR after #10840.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156419089

   Hi @jayzhan211
   I tried to convert `ApproxPercentileCont`, but I encountered some challenges. `ApproxPercentileCont` creates the accumulator based on the `percentile` parameter, but I couldn't find a way to access this parameter when defining the UDAF.
   https://github.com/goldmedal/datafusion/blob/24a08465e12bc07275cafe5310a7ac44898e39de/datafusion/physical-expr/src/aggregate/approx_percentile_cont.rs#L104
   
   The `AccumulatorArgs` doesn't carry the original argument expression, making it difficult to create the accumulator for `ApproxPercentileCont`.
   https://github.com/goldmedal/datafusion/blob/24a08465e12bc07275cafe5310a7ac44898e39de/datafusion/expr/src/function.rs#L45
   
   I think we need another PR to address this issue. For now, I just moved `TDigest` and `ApproxPercentileAccumulator` in #10840 so that I can convert `approx_median` to a UDAF.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 closed issue #10838: Convert `approx_median` to UDAF
URL: https://github.com/apache/datafusion/issues/10838


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156338904

   > approx_median
   
   I agree, we need to convert `ApproxPercentileCont ` first. For `TDigest`,  we should move it to `datafusion/physical-expr-common/src/aggregate/mod.rs` where it can used not only for `functions-aggregate` but also other users that import `physical-expr-common`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156459652

   > Hi @jayzhan211 I tried to convert `ApproxPercentileCont`, but I encountered some challenges. `ApproxPercentileCont` creates the accumulator based on the `percentile` parameter, but I couldn't find a way to access this parameter when defining the UDAF. https://github.com/goldmedal/datafusion/blob/24a08465e12bc07275cafe5310a7ac44898e39de/datafusion/physical-expr/src/aggregate/approx_percentile_cont.rs#L104
   > 
   > The `AccumulatorArgs` doesn't carry the original argument expression, making it difficult to create the accumulator for `ApproxPercentileCont`. https://github.com/goldmedal/datafusion/blob/24a08465e12bc07275cafe5310a7ac44898e39de/datafusion/expr/src/function.rs#L45
   > 
   > I think we need another PR to address this issue. For now, I just moved `TDigest` and `ApproxPercentileAccumulator` in #10840 so that I can convert `approx_median` to a UDAF.
   
   We can add `self.args` to `AccumulatorArgs` and compute `percentile` and `tdigest max size` in `accumulator`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156351460

   > I agree, we need to convert `ApproxPercentileCont ` first. For `TDigest`, we should move it to `datafusion/physical-expr-common/src/aggregate/mod.rs` where it can used not only for `functions-aggregate` but also other users that import `physical-expr-common`
   
   Thanks. I will have PR for it first.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156291208

   take


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [I] Convert `approx_median` to UDAF [datafusion]

Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on issue #10838:
URL: https://github.com/apache/datafusion/issues/10838#issuecomment-2156334969

   Hi @jayzhan211,
   
   After some research, I found that `approx_median` heavily relies on `ApproxPercentileCont` for accumulation. 
   
   https://github.com/goldmedal/datafusion/blob/1600a307f272f9c716131dcca7a55ba1761774a6/datafusion/physical-expr/src/aggregate/approx_median.rs#L46
   
   I think we should convert `ApproxPercentileCont` to a UDAF first. We can move `ApproxPercentileCont` and `TDigest` to `function-aggregate`, so that `ApproxMedian` can use them.
   
   WDYT?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org