You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "goldmedal (via GitHub)" <gi...@apache.org> on 2024/06/14 16:12:36 UTC
[PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
goldmedal opened a new pull request, #10917:
URL: https://github.com/apache/datafusion/pull/10917
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->
Closes #10870 and convert ApproxPercentileContWithWeight to udaf.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
## What changes are included in this PR?
- Pass original logical expression `args` to `AccumulatorArgs`.
- Remove args_num from `AccumulatorArgs`, use `args.len()` instead.
- convert `ApproxPercentileCont` and `ApproxPercentileContWithWeight` to udaf
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
## Are these changes tested?
yes
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
## Are there any user-facing changes?
no
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641808518
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,254 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::Field;
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Expr, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
Review Comment:
I found we can't use `Signature::numeric`. I tried to use it, but I discovered that all the input arguments will have the same type as the first argument. For example, in a SQL query like:
```
select approx_percentile_cont(float64_col, 0.5, 100) from t
```
the input arguments will be `vec![Float64, Float64, Float64]`. However, the last argument should be `int64`. This will cause a failure.
I'm not sure if it's related, but I found something weird. Why is `AccumulatorArgs#input_type` not a `vec` but a single datatype? If the number of arguments is greater than 1, how do we handle the other arguments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641808587
##########
datafusion/functions-aggregate/src/approx_percentile_cont_with_weight.rs:
##########
@@ -15,91 +15,139 @@
// specific language governing permissions and limitations
// under the License.
-use crate::expressions::ApproxPercentileCont;
-use crate::{AggregateExpr, PhysicalExpr};
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::ArrayRef,
datatypes::{DataType, Field},
};
-use datafusion_functions_aggregate::approx_percentile_cont::ApproxPercentileAccumulator;
+
+use datafusion_common::ScalarValue;
+use datafusion_common::{not_impl_err, plan_err, Result};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::NUMERICS;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
Centroid, TDigest, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::physical_expr::down_cast_any_ref;
-use datafusion_common::Result;
-use datafusion_common::ScalarValue;
-use datafusion_expr::Accumulator;
+use crate::approx_percentile_cont::{ApproxPercentileAccumulator, ApproxPercentileCont};
-use crate::aggregate::utils::down_cast_any_ref;
-use std::{any::Any, sync::Arc};
+make_udaf_expr_and_func!(
+ ApproxPercentileContWithWeight,
+ approx_percentile_cont_with_weight,
+ expression weight percentile,
+ "Computes the approximate percentile continuous with weight of a set of numbers",
+ approx_percentile_cont_with_weight_udaf
+);
/// APPROX_PERCENTILE_CONT_WITH_WEIGTH aggregate expression
-#[derive(Debug)]
pub struct ApproxPercentileContWithWeight {
+ signature: Signature,
approx_percentile_cont: ApproxPercentileCont,
- column_expr: Arc<dyn PhysicalExpr>,
- weight_expr: Arc<dyn PhysicalExpr>,
- percentile_expr: Arc<dyn PhysicalExpr>,
+}
+
+impl Debug for ApproxPercentileContWithWeight {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileContWithWeight")
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileContWithWeight {
+ fn default() -> Self {
+ Self::new()
+ }
}
impl ApproxPercentileContWithWeight {
/// Create a new [`ApproxPercentileContWithWeight`] aggregate function.
- pub fn new(
- expr: Vec<Arc<dyn PhysicalExpr>>,
- name: impl Into<String>,
- return_type: DataType,
- ) -> Result<Self> {
- // Arguments should be [ColumnExpr, WeightExpr, DesiredPercentileLiteral]
- debug_assert_eq!(expr.len(), 3);
-
- let sub_expr = vec![expr[0].clone(), expr[2].clone()];
- let approx_percentile_cont =
- ApproxPercentileCont::new(sub_expr, name, return_type)?;
-
- Ok(Self {
- approx_percentile_cont,
- column_expr: expr[0].clone(),
- weight_expr: expr[1].clone(),
- percentile_expr: expr[2].clone(),
- })
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
Review Comment:
Same as https://github.com/apache/datafusion/pull/10917#discussion_r1641808518
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2171694652
> Thanks @jayzhan211 approved it. : ) If no more comments, how about merge this PR first? I guess it may cause some conflicts for other UDAF-related PRs.
Sure!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641265848
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
Yes, it will be correct for the logical plan. However, I think it will cause some issues when processing the physical plan. Curiously, should we expect the physical roundtrip to work? I tried to add a roundtrip test for it in`datafusion/proto/tests/cases/roundtrip_physical_plan.rs` like:
```rust
#[test]
fn roundtrip_approx_percentile_cont_udaf() -> Result<()> {
let field_a = Field::new("a", DataType::Int64, false);
let field_b = Field::new("b", DataType::Float64, false);
let schema = Arc::new(Schema::new(vec![field_a, field_b]));
let udaf = (*approx_percentile_cont_udaf()).clone();
let ctx = SessionContext::new();
ctx.register_udaf(udaf.clone());
let groups: Vec<(Arc<dyn PhysicalExpr>, String)> =
vec![(col("a", &schema)?, "unused".to_string())];
let aggregates: Vec<Arc<dyn AggregateExpr>> = vec![udaf::create_aggregate_expr(
&udaf,
&[col("b", &schema)?, lit(ScalarValue::Float64(Some(0.5)))],
&[datafusion_expr::col("b"), datafusion_expr::lit(ScalarValue::Float64(Some(0.5)))],
&[],
&[],
&schema,
"approx_percentile_cont_agg",
false,
false,
)?];
roundtrip_test_with_context(
Arc::new(AggregateExec::try_new(
AggregateMode::Final,
PhysicalGroupBy::new_single(groups.clone()),
aggregates.clone(),
vec![None],
Arc::new(EmptyExec::new(schema.clone())),
schema,
)?),
&ctx,
)
}
```
It will lose the logical_plan when converting back to the physical plan from protobuf.
Is this test necessary? I only see an physical-plan roudtrip testing of udaf called `roundtrip_aggregate_udaf`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1640557359
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
what is the error if we leave it empty?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2170996403
> Yes, it will be correct for the logical plan. However, I think it will cause some issues when processing the physical plan. Curiously, should we expect the physical roundtrip to work? I tried to add a roundtrip test for it in`datafusion/proto/tests/cases/roundtrip_physical_plan.rs` like:
I have an idea of moving the accumulator and state_fields computation up to create_aggregate_expr, so we don't need to pass logical expression downward to AggregateFunctionExpr
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641818096
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,254 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::Field;
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Expr, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
Review Comment:
> the input arguments will be vec![Float64, Float64, Float64]
Oh, yes. It will coerce types to the unify one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1640141816
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
I can't find an easy way to get the logical plan from protobuf. I think maybe we need to convert the physical plan back to the logical plan. Otherwise, `approx_percentile_cont` and `approx_percentile_cont_weighted` can't be supported for protobuf.
@jayzhan211, what do you think? Are there any existing functions or tools that can do this?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641813923
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,254 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::Field;
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Expr, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
Review Comment:
It is single type because we have not yet meet any function that takes multiple input to build accumulator. If there is, then we might need to extend it to Vec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2170471336
It's really weird. I can't reproduce the CI failed in my local environment.
https://github.com/apache/datafusion/actions/runs/9529829557/job/26268998329?pr=10917
```
External error: statement is expected to fail with error:
(regex) DataFusion error: Error during planning: .* No function matches the given name and argument types 'approx_percentile_cont_with_weight\(Utf8, Int8, Float64\)'\. You might need to add explicit type casts\.(.|\n)*
but got error:
DataFusion error: Error during planning: Error during planning: Coercion from [Utf8, Int8, Float64] to the signature OneOf([Exact([Int8, Int8, Float64]), Exact([Int16, Int16, Float64]), Exact([Int32, Int32, Float64]), Exact([Int64, Int64, Float64]), Exact([UInt8, UInt8, Float64]), Exact([UInt16, UInt16, Float64]), Exact([UInt32, UInt32, Float64]), Exact([UInt64, UInt64, Float64]), Exact([Float32, Float32, Float64]), Exact([Float64, Float64, Float64])]) failed.
[SQL] SELECT approx_percentile_cont_with_weight(c1, c2, 0.95) FROM aggregate_test_100
at test_files/aggregate.slt:79
```
The error message thrown in my local is difference from the CI environment... :(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 merged PR #10917:
URL: https://github.com/apache/datafusion/pull/10917
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2171695849
Thanks again @jayzhan211 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641668286
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
I agree. I thought Flight SQL might be a use case for `from_proto`. However, I tried to query `approx_percentile_cont` through Flight SQL, and it worked well. Maybe we can just ignore it and leave this TODO comment until we find any `from_proto` use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1640557359
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
what is the error if we leave it empty? It run the test in logical plan that includes `approx_percentile_cont` and it seems correct 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2171687704
Thanks @jayzhan211 approved it. : )
If no more comments, how about merge this PR first? I guess it may cause some conflicts for other UDAF-related PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1640557359
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
what is the error if we leave it empty? I run the test in logical plan that includes `approx_percentile_cont` and it seems correct 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1640559776
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,254 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::Field;
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Expr, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
+ }
+ }
+
+ pub(crate) fn create_accumulator(
+ &self,
+ args: AccumulatorArgs,
+ ) -> datafusion_common::Result<ApproxPercentileAccumulator> {
+ let percentile = validate_input_percentile_expr(&args.args[1])?;
Review Comment:
could we rename `args.args` to `args.input_exprs`? I think it would be more clear and consistent with the name of `args.input_type`
##########
datafusion/functions-aggregate/src/approx_percentile_cont_with_weight.rs:
##########
@@ -15,91 +15,139 @@
// specific language governing permissions and limitations
// under the License.
-use crate::expressions::ApproxPercentileCont;
-use crate::{AggregateExpr, PhysicalExpr};
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::ArrayRef,
datatypes::{DataType, Field},
};
-use datafusion_functions_aggregate::approx_percentile_cont::ApproxPercentileAccumulator;
+
+use datafusion_common::ScalarValue;
+use datafusion_common::{not_impl_err, plan_err, Result};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::NUMERICS;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
Centroid, TDigest, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::physical_expr::down_cast_any_ref;
-use datafusion_common::Result;
-use datafusion_common::ScalarValue;
-use datafusion_expr::Accumulator;
+use crate::approx_percentile_cont::{ApproxPercentileAccumulator, ApproxPercentileCont};
-use crate::aggregate::utils::down_cast_any_ref;
-use std::{any::Any, sync::Arc};
+make_udaf_expr_and_func!(
+ ApproxPercentileContWithWeight,
+ approx_percentile_cont_with_weight,
+ expression weight percentile,
+ "Computes the approximate percentile continuous with weight of a set of numbers",
+ approx_percentile_cont_with_weight_udaf
+);
/// APPROX_PERCENTILE_CONT_WITH_WEIGTH aggregate expression
-#[derive(Debug)]
pub struct ApproxPercentileContWithWeight {
+ signature: Signature,
approx_percentile_cont: ApproxPercentileCont,
- column_expr: Arc<dyn PhysicalExpr>,
- weight_expr: Arc<dyn PhysicalExpr>,
- percentile_expr: Arc<dyn PhysicalExpr>,
+}
+
+impl Debug for ApproxPercentileContWithWeight {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileContWithWeight")
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileContWithWeight {
+ fn default() -> Self {
+ Self::new()
+ }
}
impl ApproxPercentileContWithWeight {
/// Create a new [`ApproxPercentileContWithWeight`] aggregate function.
- pub fn new(
- expr: Vec<Arc<dyn PhysicalExpr>>,
- name: impl Into<String>,
- return_type: DataType,
- ) -> Result<Self> {
- // Arguments should be [ColumnExpr, WeightExpr, DesiredPercentileLiteral]
- debug_assert_eq!(expr.len(), 3);
-
- let sub_expr = vec![expr[0].clone(), expr[2].clone()];
- let approx_percentile_cont =
- ApproxPercentileCont::new(sub_expr, name, return_type)?;
-
- Ok(Self {
- approx_percentile_cont,
- column_expr: expr[0].clone(),
- weight_expr: expr[1].clone(),
- percentile_expr: expr[2].clone(),
- })
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
+ // Accept any numeric value paired with a float64 percentile
+ NUMERICS
+ .iter()
+ .map(|t| {
+ TypeSignature::Exact(vec![
+ t.clone(),
+ t.clone(),
+ DataType::Float64,
+ ])
+ })
+ .collect(),
+ Volatility::Immutable,
+ ),
+ approx_percentile_cont: ApproxPercentileCont::new(),
+ }
}
}
-impl AggregateExpr for ApproxPercentileContWithWeight {
+impl AggregateUDFImpl for ApproxPercentileContWithWeight {
fn as_any(&self) -> &dyn Any {
self
}
- fn field(&self) -> Result<Field> {
- self.approx_percentile_cont.field()
+ fn name(&self) -> &str {
+ "approx_percentile_cont_with_weight"
}
- #[allow(rustdoc::private_intra_doc_links)]
- /// See [`TDigest::to_scalar_state()`] for a description of the serialised
- /// state.
- fn state_fields(&self) -> Result<Vec<Field>> {
- self.approx_percentile_cont.state_fields()
+ fn signature(&self) -> &Signature {
+ &self.signature
}
- fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
- vec![
- self.column_expr.clone(),
- self.weight_expr.clone(),
- self.percentile_expr.clone(),
- ]
+ fn return_type(&self, arg_types: &[DataType]) -> Result<DataType> {
+ if !arg_types[0].is_numeric() {
+ return plan_err!(
+ "approx_percentile_cont_with_weight requires numeric input types"
+ );
+ }
+ if !arg_types[1].is_numeric() {
+ return plan_err!(
+ "approx_percentile_cont_with_weight requires numeric weight input types"
+ );
+ }
+ if !arg_types[2].is_floating() {
Review Comment:
with signature::numeric(3), we can just check if the third arg is f64
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,254 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::Field;
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Expr, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
Review Comment:
Is it better to have `Signature::one_of(vec![Numeric::(2), Numeric::(3)])`?
##########
datafusion/functions-aggregate/src/approx_percentile_cont_with_weight.rs:
##########
@@ -15,91 +15,139 @@
// specific language governing permissions and limitations
// under the License.
-use crate::expressions::ApproxPercentileCont;
-use crate::{AggregateExpr, PhysicalExpr};
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+
use arrow::{
array::ArrayRef,
datatypes::{DataType, Field},
};
-use datafusion_functions_aggregate::approx_percentile_cont::ApproxPercentileAccumulator;
+
+use datafusion_common::ScalarValue;
+use datafusion_common::{not_impl_err, plan_err, Result};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::NUMERICS;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, Signature, TypeSignature, Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
Centroid, TDigest, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::physical_expr::down_cast_any_ref;
-use datafusion_common::Result;
-use datafusion_common::ScalarValue;
-use datafusion_expr::Accumulator;
+use crate::approx_percentile_cont::{ApproxPercentileAccumulator, ApproxPercentileCont};
-use crate::aggregate::utils::down_cast_any_ref;
-use std::{any::Any, sync::Arc};
+make_udaf_expr_and_func!(
+ ApproxPercentileContWithWeight,
+ approx_percentile_cont_with_weight,
+ expression weight percentile,
+ "Computes the approximate percentile continuous with weight of a set of numbers",
+ approx_percentile_cont_with_weight_udaf
+);
/// APPROX_PERCENTILE_CONT_WITH_WEIGTH aggregate expression
-#[derive(Debug)]
pub struct ApproxPercentileContWithWeight {
+ signature: Signature,
approx_percentile_cont: ApproxPercentileCont,
- column_expr: Arc<dyn PhysicalExpr>,
- weight_expr: Arc<dyn PhysicalExpr>,
- percentile_expr: Arc<dyn PhysicalExpr>,
+}
+
+impl Debug for ApproxPercentileContWithWeight {
+ fn fmt(&self, f: &mut Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileContWithWeight")
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileContWithWeight {
+ fn default() -> Self {
+ Self::new()
+ }
}
impl ApproxPercentileContWithWeight {
/// Create a new [`ApproxPercentileContWithWeight`] aggregate function.
- pub fn new(
- expr: Vec<Arc<dyn PhysicalExpr>>,
- name: impl Into<String>,
- return_type: DataType,
- ) -> Result<Self> {
- // Arguments should be [ColumnExpr, WeightExpr, DesiredPercentileLiteral]
- debug_assert_eq!(expr.len(), 3);
-
- let sub_expr = vec![expr[0].clone(), expr[2].clone()];
- let approx_percentile_cont =
- ApproxPercentileCont::new(sub_expr, name, return_type)?;
-
- Ok(Self {
- approx_percentile_cont,
- column_expr: expr[0].clone(),
- weight_expr: expr[1].clone(),
- percentile_expr: expr[2].clone(),
- })
+ pub fn new() -> Self {
+ Self {
+ signature: Signature::one_of(
Review Comment:
Maybe use Signature::Numeric(3)?
##########
datafusion/functions-aggregate/src/approx_percentile_cont.rs:
##########
@@ -15,19 +15,265 @@
// specific language governing permissions and limitations
// under the License.
+use std::any::Any;
+use std::fmt::{Debug, Formatter};
+use std::sync::Arc;
+
+use arrow::array::RecordBatch;
use arrow::{
array::{
ArrayRef, Float32Array, Float64Array, Int16Array, Int32Array, Int64Array,
Int8Array, UInt16Array, UInt32Array, UInt64Array, UInt8Array,
},
datatypes::DataType,
};
+use arrow_schema::{Field, Schema};
-use datafusion_common::{downcast_value, internal_err, DataFusionError, ScalarValue};
-use datafusion_expr::Accumulator;
+use datafusion_common::{
+ downcast_value, internal_err, not_impl_err, plan_err, DataFusionError, ScalarValue,
+};
+use datafusion_expr::function::{AccumulatorArgs, StateFieldsArgs};
+use datafusion_expr::type_coercion::aggregates::{INTEGERS, NUMERICS};
+use datafusion_expr::utils::format_state_name;
+use datafusion_expr::{
+ Accumulator, AggregateUDFImpl, ColumnarValue, Expr, Signature, TypeSignature,
+ Volatility,
+};
use datafusion_physical_expr_common::aggregate::tdigest::{
TDigest, TryIntoF64, DEFAULT_MAX_SIZE,
};
+use datafusion_physical_expr_common::aggregate::utils::down_cast_any_ref;
+use datafusion_physical_expr_common::utils::limited_convert_logical_expr_to_physical_expr;
+
+make_udaf_expr_and_func!(
+ ApproxPercentileCont,
+ approx_percentile_cont,
+ expression percentile,
+ "Computes the approximate percentile continuous of a set of numbers",
+ approx_percentile_cont_udaf
+);
+
+pub struct ApproxPercentileCont {
+ signature: Signature,
+}
+
+impl Debug for ApproxPercentileCont {
+ fn fmt(&self, f: &mut Formatter) -> std::fmt::Result {
+ f.debug_struct("ApproxPercentileCont")
+ .field("name", &self.name())
+ .field("signature", &self.signature)
+ .finish()
+ }
+}
+
+impl Default for ApproxPercentileCont {
+ fn default() -> Self {
+ Self::new()
+ }
+}
+
+impl ApproxPercentileCont {
+ /// Create a new [`ApproxPercentileCont`] aggregate function.
+ pub fn new() -> Self {
+ let mut variants = Vec::with_capacity(NUMERICS.len() * (INTEGERS.len() + 1));
+ // Accept any numeric value paired with a float64 percentile
+ for num in NUMERICS {
+ variants.push(TypeSignature::Exact(vec![num.clone(), DataType::Float64]));
+ // Additionally accept an integer number of centroids for T-Digest
+ for int in INTEGERS {
+ variants.push(TypeSignature::Exact(vec![
+ num.clone(),
+ DataType::Float64,
+ int.clone(),
+ ]))
+ }
+ }
+ Self {
+ signature: Signature::one_of(variants, Volatility::Immutable),
+ }
+ }
+
+ pub(crate) fn create_accumulator(
+ &self,
+ args: AccumulatorArgs,
+ ) -> datafusion_common::Result<ApproxPercentileAccumulator> {
+ let percentile = validate_input_percentile_expr(&args.args[1])?;
+ let tdigest_max_size = if args.args.len() == 3 {
+ Some(validate_input_max_size_expr(&args.args[2])?)
+ } else {
+ None
+ };
+
+ let accumulator: ApproxPercentileAccumulator = match args.input_type {
+ t @ (DataType::UInt8
+ | DataType::UInt16
+ | DataType::UInt32
+ | DataType::UInt64
+ | DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Int64
+ | DataType::Float32
+ | DataType::Float64) => {
+ if let Some(max_size) = tdigest_max_size {
+ ApproxPercentileAccumulator::new_with_max_size(percentile, t.clone(), max_size)
+ }else{
+ ApproxPercentileAccumulator::new(percentile, t.clone())
+
+ }
+ }
+ other => {
+ return not_impl_err!(
+ "Support for 'APPROX_PERCENTILE_CONT' for data type {other} is not implemented"
+ )
+ }
+ };
+
+ Ok(accumulator)
+ }
+}
+
+impl PartialEq for ApproxPercentileCont {
+ fn eq(&self, other: &ApproxPercentileCont) -> bool {
+ self.signature == other.signature
+ }
+}
+
+impl PartialEq<dyn Any> for ApproxPercentileCont {
+ fn eq(&self, other: &dyn Any) -> bool {
+ down_cast_any_ref(other)
+ .downcast_ref::<Self>()
+ .map(|x| self.eq(x))
+ .unwrap_or(false)
+ }
+}
+
+fn get_lit_value(expr: &Expr) -> datafusion_common::Result<ScalarValue> {
+ let empty_schema = Arc::new(Schema::empty());
+ let empty_batch = RecordBatch::new_empty(Arc::clone(&empty_schema));
+ let expr = limited_convert_logical_expr_to_physical_expr(expr, &empty_schema)?;
+ let result = expr.evaluate(&empty_batch)?;
+ match result {
+ ColumnarValue::Array(_) => Err(DataFusionError::Internal(format!(
+ "The expr {:?} can't be evaluated to scalar value",
+ expr
+ ))),
+ ColumnarValue::Scalar(scalar_value) => Ok(scalar_value),
+ }
+}
+
+fn validate_input_percentile_expr(expr: &Expr) -> datafusion_common::Result<f64> {
+ let lit = get_lit_value(expr)?;
+ let percentile = match &lit {
+ ScalarValue::Float32(Some(q)) => *q as f64,
+ ScalarValue::Float64(Some(q)) => *q,
+ got => return not_impl_err!(
+ "Percentile value for 'APPROX_PERCENTILE_CONT' must be Float32 or Float64 literal (got data type {})",
+ got.data_type()
+ )
+ };
+
+ // Ensure the percentile is between 0 and 1.
+ if !(0.0..=1.0).contains(&percentile) {
+ return plan_err!(
+ "Percentile value must be between 0.0 and 1.0 inclusive, {percentile} is invalid"
+ );
+ }
+ Ok(percentile)
+}
+
+fn validate_input_max_size_expr(expr: &Expr) -> datafusion_common::Result<usize> {
+ let lit = get_lit_value(expr)?;
+ let max_size = match &lit {
+ ScalarValue::UInt8(Some(q)) => *q as usize,
+ ScalarValue::UInt16(Some(q)) => *q as usize,
+ ScalarValue::UInt32(Some(q)) => *q as usize,
+ ScalarValue::UInt64(Some(q)) => *q as usize,
+ ScalarValue::Int32(Some(q)) if *q > 0 => *q as usize,
+ ScalarValue::Int64(Some(q)) if *q > 0 => *q as usize,
+ ScalarValue::Int16(Some(q)) if *q > 0 => *q as usize,
+ ScalarValue::Int8(Some(q)) if *q > 0 => *q as usize,
+ got => return not_impl_err!(
+ "Tdigest max_size value for 'APPROX_PERCENTILE_CONT' must be UInt > 0 literal (got data type {}).",
+ got.data_type()
+ )
+ };
+ Ok(max_size)
+}
+
+impl AggregateUDFImpl for ApproxPercentileCont {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ #[allow(rustdoc::private_intra_doc_links)]
+ /// See [`datafusion_physical_expr_common::aggregate::tdigest::TDigest::to_scalar_state()`] for a description of the serialised
+ /// state.
+ fn state_fields(
+ &self,
+ args: StateFieldsArgs,
+ ) -> datafusion_common::Result<Vec<Field>> {
+ Ok(vec![
+ Field::new(
+ format_state_name(args.name, "max_size"),
+ DataType::UInt64,
+ false,
+ ),
+ Field::new(
+ format_state_name(args.name, "sum"),
+ DataType::Float64,
+ false,
+ ),
+ Field::new(
+ format_state_name(args.name, "count"),
+ DataType::Float64,
+ false,
+ ),
+ Field::new(
+ format_state_name(args.name, "max"),
+ DataType::Float64,
+ false,
+ ),
+ Field::new(
+ format_state_name(args.name, "min"),
+ DataType::Float64,
+ false,
+ ),
+ Field::new_list(
+ format_state_name(args.name, "centroids"),
+ Field::new("item", DataType::Float64, true),
+ false,
+ ),
+ ])
+ }
+
+ fn name(&self) -> &str {
+ "approx_percentile_cont"
+ }
+
+ fn signature(&self) -> &Signature {
+ &self.signature
+ }
+
+ fn accumulator(
+ &self,
+ acc_args: AccumulatorArgs,
+ ) -> datafusion_common::Result<Box<dyn Accumulator>> {
+ Ok(Box::new(self.create_accumulator(acc_args)?))
Review Comment:
We can inline the code of `create_accumulator` here
##########
datafusion/functions-aggregate/src/approx_percentile_cont_with_weight.rs:
##########
@@ -108,10 +156,8 @@ impl PartialEq<dyn Any> for ApproxPercentileContWithWeight {
down_cast_any_ref(other)
.downcast_ref::<Self>()
.map(|x| {
- self.approx_percentile_cont == x.approx_percentile_cont
- && self.column_expr.eq(&x.column_expr)
- && self.weight_expr.eq(&x.weight_expr)
- && self.percentile_expr.eq(&x.percentile_expr)
+ self.signature == x.signature
Review Comment:
I don't think we need this, we should get `AggregateFunctionExpr`, and it has partialEq implemented already
##########
datafusion/proto/src/physical_plan/mod.rs:
##########
@@ -496,11 +496,14 @@ impl AsExecutionPlan for protobuf::PhysicalPlanNode {
}
AggregateFunction::UserDefinedAggrFunction(udaf_name) => {
let agg_udf = registry.udaf(udaf_name)?;
+ // TODO: 'logical_exprs' is not supported for UDAF yet.
+ // approx_percentile_cont and approx_percentile_cont_weight are not supported for UDAF from protobuf yet.
+ let logical_exprs = &[];
Review Comment:
I think we don't need to care about logical plan conversion in physical plan roundtrip, but I'm not pretty sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on code in PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#discussion_r1641592218
##########
datafusion/functions-aggregate/src/approx_percentile_cont_with_weight.rs:
##########
@@ -108,10 +156,8 @@ impl PartialEq<dyn Any> for ApproxPercentileContWithWeight {
down_cast_any_ref(other)
.downcast_ref::<Self>()
.map(|x| {
- self.approx_percentile_cont == x.approx_percentile_cont
- && self.column_expr.eq(&x.column_expr)
- && self.weight_expr.eq(&x.weight_expr)
- && self.percentile_expr.eq(&x.percentile_expr)
+ self.signature == x.signature
Review Comment:
I don't think we need this, we can compare them with the function name
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "goldmedal (via GitHub)" <gi...@apache.org>.
goldmedal commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2171470071
> > External error: statement is expected to fail with error:
> > (regex) DataFusion error: Error during planning: .* No function matches the given name and argument types
> I usually just remove the error message, although it is not an idea solution, but I don't know why the CI message is different from local too.
I modified the expected pattern of error message to match the message thrown in CI flow. The full message is
```
DataFusion error: Error during planning: Error during planning: Coercion from [Utf8, Int8, Float64] to the signature OneOf([Exact([Int8, Int8, Float64]), Exact([Int16, Int16, Float64]), Exact([Int32, Int32, Float64]), Exact([Int64, Int64, Float64]), Exact([UInt8, UInt8, Float64]), Exact([UInt16, UInt16, Float64]), Exact([UInt32, UInt32, Float64]), Exact([UInt64, UInt64, Float64]), Exact([Float32, Float32, Float64]), Exact([Float64, Float64, Float64])]) failed. No function matches the given name and argument types 'approx_percentile_cont_with_weight(Utf8, Int8, Float64)'. You might need to add explicit type casts.
Candidate functions:
approx_percentile_cont_with_weight(Int8, Int8, Float64)
approx_percentile_cont_with_weight(Int16, Int16, Float64)
approx_percentile_cont_with_weight(Int32, Int32, Float64)
approx_percentile_cont_with_weight(Int64, Int64, Float64)
approx_percentile_cont_with_weight(UInt8, UInt8, Float64)
approx_percentile_cont_with_weight(UInt16, UInt16, Float64)
approx_percentile_cont_with_weight(UInt32, UInt32, Float64)
approx_percentile_cont_with_weight(UInt64, UInt64, Float64)
approx_percentile_cont_with_weight(Float32, Float32, Float64)
approx_percentile_cont_with_weight(Float64, Float64, Float64)
```
I noticed that the CI flow is missing the tail part of the error message: `You might need to add explicit type casts....`. There might be a length limitation for the error message, but I'm not sure. Just making a note of it here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [PR] Convert ApproxPercentileCont and ApproxPercentileContWithWeight to UDAF [datafusion]
Posted by "jayzhan211 (via GitHub)" <gi...@apache.org>.
jayzhan211 commented on PR #10917:
URL: https://github.com/apache/datafusion/pull/10917#issuecomment-2170996602
> It's really weird. I can't reproduce the CI failed in my local environment. https://github.com/apache/datafusion/actions/runs/9529829557/job/26268998329?pr=10917
>
> ```
> External error: statement is expected to fail with error:
> (regex) DataFusion error: Error during planning: .* No function matches the given name and argument types 'approx_percentile_cont_with_weight\(Utf8, Int8, Float64\)'\. You might need to add explicit type casts\.(.|\n)*
> but got error:
> DataFusion error: Error during planning: Error during planning: Coercion from [Utf8, Int8, Float64] to the signature OneOf([Exact([Int8, Int8, Float64]), Exact([Int16, Int16, Float64]), Exact([Int32, Int32, Float64]), Exact([Int64, Int64, Float64]), Exact([UInt8, UInt8, Float64]), Exact([UInt16, UInt16, Float64]), Exact([UInt32, UInt32, Float64]), Exact([UInt64, UInt64, Float64]), Exact([Float32, Float32, Float64]), Exact([Float64, Float64, Float64])]) failed.
> [SQL] SELECT approx_percentile_cont_with_weight(c1, c2, 0.95) FROM aggregate_test_100
> at test_files/aggregate.slt:79
> ```
>
> The error message thrown in my local is difference from the CI environment... :(
I usually just remove the error message, although it is not an idea solution, but I don't know why the CI message is different from local too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org