You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/18 02:55:41 UTC
[GitHub] [arrow-datafusion] jychen7 opened a new pull request #2031: #2004 approx percentile with weight
jychen7 opened a new pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031
# Which issue does this PR close?
Closes https://github.com/apache/arrow-datafusion/issues/2004
# Rationale for this change
In the scenario of low latency query OLAP system (e.g. Druid), one common way is to pre-aggregate sketches during ingestion time (e.g. Spark/Flink -> DataStore), then merge sketches in query time (e.g. DataStore -> Datafusion).
# What changes are included in this PR?
There are 12 files change, but most of them are standard ones. Would be easy to review by commits.
The main change is improving `tdigest/mod.rs` and `approx_percentile_cont.rs` to share code usage to the new `appro_percentile_cont_with_weight.rs`
# Are there any user-facing changes?
NA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833780479
##########
File path: datafusion-physical-expr/src/coercion_rule/aggregate_rule.rs
##########
@@ -152,6 +152,27 @@ pub fn coerce_types(
}
Ok(input_types.to_vec())
}
+ AggregateFunction::ApproxPercentileContWithWeight => {
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[0]) {
+ return Err(DataFusionError::Plan(format!(
+ "The function {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
+ )));
+ }
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[1]) {
+ return Err(DataFusionError::Plan(format!(
+ "The weight argument for {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
Review comment:
right, let me fix that
will add a test case to cover the type checking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r829662200
##########
File path: datafusion-physical-expr/src/tdigest/mod.rs
##########
@@ -708,29 +714,12 @@ mod tests {
#[test]
fn test_int64_uniform() {
- let values = (1i64..=1000).map(|v| ScalarValue::Int64(Some(v)));
-
- let t = TDigest::new(100);
- let t = t.merge_unsorted(values).unwrap();
-
- assert_error_bounds!(t, quantile = 0.1, want = 100.0);
- assert_error_bounds!(t, quantile = 0.5, want = 500.0);
- assert_error_bounds!(t, quantile = 0.9, want = 900.0);
- assert_state_roundtrip!(t);
- }
-
- #[test]
- fn test_int64_uniform_with_nulls() {
Review comment:
remove this test case because now `merge_unsorted_f64` will not accept None, the None filter may be test in `try_as_f64`
https://github.com/apache/arrow-datafusion/blob/136e9e9803b17c0a248939e904e22f218f3efb4b/datafusion-physical-expr/src/tdigest/mod.rs#L53-L60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb merged pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r829662200
##########
File path: datafusion-physical-expr/src/tdigest/mod.rs
##########
@@ -708,29 +714,12 @@ mod tests {
#[test]
fn test_int64_uniform() {
- let values = (1i64..=1000).map(|v| ScalarValue::Int64(Some(v)));
-
- let t = TDigest::new(100);
- let t = t.merge_unsorted(values).unwrap();
-
- assert_error_bounds!(t, quantile = 0.1, want = 100.0);
- assert_error_bounds!(t, quantile = 0.5, want = 500.0);
- assert_error_bounds!(t, quantile = 0.9, want = 900.0);
- assert_state_roundtrip!(t);
- }
-
- #[test]
- fn test_int64_uniform_with_nulls() {
Review comment:
remove this test case because now `merge_unsorted_f64` will not accept None, the None filter should be test in `try_as_f64`
https://github.com/apache/arrow-datafusion/blob/136e9e9803b17c0a248939e904e22f218f3efb4b/datafusion-physical-expr/src/tdigest/mod.rs#L53-L60
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1076796542
The other thing I was thinking is that https://github.com/apache/arrow-datafusion/issues/2004 has a really nice writeup on it -- it seems a shame that it will be likely lost on the ticket description. I wonder if we could copy the description somewhere -- perhaps in the docstrings 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 edited a comment on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 edited a comment on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1076895570
> The other thing I was thinking is that https://github.com/apache/arrow-datafusion/issues/2004 has a really nice writeup on it
Thank you!
> I wonder if we could copy the description somewhere -- perhaps in the docstrings
how about adding a user guide about approximate function? so we can explain how to use them and why
https://github.com/apache/arrow-datafusion/tree/master/docs/source/user-guide/sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1074651614
sorry, @alamb, not urgent, just want to check if you are interested to review this.
Or if not, do you know who would be more appropriate? Thank you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833498405
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont_with_weight.rs
##########
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::expressions::approx_percentile_cont::ApproxPercentileAccumulator;
+use crate::expressions::ApproxPercentileCont;
+use crate::{
+ tdigest::{Centroid, TDigest, DEFAULT_MAX_SIZE},
+ AggregateExpr, PhysicalExpr,
+};
+use arrow::{
+ array::ArrayRef,
+ datatypes::{DataType, Field},
+};
+
+use datafusion_common::Result;
+use datafusion_common::ScalarValue;
+use datafusion_expr::Accumulator;
+
+use std::{any::Any, sync::Arc};
+
+/// APPROX_PERCENTILE_CONT_WITH_WEIGTH aggregate expression
+#[derive(Debug)]
+pub struct ApproxPercentileContWithWeight {
+ approx_percentile_cont: ApproxPercentileCont,
+ column_expr: Arc<dyn PhysicalExpr>,
+ weight_expr: Arc<dyn PhysicalExpr>,
+}
+
+impl ApproxPercentileContWithWeight {
+ /// Create a new [`ApproxPercentileContWithWeight`] aggregate function.
+ pub fn new(
+ expr: Vec<Arc<dyn PhysicalExpr>>,
+ name: impl Into<String>,
+ return_type: DataType,
+ ) -> Result<Self> {
+ // Arguments should be [ColumnExpr, WeightExpr, DesiredPercentileLiteral]
+ debug_assert_eq!(expr.len(), 3);
+
+ let sub_expr = vec![expr[0].clone(), expr[2].clone()];
+ let approx_percentile_cont =
+ ApproxPercentileCont::new(sub_expr, name, return_type)?;
+
+ Ok(Self {
+ approx_percentile_cont,
+ column_expr: expr[0].clone(),
+ weight_expr: expr[1].clone(),
+ })
+ }
+}
+
+impl AggregateExpr for ApproxPercentileContWithWeight {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn field(&self) -> Result<Field> {
+ self.approx_percentile_cont.field()
+ }
+
+ #[allow(rustdoc::private_intra_doc_links)]
+ /// See [`TDigest::to_scalar_state()`] for a description of the serialised
+ /// state.
+ fn state_fields(&self) -> Result<Vec<Field>> {
+ self.approx_percentile_cont.state_fields()
+ }
+
+ fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
+ vec![self.column_expr.clone(), self.weight_expr.clone()]
+ }
+
+ fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
+ let approx_percentile_cont_accumulator =
+ self.approx_percentile_cont.create_plain_accumulator()?;
+ let accumulator = ApproxPercentileWithWeightAccumulator::new(
+ approx_percentile_cont_accumulator,
+ );
+ Ok(Box::new(accumulator))
+ }
+
+ fn name(&self) -> &str {
+ self.approx_percentile_cont.name()
+ }
+}
+
+#[derive(Debug)]
+pub struct ApproxPercentileWithWeightAccumulator {
+ approx_percentile_cont_accumulator: ApproxPercentileAccumulator,
+}
+
+impl ApproxPercentileWithWeightAccumulator {
+ pub fn new(approx_percentile_cont_accumulator: ApproxPercentileAccumulator) -> Self {
+ Self {
+ approx_percentile_cont_accumulator,
+ }
+ }
+}
+
+impl Accumulator for ApproxPercentileWithWeightAccumulator {
+ fn state(&self) -> Result<Vec<ScalarValue>> {
+ self.approx_percentile_cont_accumulator.state()
+ }
+
+ fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
+ debug_assert_eq!(
+ values.len(),
+ 2,
+ "invalid number of values in batch percentile update"
+ );
+ let means = &values[0];
+ let weights = &values[1];
+ debug_assert_eq!(
+ means.len(),
+ weights.len(),
+ "invalid number of values in means and weights"
+ );
+ let means_f64 = ApproxPercentileAccumulator::convert_to_ordered_float(means)?;
+ let weights_f64 = ApproxPercentileAccumulator::convert_to_ordered_float(weights)?;
+ let mut digests: Vec<TDigest> = vec![];
+ for (mean, weight) in means_f64.iter().zip(weights_f64.iter()) {
+ digests.push(TDigest::new_with_centroid(
+ DEFAULT_MAX_SIZE,
+ Centroid::new(*mean, *weight),
+ ))
+ }
+ self.approx_percentile_cont_accumulator
+ .merge_digests(&digests);
Review comment:
@Ted-Jiang good question.
I separate them out because their input arguments are slightly different, one is `(column_expr, percentile)`, another is `(column_expr, weight_expr, percentile)`.
ps: NOT using `(column_expr, percentile, weight_expr)` because I am following Presto/Trino
https://github.com/apache/arrow-datafusion/blob/cda79e1b8a9e27acc604379ad4de18dc00d11275/datafusion/src/physical_plan/aggregates.rs#L359-L376
---
If we want to combine the code, after merge, we could simplify `approx_percentile_cont` by moving all its logic to `approx_percentile_cont_with_weight` and add an optimization rule to rewrite `approx_percentile_cont(x, p)` to `approx_percentile_cont_with_weight(x, 1, p)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833775422
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
Review comment:
yes, the value is expected to be different, since we use `c2` for weight and some weight is larger than
by comparing the difference with above weight 1, I think it is reasonable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1075511892
I am sorry but I don't think I will have time to review this today -- it is near the top of my list however
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833685712
##########
File path: datafusion-physical-expr/src/coercion_rule/aggregate_rule.rs
##########
@@ -152,6 +152,27 @@ pub fn coerce_types(
}
Ok(input_types.to_vec())
}
+ AggregateFunction::ApproxPercentileContWithWeight => {
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[0]) {
+ return Err(DataFusionError::Plan(format!(
+ "The function {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
+ )));
+ }
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[1]) {
+ return Err(DataFusionError::Plan(format!(
+ "The weight argument for {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
Review comment:
```suggestion
agg_fun, input_types[1]
```
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
Review comment:
these values are different -- I don't honestly know if they are correct or not 🤷
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont.rs
##########
@@ -194,75 +204,125 @@ pub struct ApproxPercentileAccumulator {
impl ApproxPercentileAccumulator {
pub fn new(percentile: f64, return_type: DataType) -> Self {
Self {
- digest: TDigest::new(100),
+ digest: TDigest::new(DEFAULT_MAX_SIZE),
percentile,
return_type,
}
}
-}
-impl Accumulator for ApproxPercentileAccumulator {
- fn state(&self) -> Result<Vec<ScalarValue>> {
- Ok(self.digest.to_scalar_state())
+ pub(crate) fn merge_digests(&mut self, digests: &[TDigest]) {
+ self.digest = TDigest::merge_digests(digests);
}
- fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- debug_assert_eq!(
- values.len(),
- 1,
- "invalid number of values in batch percentile update"
- );
- let values = &values[0];
-
- self.digest = match values.data_type() {
+ pub(crate) fn convert_to_ordered_float(
+ values: &ArrayRef,
+ ) -> Result<Vec<OrderedFloat<f64>>> {
+ match values.data_type() {
DataType::Float64 => {
let array = values.as_any().downcast_ref::<Float64Array>().unwrap();
- self.digest.merge_unsorted(array.values().iter().cloned())?
+ Ok(array
+ .values()
+ .iter()
+ .filter_map(|v| v.try_as_f64().transpose())
+ .collect::<Result<Vec<_>>>()?)
Review comment:
Yeah, it is tough because the type of the various branches are different.
You could could save the copy by doing something like letting the caller provide a function that gets invoked for each element (untested):
```rust
pub(crate) fn convert_to_ordered_float(
values: &ArrayRef,
f: impl FnMut(Option<OrderedFloat<f64>))
) -> Result<()> {
```
And then call `f()` on each element;
```rust
...
DataType::Float32 => {
let array = values.as_any().downcast_ref::<Float32Array>().unwrap();
array
.values()
.iter()
.try_for_each(|v| {
f(v.try_as_f64()?)
})
}
...
```
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 74 |",
+ "| b | 68 |",
+ "| c | 123 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+ Ok(())
+}
+
Review comment:
I wonder if it is worth one or two error cases here (e.g. try and invoke this function on a `StringArray`)
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont.rs
##########
@@ -194,75 +204,125 @@ pub struct ApproxPercentileAccumulator {
impl ApproxPercentileAccumulator {
pub fn new(percentile: f64, return_type: DataType) -> Self {
Self {
- digest: TDigest::new(100),
+ digest: TDigest::new(DEFAULT_MAX_SIZE),
Review comment:
changing to a symbolic constant `DEFAULT_MAX_SIZE` is a nice improvement 👍
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
Review comment:
these values seem to be the same as the ones with `approx_percentile_cont(c3, 0.95)` which I think is the point of the test. Perhaps we could encode that into the test:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833782087
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
Review comment:
yes, let me consolidate as one expected for `approx_percentile_cont` and `approx_percentile_cont_with_weight`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1077868883
Thanks again @jychen7
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 edited a comment on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 edited a comment on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1076895570
> The other thing I was thinking is that https://github.com/apache/arrow-datafusion/issues/2004 has a really nice writeup on it
Thank you!
> I wonder if we could copy the description somewhere -- perhaps in the docstrings
how about adding a user guide about approximate function? so we can explain how to use them and when
https://github.com/apache/arrow-datafusion/tree/master/docs/source/user-guide/sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r829662200
##########
File path: datafusion-physical-expr/src/tdigest/mod.rs
##########
@@ -708,29 +714,12 @@ mod tests {
#[test]
fn test_int64_uniform() {
- let values = (1i64..=1000).map(|v| ScalarValue::Int64(Some(v)));
-
- let t = TDigest::new(100);
- let t = t.merge_unsorted(values).unwrap();
-
- assert_error_bounds!(t, quantile = 0.1, want = 100.0);
- assert_error_bounds!(t, quantile = 0.5, want = 500.0);
- assert_error_bounds!(t, quantile = 0.9, want = 900.0);
- assert_state_roundtrip!(t);
- }
-
- #[test]
- fn test_int64_uniform_with_nulls() {
Review comment:
remove this test case because now `merge_unsorted_f64` will not accept None, the None filter should be test in `try_as_f64`
https://github.com/apache/arrow-datafusion/blob/136e9e9803b17c0a248939e904e22f218f3efb4b/datafusion-physical-expr/src/tdigest/mod.rs#L53-L60
##########
File path: datafusion-physical-expr/src/tdigest/mod.rs
##########
@@ -708,29 +714,12 @@ mod tests {
#[test]
fn test_int64_uniform() {
- let values = (1i64..=1000).map(|v| ScalarValue::Int64(Some(v)));
-
- let t = TDigest::new(100);
- let t = t.merge_unsorted(values).unwrap();
-
- assert_error_bounds!(t, quantile = 0.1, want = 100.0);
- assert_error_bounds!(t, quantile = 0.5, want = 500.0);
- assert_error_bounds!(t, quantile = 0.9, want = 900.0);
- assert_state_roundtrip!(t);
- }
-
- #[test]
- fn test_int64_uniform_with_nulls() {
Review comment:
remove this test case because now `merge_unsorted_f64` will not accept None, the None filter may be test in `try_as_f64`
https://github.com/apache/arrow-datafusion/blob/136e9e9803b17c0a248939e904e22f218f3efb4b/datafusion-physical-expr/src/tdigest/mod.rs#L53-L60
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont.rs
##########
@@ -194,75 +204,125 @@ pub struct ApproxPercentileAccumulator {
impl ApproxPercentileAccumulator {
pub fn new(percentile: f64, return_type: DataType) -> Self {
Self {
- digest: TDigest::new(100),
+ digest: TDigest::new(DEFAULT_MAX_SIZE),
percentile,
return_type,
}
}
-}
-impl Accumulator for ApproxPercentileAccumulator {
- fn state(&self) -> Result<Vec<ScalarValue>> {
- Ok(self.digest.to_scalar_state())
+ pub(crate) fn merge_digests(&mut self, digests: &[TDigest]) {
+ self.digest = TDigest::merge_digests(digests);
}
- fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- debug_assert_eq!(
- values.len(),
- 1,
- "invalid number of values in batch percentile update"
- );
- let values = &values[0];
-
- self.digest = match values.data_type() {
+ pub(crate) fn convert_to_ordered_float(
+ values: &ArrayRef,
+ ) -> Result<Vec<OrderedFloat<f64>>> {
+ match values.data_type() {
DataType::Float64 => {
let array = values.as_any().downcast_ref::<Float64Array>().unwrap();
- self.digest.merge_unsorted(array.values().iter().cloned())?
+ Ok(array
+ .values()
+ .iter()
+ .filter_map(|v| v.try_as_f64().transpose())
+ .collect::<Result<Vec<_>>>()?)
Review comment:
originally, I want to just return `array.values().iter().cloned()` and then pass to `merge_sorted` as `impl IntoIterator<Item = T>`, but I didn't figure out how to correct `type` it, so I change to `Vec<OrderedFloat<f64>>`
https://github.com/apache/arrow-datafusion/blob/485bfa6ffcf8dc836a482127ecd367d9abfa7248/datafusion-physical-expr/src/tdigest/mod.rs#L252-L255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1075816866
> it is near the top of my list however
@alamb thank you, glad to hear that. Looking forward
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] Ted-Jiang commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
Ted-Jiang commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r832880777
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont_with_weight.rs
##########
@@ -0,0 +1,154 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use crate::expressions::approx_percentile_cont::ApproxPercentileAccumulator;
+use crate::expressions::ApproxPercentileCont;
+use crate::{
+ tdigest::{Centroid, TDigest, DEFAULT_MAX_SIZE},
+ AggregateExpr, PhysicalExpr,
+};
+use arrow::{
+ array::ArrayRef,
+ datatypes::{DataType, Field},
+};
+
+use datafusion_common::Result;
+use datafusion_common::ScalarValue;
+use datafusion_expr::Accumulator;
+
+use std::{any::Any, sync::Arc};
+
+/// APPROX_PERCENTILE_CONT_WITH_WEIGTH aggregate expression
+#[derive(Debug)]
+pub struct ApproxPercentileContWithWeight {
+ approx_percentile_cont: ApproxPercentileCont,
+ column_expr: Arc<dyn PhysicalExpr>,
+ weight_expr: Arc<dyn PhysicalExpr>,
+}
+
+impl ApproxPercentileContWithWeight {
+ /// Create a new [`ApproxPercentileContWithWeight`] aggregate function.
+ pub fn new(
+ expr: Vec<Arc<dyn PhysicalExpr>>,
+ name: impl Into<String>,
+ return_type: DataType,
+ ) -> Result<Self> {
+ // Arguments should be [ColumnExpr, WeightExpr, DesiredPercentileLiteral]
+ debug_assert_eq!(expr.len(), 3);
+
+ let sub_expr = vec![expr[0].clone(), expr[2].clone()];
+ let approx_percentile_cont =
+ ApproxPercentileCont::new(sub_expr, name, return_type)?;
+
+ Ok(Self {
+ approx_percentile_cont,
+ column_expr: expr[0].clone(),
+ weight_expr: expr[1].clone(),
+ })
+ }
+}
+
+impl AggregateExpr for ApproxPercentileContWithWeight {
+ fn as_any(&self) -> &dyn Any {
+ self
+ }
+
+ fn field(&self) -> Result<Field> {
+ self.approx_percentile_cont.field()
+ }
+
+ #[allow(rustdoc::private_intra_doc_links)]
+ /// See [`TDigest::to_scalar_state()`] for a description of the serialised
+ /// state.
+ fn state_fields(&self) -> Result<Vec<Field>> {
+ self.approx_percentile_cont.state_fields()
+ }
+
+ fn expressions(&self) -> Vec<Arc<dyn PhysicalExpr>> {
+ vec![self.column_expr.clone(), self.weight_expr.clone()]
+ }
+
+ fn create_accumulator(&self) -> Result<Box<dyn Accumulator>> {
+ let approx_percentile_cont_accumulator =
+ self.approx_percentile_cont.create_plain_accumulator()?;
+ let accumulator = ApproxPercentileWithWeightAccumulator::new(
+ approx_percentile_cont_accumulator,
+ );
+ Ok(Box::new(accumulator))
+ }
+
+ fn name(&self) -> &str {
+ self.approx_percentile_cont.name()
+ }
+}
+
+#[derive(Debug)]
+pub struct ApproxPercentileWithWeightAccumulator {
+ approx_percentile_cont_accumulator: ApproxPercentileAccumulator,
+}
+
+impl ApproxPercentileWithWeightAccumulator {
+ pub fn new(approx_percentile_cont_accumulator: ApproxPercentileAccumulator) -> Self {
+ Self {
+ approx_percentile_cont_accumulator,
+ }
+ }
+}
+
+impl Accumulator for ApproxPercentileWithWeightAccumulator {
+ fn state(&self) -> Result<Vec<ScalarValue>> {
+ self.approx_percentile_cont_accumulator.state()
+ }
+
+ fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
+ debug_assert_eq!(
+ values.len(),
+ 2,
+ "invalid number of values in batch percentile update"
+ );
+ let means = &values[0];
+ let weights = &values[1];
+ debug_assert_eq!(
+ means.len(),
+ weights.len(),
+ "invalid number of values in means and weights"
+ );
+ let means_f64 = ApproxPercentileAccumulator::convert_to_ordered_float(means)?;
+ let weights_f64 = ApproxPercentileAccumulator::convert_to_ordered_float(weights)?;
+ let mut digests: Vec<TDigest> = vec![];
+ for (mean, weight) in means_f64.iter().zip(weights_f64.iter()) {
+ digests.push(TDigest::new_with_centroid(
+ DEFAULT_MAX_SIZE,
+ Centroid::new(*mean, *weight),
+ ))
+ }
+ self.approx_percentile_cont_accumulator
+ .merge_digests(&digests);
Review comment:
IMO, still use `approx_percentile_cont_accumulator.merge_digests`, why not change the struct `[approx_percentile_cont.rs]` to support `weight `. If i miss something plz tell me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833775422
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
Review comment:
yes, the value is expected to be different, since we use `c2` for weight and some weight is larger than 1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833834102
##########
File path: datafusion-physical-expr/src/coercion_rule/aggregate_rule.rs
##########
@@ -152,6 +152,27 @@ pub fn coerce_types(
}
Ok(input_types.to_vec())
}
+ AggregateFunction::ApproxPercentileContWithWeight => {
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[0]) {
+ return Err(DataFusionError::Plan(format!(
+ "The function {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
+ )));
+ }
+ if !is_approx_percentile_cont_supported_arg_type(&input_types[1]) {
+ return Err(DataFusionError::Plan(format!(
+ "The weight argument for {:?} does not support inputs of type {:?}.",
+ agg_fun, input_types[0]
Review comment:
fixed at 70834726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833834204
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
Review comment:
fixed at 70834726
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 74 |",
+ "| b | 68 |",
+ "| c | 123 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+ Ok(())
+}
+
Review comment:
fixed at 70834726
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r829663723
##########
File path: datafusion-physical-expr/src/expressions/approx_percentile_cont.rs
##########
@@ -194,75 +204,125 @@ pub struct ApproxPercentileAccumulator {
impl ApproxPercentileAccumulator {
pub fn new(percentile: f64, return_type: DataType) -> Self {
Self {
- digest: TDigest::new(100),
+ digest: TDigest::new(DEFAULT_MAX_SIZE),
percentile,
return_type,
}
}
-}
-impl Accumulator for ApproxPercentileAccumulator {
- fn state(&self) -> Result<Vec<ScalarValue>> {
- Ok(self.digest.to_scalar_state())
+ pub(crate) fn merge_digests(&mut self, digests: &[TDigest]) {
+ self.digest = TDigest::merge_digests(digests);
}
- fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- debug_assert_eq!(
- values.len(),
- 1,
- "invalid number of values in batch percentile update"
- );
- let values = &values[0];
-
- self.digest = match values.data_type() {
+ pub(crate) fn convert_to_ordered_float(
+ values: &ArrayRef,
+ ) -> Result<Vec<OrderedFloat<f64>>> {
+ match values.data_type() {
DataType::Float64 => {
let array = values.as_any().downcast_ref::<Float64Array>().unwrap();
- self.digest.merge_unsorted(array.values().iter().cloned())?
+ Ok(array
+ .values()
+ .iter()
+ .filter_map(|v| v.try_as_f64().transpose())
+ .collect::<Result<Vec<_>>>()?)
Review comment:
originally, I want to just return `array.values().iter().cloned()` and then pass to `merge_sorted` as `impl IntoIterator<Item = T>`, but I didn't figure out how to correct `type` it, so I change to `Vec<OrderedFloat<f64>>`
https://github.com/apache/arrow-datafusion/blob/485bfa6ffcf8dc836a482127ecd367d9abfa7248/datafusion-physical-expr/src/tdigest/mod.rs#L252-L255
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
alamb commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1075030108
Hi @jychen7 -- sorry for the delay. I will review this later today
cc @realno and @domodwyer who did previous work in this area.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] houqp commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
houqp commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1080012838
Thanks @jychen7 , cool feature :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on a change in pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on a change in pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833781706
##########
File path: datafusion/tests/sql/aggregates.rs
##########
@@ -476,6 +476,59 @@ async fn csv_query_approx_percentile_cont() -> Result<()> {
Ok(())
}
+#[tokio::test]
+async fn csv_query_approx_percentile_cont_with_weight() -> Result<()> {
+ let mut ctx = SessionContext::new();
+ register_aggregate_csv(&mut ctx).await?;
+
+ // compare approx_percentile_cont and approx_percentile_cont_with_weight
+ let sql = "SELECT c1, approx_percentile_cont(c3, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, 1, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 73 |",
+ "| b | 68 |",
+ "| c | 122 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+
+ let sql = "SELECT c1, approx_percentile_cont_with_weight(c3, c2, 0.95) AS c3_p95 FROM aggregate_test_100 GROUP BY 1 ORDER BY 1";
+ let actual = execute_to_batches(&ctx, sql).await;
+ let expected = vec![
+ "+----+--------+",
+ "| c1 | c3_p95 |",
+ "+----+--------+",
+ "| a | 74 |",
+ "| b | 68 |",
+ "| c | 123 |",
+ "| d | 124 |",
+ "| e | 115 |",
+ "+----+--------+",
+ ];
+ assert_batches_eq!(expected, &actual);
+ Ok(())
+}
+
Review comment:
yes, let me add error cases in https://github.com/apache/arrow-datafusion/pull/2031#discussion_r833780479
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] jychen7 commented on pull request #2031: #2004 approx percentile with weight
Posted by GitBox <gi...@apache.org>.
jychen7 commented on pull request #2031:
URL: https://github.com/apache/arrow-datafusion/pull/2031#issuecomment-1076895570
> I wonder if we could copy the description somewhere -- perhaps in the docstrings
how about adding a user guide about approximate function? so we can explain how to use them and why
https://github.com/apache/arrow-datafusion/tree/master/docs/source/user-guide/sql
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org