You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/12/02 19:40:10 UTC
[GitHub] [arrow-datafusion] alamb opened a new pull request, #4488: Fix panic in median "AggregateState is not a scalar aggregate"
alamb opened a new pull request, #4488:
URL: https://github.com/apache/arrow-datafusion/pull/4488
# Which issue does this PR close?
closes https://github.com/apache/arrow-datafusion/issues/3105
# Rationale for this change
I think median as currently implemented is unusable except for everything but the simplest of queries (where there is no intermediate repartitioning)
Median was the only aggregate that stored it state using `AggregateState::Array` while all other aggregages use `AggregateState::Scalar`
It turns out that `AggregateState::Array` doesn't support partial aggregates where partial state has to be marshalled into an `Array` and combined. Thus, `median` likely never worked for any DataFusion plan that has more than one partition (where the data is repartitioned for parallel aggregation).
# What changes are included in this PR?
I tried a few ways to implement partial state for `AggregateState::Array` and they all got messy very quickly.
Thus, I opted for what I think is the simpler, possibly less performant, approach of using Scalars rather than Arrays.
Median was added in https://github.com/apache/arrow-datafusion/pull/3009 by @andygrove (and sadly I think I have have suggested the array based approach without fully understanding the implications)
For any real world dataset where the performance different between approaches would be measurable, I believe the the current implementation is going to error anyways.
# Are these changes tested?
Yes
# Are there any user-facing changes?
Less error
# Proposed follow on
If this approach is accepted, I propose removing the AggregateState enum and update the Accumulator trait to simplify it as a follow on PR.
cc @andygrove as he originally wrote this code in https://github.com/apache/arrow-datafusion/pull/3009
cc @tustvold as I think he is thinking about how to improve the grouping situation overall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4488: Fix panic in median "AggregateState is not a scalar aggregate"
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4488:
URL: https://github.com/apache/arrow-datafusion/pull/4488#discussion_r1044345607
##########
datafusion/core/tests/sqllogictests/test_files/aggregate.slt:
##########
@@ -216,6 +216,70 @@ SELECT approx_median(a) FROM median_f64_nan
----
NaN
+# median_multi
Review Comment:
I ported the tests to sqllogictest as much of the rest of the aggregate tests had been ported too
##########
datafusion/physical-expr/src/aggregate/median.rs:
##########
@@ -91,157 +91,124 @@ impl AggregateExpr for Median {
}
#[derive(Debug)]
+/// The median accumulator accumulates the raw input values
+/// as `ScalarValue`s
+///
+/// The intermediate state is represented as a List of those scalars
struct MedianAccumulator {
data_type: DataType,
- all_values: Vec<ArrayRef>,
-}
-
-macro_rules! median {
- ($SELF:ident, $TY:ty, $SCALAR_TY:ident, $TWO:expr) => {{
- let combined = combine_arrays::<$TY>($SELF.all_values.as_slice())?;
- if combined.is_empty() {
- return Ok(ScalarValue::Null);
- }
- let sorted = sort(&combined, None)?;
- let array = as_primitive_array::<$TY>(&sorted)?;
- let len = sorted.len();
- let mid = len / 2;
- if len % 2 == 0 {
- Ok(ScalarValue::$SCALAR_TY(Some(
- (array.value(mid - 1) + array.value(mid)) / $TWO,
- )))
- } else {
- Ok(ScalarValue::$SCALAR_TY(Some(array.value(mid))))
- }
- }};
+ all_values: Vec<ScalarValue>,
}
impl Accumulator for MedianAccumulator {
fn state(&self) -> Result<Vec<AggregateState>> {
- let mut vec: Vec<AggregateState> = self
- .all_values
- .iter()
- .map(|v| AggregateState::Array(v.clone()))
- .collect();
- if vec.is_empty() {
- match self.data_type {
- DataType::UInt8 => vec.push(empty_array::<UInt8Type>()),
- DataType::UInt16 => vec.push(empty_array::<UInt16Type>()),
- DataType::UInt32 => vec.push(empty_array::<UInt32Type>()),
- DataType::UInt64 => vec.push(empty_array::<UInt64Type>()),
- DataType::Int8 => vec.push(empty_array::<Int8Type>()),
- DataType::Int16 => vec.push(empty_array::<Int16Type>()),
- DataType::Int32 => vec.push(empty_array::<Int32Type>()),
- DataType::Int64 => vec.push(empty_array::<Int64Type>()),
- DataType::Float32 => vec.push(empty_array::<Float32Type>()),
- DataType::Float64 => vec.push(empty_array::<Float64Type>()),
- _ => {
- return Err(DataFusionError::Execution(
- "unsupported data type for median".to_string(),
- ))
- }
- }
- }
- Ok(vec)
+ let state =
+ ScalarValue::new_list(Some(self.all_values.clone()), self.data_type.clone());
+ Ok(vec![AggregateState::Scalar(state)])
}
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- let x = values[0].clone();
- self.all_values.extend_from_slice(&[x]);
- Ok(())
- }
+ assert_eq!(values.len(), 1);
+ let array = &values[0];
- fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
- for array in states {
- self.all_values.extend_from_slice(&[array.clone()]);
+ self.all_values.reserve(self.all_values.len() + array.len());
Review Comment:
For what it is worth, this assert fails -- the correct assertion is
```rust
assert_eq!(array.data_type(), &self.data_type);
```
Which I have fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb merged pull request #4488: Fix panic in median "AggregateState is not a scalar aggregate"
Posted by GitBox <gi...@apache.org>.
alamb merged PR #4488:
URL: https://github.com/apache/arrow-datafusion/pull/4488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on a diff in pull request #4488: Fix panic in median "AggregateState is not a scalar aggregate"
Posted by GitBox <gi...@apache.org>.
alamb commented on code in PR #4488:
URL: https://github.com/apache/arrow-datafusion/pull/4488#discussion_r1038470488
##########
datafusion/core/tests/sql/aggregates.rs:
##########
@@ -436,6 +436,90 @@ async fn median_test(
Ok(())
}
+#[tokio::test]
+// test case for https://github.com/apache/arrow-datafusion/issues/3105
Review Comment:
all of the tests added in this file panic on master
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] tustvold commented on a diff in pull request #4488: Fix panic in median "AggregateState is not a scalar aggregate"
Posted by GitBox <gi...@apache.org>.
tustvold commented on code in PR #4488:
URL: https://github.com/apache/arrow-datafusion/pull/4488#discussion_r1043705396
##########
datafusion/common/src/scalar.rs:
##########
@@ -720,7 +720,7 @@ impl std::hash::Hash for ScalarValue {
/// dictionary array
#[inline]
fn get_dict_value<K: ArrowDictionaryKeyType>(
- array: &ArrayRef,
+ array: &dyn Array,
Review Comment:
:heart:
##########
datafusion/physical-expr/src/aggregate/median.rs:
##########
@@ -91,157 +91,124 @@ impl AggregateExpr for Median {
}
#[derive(Debug)]
+/// The median accumulator accumulates the raw input values
+/// as `ScalarValue`s
+///
+/// The intermediate state is represented as a List of those scalars
struct MedianAccumulator {
data_type: DataType,
- all_values: Vec<ArrayRef>,
-}
-
-macro_rules! median {
- ($SELF:ident, $TY:ty, $SCALAR_TY:ident, $TWO:expr) => {{
- let combined = combine_arrays::<$TY>($SELF.all_values.as_slice())?;
- if combined.is_empty() {
- return Ok(ScalarValue::Null);
- }
- let sorted = sort(&combined, None)?;
- let array = as_primitive_array::<$TY>(&sorted)?;
- let len = sorted.len();
- let mid = len / 2;
- if len % 2 == 0 {
- Ok(ScalarValue::$SCALAR_TY(Some(
- (array.value(mid - 1) + array.value(mid)) / $TWO,
- )))
- } else {
- Ok(ScalarValue::$SCALAR_TY(Some(array.value(mid))))
- }
- }};
+ all_values: Vec<ScalarValue>,
}
impl Accumulator for MedianAccumulator {
fn state(&self) -> Result<Vec<AggregateState>> {
- let mut vec: Vec<AggregateState> = self
- .all_values
- .iter()
- .map(|v| AggregateState::Array(v.clone()))
- .collect();
- if vec.is_empty() {
- match self.data_type {
- DataType::UInt8 => vec.push(empty_array::<UInt8Type>()),
- DataType::UInt16 => vec.push(empty_array::<UInt16Type>()),
- DataType::UInt32 => vec.push(empty_array::<UInt32Type>()),
- DataType::UInt64 => vec.push(empty_array::<UInt64Type>()),
- DataType::Int8 => vec.push(empty_array::<Int8Type>()),
- DataType::Int16 => vec.push(empty_array::<Int16Type>()),
- DataType::Int32 => vec.push(empty_array::<Int32Type>()),
- DataType::Int64 => vec.push(empty_array::<Int64Type>()),
- DataType::Float32 => vec.push(empty_array::<Float32Type>()),
- DataType::Float64 => vec.push(empty_array::<Float64Type>()),
- _ => {
- return Err(DataFusionError::Execution(
- "unsupported data type for median".to_string(),
- ))
- }
- }
- }
- Ok(vec)
+ let state =
+ ScalarValue::new_list(Some(self.all_values.clone()), self.data_type.clone());
+ Ok(vec![AggregateState::Scalar(state)])
}
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- let x = values[0].clone();
- self.all_values.extend_from_slice(&[x]);
- Ok(())
- }
+ assert_eq!(values.len(), 1);
+ let array = &values[0];
- fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
- for array in states {
- self.all_values.extend_from_slice(&[array.clone()]);
+ self.all_values.reserve(self.all_values.len() + array.len());
Review Comment:
```suggestion
assert!(matches!(array.data_type(), DataType::List(_)));
self.all_values.reserve(self.all_values.len() + array.len());
```
?
##########
datafusion/physical-expr/src/aggregate/median.rs:
##########
@@ -91,157 +91,124 @@ impl AggregateExpr for Median {
}
#[derive(Debug)]
+/// The median accumulator accumulates the raw input values
+/// as `ScalarValue`s
+///
+/// The intermediate state is represented as a List of those scalars
struct MedianAccumulator {
data_type: DataType,
- all_values: Vec<ArrayRef>,
-}
-
-macro_rules! median {
- ($SELF:ident, $TY:ty, $SCALAR_TY:ident, $TWO:expr) => {{
- let combined = combine_arrays::<$TY>($SELF.all_values.as_slice())?;
- if combined.is_empty() {
- return Ok(ScalarValue::Null);
- }
- let sorted = sort(&combined, None)?;
- let array = as_primitive_array::<$TY>(&sorted)?;
- let len = sorted.len();
- let mid = len / 2;
- if len % 2 == 0 {
- Ok(ScalarValue::$SCALAR_TY(Some(
- (array.value(mid - 1) + array.value(mid)) / $TWO,
- )))
- } else {
- Ok(ScalarValue::$SCALAR_TY(Some(array.value(mid))))
- }
- }};
+ all_values: Vec<ScalarValue>,
}
impl Accumulator for MedianAccumulator {
fn state(&self) -> Result<Vec<AggregateState>> {
- let mut vec: Vec<AggregateState> = self
- .all_values
- .iter()
- .map(|v| AggregateState::Array(v.clone()))
- .collect();
- if vec.is_empty() {
- match self.data_type {
- DataType::UInt8 => vec.push(empty_array::<UInt8Type>()),
- DataType::UInt16 => vec.push(empty_array::<UInt16Type>()),
- DataType::UInt32 => vec.push(empty_array::<UInt32Type>()),
- DataType::UInt64 => vec.push(empty_array::<UInt64Type>()),
- DataType::Int8 => vec.push(empty_array::<Int8Type>()),
- DataType::Int16 => vec.push(empty_array::<Int16Type>()),
- DataType::Int32 => vec.push(empty_array::<Int32Type>()),
- DataType::Int64 => vec.push(empty_array::<Int64Type>()),
- DataType::Float32 => vec.push(empty_array::<Float32Type>()),
- DataType::Float64 => vec.push(empty_array::<Float64Type>()),
- _ => {
- return Err(DataFusionError::Execution(
- "unsupported data type for median".to_string(),
- ))
- }
- }
- }
- Ok(vec)
+ let state =
+ ScalarValue::new_list(Some(self.all_values.clone()), self.data_type.clone());
+ Ok(vec![AggregateState::Scalar(state)])
}
fn update_batch(&mut self, values: &[ArrayRef]) -> Result<()> {
- let x = values[0].clone();
- self.all_values.extend_from_slice(&[x]);
- Ok(())
- }
+ assert_eq!(values.len(), 1);
+ let array = &values[0];
- fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
- for array in states {
- self.all_values.extend_from_slice(&[array.clone()]);
+ self.all_values.reserve(self.all_values.len() + array.len());
+ for index in 0..array.len() {
+ self.all_values
+ .push(ScalarValue::try_from_array(array, index)?);
}
+
Ok(())
}
- fn evaluate(&self) -> Result<ScalarValue> {
- match self.all_values[0].data_type() {
- DataType::Int8 => median!(self, arrow::datatypes::Int8Type, Int8, 2),
- DataType::Int16 => median!(self, arrow::datatypes::Int16Type, Int16, 2),
- DataType::Int32 => median!(self, arrow::datatypes::Int32Type, Int32, 2),
- DataType::Int64 => median!(self, arrow::datatypes::Int64Type, Int64, 2),
- DataType::UInt8 => median!(self, arrow::datatypes::UInt8Type, UInt8, 2),
- DataType::UInt16 => median!(self, arrow::datatypes::UInt16Type, UInt16, 2),
- DataType::UInt32 => median!(self, arrow::datatypes::UInt32Type, UInt32, 2),
- DataType::UInt64 => median!(self, arrow::datatypes::UInt64Type, UInt64, 2),
- DataType::Float32 => {
- median!(self, arrow::datatypes::Float32Type, Float32, 2_f32)
- }
- DataType::Float64 => {
- median!(self, arrow::datatypes::Float64Type, Float64, 2_f64)
+ fn merge_batch(&mut self, states: &[ArrayRef]) -> Result<()> {
+ assert_eq!(states.len(), 1);
+
+ let array = &states[0];
+ for index in 0..array.len() {
Review Comment:
```suggestion
assert!(matches!(array.data_type(), DataType::List(_)));
for index in 0..array.len() {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org