You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "khwilson (via GitHub)" <gi...@apache.org> on 2024/09/23 02:05:04 UTC

[PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

khwilson opened a new pull request, #44184:
URL: https://github.com/apache/arrow/pull/44184

   ### Rationale for this change
   
   As documented in #35166, when Arrow performs a sum, product, or mean of an array of type `decimalX(P, S)`, it returns a scalar of type `decimalX(P, S)`. This is true even if the aggregate does not fit in the specified precision. For instance, a sum of two `decimal128(1, 0)`'s such as `1 + 9` is a `decimal128(2, 0)`. But (in Python):
   
   ```python
   import pyarrow as pa
   from decimal import Decimal
   
   arr = pa.array([Decimal("1"), Decimal("9")], type=pa.decimal128(1, 0))
   assert arr.sum().type == pa.decimal128(1, 0)
   ```
   
   This is recognized in the rules for _binary_ addition and multiplication of decimals (see footnote 1 in [this section](https://arrow.apache.org/docs/cpp/compute.html#arithmetic-functions)), but this does not apply to array aggregates.
   
   In #35166 I did a bit of research following a question from @westonpace , and it seems that there's no standard approach to this across DBMS's, but a common solution is to set the precision of the result of a sum to the maximum possible precision of the underlying type. That is, a sum of `decimal128(1, 0)`'s becomes a `decimal128(38, 0)`.
   
   However, products and means differ further. For instance, in both instances, duckdb converts a decimal to a double, which makes sense as the precision of the product of an array of decimals would likely be huge, e.g., an array of size N with precision 2 decimals would have precision at least 2^N.
   
   This PR implements the minimum possible change: replace all return types of a product, sum, or mean aggregate of `decimal128(P, S)` to `decimal128(38, S)` and `decimal256(P, S)` to `decimal256(76, S)`.
   
   Please note, this PR is not done (see the checklist below), as it would be good to get feedback on the following before going through the whole checklist:
   * Is this the correct change to make (especially for products and means); and
   * The implementation relies on overriding "out types," which is how the current mean implementation works, but perhaps there's a better way to approach this.
   
   ### What changes are included in this PR?
   
   - [x] Update C++ kernels to support the change
   - [ ] Update docs to reflect the change
   - [ ] Fix tests in the languages that depend on the C++ engine
   - [ ] Determine if there are other languages which do _not_ depend on the C++ engine which should also be updated
   
   ### Are these changes tested?
   
   They are tested in the following implementations:
   
   - [x] Python
   - [ ] C++
   - [ ] Java
   
   ### Are there any user-facing changes?
   
   Yes. This changes the return type of a scalar aggregate of decimals.
   
   **This PR includes breaking changes to public APIs.**
   
   Specifically, the return type of a scalar aggregate of decimals changes. This is unlikely to break downstream applications as the underlying data has not changed, but if an application relies on the (incorrect!) type information for some reason, it would break.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2400998654

   Hi @zeroshade just checking in! Thanks again for taking a look


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2181923252


##########
docs/source/cpp/compute.rst:
##########
@@ -276,7 +275,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the precision for its width. For instance, an array

Review Comment:
   I believe this is a typo. It was initially modified in [`7c69607` (#44184)](https://github.com/apache/arrow/pull/44184/commits/7c696075bf83e34e7778394b9fadb5445d12bb3f) the same way as bullet `(9)`. Later on we agreed to NOT change `mean` behavior in this PR, thus it was subsequently reverted in [`41fafbf` (#44184)](https://github.com/apache/arrow/pull/44184/commits/41fafbf98e57dfb045d5e5d1d02f454cd541fc42#diff-ce5b94577014735990903d3d03bd4ea4b8c8e6d32f5227592e60b7dd6a912d59) however it was not fully done.
   
   Let's revert it back to the very baseline. 



##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   This actually made me notice another nit: bullet `(9)` applies to both `product` and `sum`, thus here we have to differentiate `sum` from `product` for the precision promotion. Do you think it's better to move the promotion description (for `sum`) to an individual bullet, say `(13)`, and append `(13)` to the "Notes" for `sum`, like: `sum, ..., (9) (13)`.



##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   I'm not sure what you mean. The example shows `decimal128(3, 2)` promoted to `decimal128(38, 2)`.
   
   The scale remains though, if that is what you are confused about?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2148076532


##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2956606467

   Thank you @khwilson for the update. I still think it worth to further refine the compile-time constant into runtime check. I create a little patch below, does that look reasonable to you? Thanks.
   
   ```diff
   diff --git a/cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc b/cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc
   index 31c5ca9a42..f5feb17bf2 100644
   --- a/cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc
   +++ b/cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc
   @@ -157,7 +157,7 @@ struct NullSumImpl : public NullImpl<ArrowType> {
      }
    };
    
   -template <template <typename> class KernelClass, bool PromoteDecimal = true>
   +template <template <typename> class KernelClass>
    struct SumLikeInit {
      std::unique_ptr<KernelState> state;
      KernelContext* ctx;
   @@ -191,7 +191,7 @@ struct SumLikeInit {
      /// However, this may not be the desired behaviour (see, e.g., MeanKernelInit)
      template <typename Type>
      enable_if_decimal<Type, Status> Visit(const Type&) {
   -    if constexpr (PromoteDecimal) {
   +    if (PromoteDecimal()) {
          ARROW_ASSIGN_OR_RAISE(auto ty, WidenDecimalToMaxPrecision(type));
          state.reset(new KernelClass<Type>(ty, options));
          return Status::OK();
   @@ -210,6 +210,11 @@ struct SumLikeInit {
        ARROW_RETURN_NOT_OK(VisitTypeInline(*type, this));
        return std::move(state);
      }
   +
   +  /// Wether the resulting decimal type should be widened to the maximum precision.
   +  /// XXX Ideally this should be able to be configured in the function options, e.g.,
   +  /// enum PrecisionPolicy {PROMOTE_TO_MAX, DEMOTE_TO_DOUBLE, NO_PROMOTION};
   +  virtual bool PromoteDecimal() const { return true; }
    };
    
    // ----------------------------------------------------------------------
   @@ -278,15 +283,17 @@ struct MeanImpl<ArrowType, SimdLevel,
    };
    
    template <template <typename> class KernelClass>
   -struct MeanKernelInit : public SumLikeInit<KernelClass, /*PromoteDecimal=*/false> {
   +struct MeanKernelInit : public SumLikeInit<KernelClass> {
      MeanKernelInit(KernelContext* ctx, std::shared_ptr<DataType> type,
                     const ScalarAggregateOptions& options)
   -      : SumLikeInit<KernelClass, /*PromoteDecimal=*/false>(ctx, type, options) {}
   +      : SumLikeInit<KernelClass>(ctx, type, options) {}
    
      Status Visit(const NullType&) override {
        this->state.reset(new NullSumImpl<DoubleType>(this->options));
        return Status::OK();
      }
   +
   +  bool PromoteDecimal() const override { return false; }
    };
    
    // ----------------------------------------------------------------------
   diff --git a/cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc b/cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc
   index bdb1db7c92..81f017679e 100644
   --- a/cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc
   +++ b/cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc
   @@ -42,8 +42,7 @@ namespace {
    // Sum/Mean/Product implementation
    
    template <typename Type, typename Impl,
   -          typename AccumulateType = typename FindAccumulatorType<Type>::Type,
   -          bool PromoteDecimal = false>
   +          typename AccumulateType = typename FindAccumulatorType<Type>::Type>
    struct GroupedReducingAggregator : public GroupedAggregator {
      using AccType = AccumulateType;
      using CType = typename TypeTraits<AccType>::CType;
   @@ -86,8 +85,7 @@ struct GroupedReducingAggregator : public GroupedAggregator {
      Status Merge(GroupedAggregator&& raw_other,
                   const ArrayData& group_id_mapping) override {
        auto other =
   -        checked_cast<GroupedReducingAggregator<Type, Impl, AccType, PromoteDecimal>*>(
   -            &raw_other);
   +        checked_cast<GroupedReducingAggregator<Type, Impl, AccType>*>(&raw_other);
    
        CType* reduced = reduced_.mutable_data();
        int64_t* counts = counts_.mutable_data();
   @@ -156,21 +154,26 @@ struct GroupedReducingAggregator : public GroupedAggregator {
      std::shared_ptr<DataType> out_type() const override { return out_type_; }
    
      template <typename T = Type>
   -  static enable_if_t<!is_decimal_type<T>::value, Result<std::shared_ptr<DataType>>>
   -  GetOutType(const std::shared_ptr<DataType>& in_type) {
   +  enable_if_t<!is_decimal_type<T>::value, Result<std::shared_ptr<DataType>>> GetOutType(
   +      const std::shared_ptr<DataType>& in_type) {
        return TypeTraits<AccType>::type_singleton();
      }
    
      template <typename T = Type>
   -  static enable_if_decimal<T, Result<std::shared_ptr<DataType>>> GetOutType(
   +  enable_if_decimal<T, Result<std::shared_ptr<DataType>>> GetOutType(
          const std::shared_ptr<DataType>& in_type) {
   -    if constexpr (PromoteDecimal) {
   +    if (PromoteDecimal()) {
          return WidenDecimalToMaxPrecision(in_type);
        } else {
          return in_type;
        }
      }
    
   +  /// Wether the resulting decimal type should be widened to the maximum precision.
   +  /// XXX Ideally this should be able to be configured in the function options, e.g.,
   +  /// enum PrecisionPolicy {PROMOTE_TO_MAX, DEMOTE_TO_DOUBLE, NO_PROMOTION};
   +  virtual bool PromoteDecimal() const { return false; }
   +
      int64_t num_groups_ = 0;
      ScalarAggregateOptions options_;
      TypedBufferBuilder<CType> reduced_;
   @@ -267,11 +270,9 @@ struct GroupedReducingFactory {
    template <typename Type>
    struct GroupedSumImpl
        : public GroupedReducingAggregator<Type, GroupedSumImpl<Type>,
   -                                       typename FindAccumulatorType<Type>::Type,
   -                                       /*PromoteDecimal=*/true> {
   +                                       typename FindAccumulatorType<Type>::Type> {
      using Base = GroupedReducingAggregator<Type, GroupedSumImpl<Type>,
   -                                         typename FindAccumulatorType<Type>::Type,
   -                                         /*PromoteDecimal=*/true>;
   +                                         typename FindAccumulatorType<Type>::Type>;
      using CType = typename Base::CType;
      using InputCType = typename Base::InputCType;
    
   @@ -288,6 +289,8 @@ struct GroupedSumImpl
        return static_cast<CType>(to_unsigned(u) + to_unsigned(v));
      }
    
   +  bool PromoteDecimal() const override { return true; }
   +
      using Base::Finish;
    };
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137115772


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -528,6 +533,28 @@ Status CastDecimalArgs(TypeHolder* begin, size_t count) {
   return Status::OK();
 }
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(
+    std::shared_ptr<DataType> type) {
+  DCHECK(is_decimal(type->id()));

Review Comment:
   I think this is an invariant that this function is not supposed to accept a non-decimal type argument, thus the DCHECK.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2495541780

   Hi @pitrou sorry for dropping the ball on this a bit earlier. Are you still interested in merging this at some point? I'm happy to explore some alternative options for how to do the decimal expansion, or alternatively am happy to push #37536 forward


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1793212262


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -76,6 +76,26 @@ Result<TypeHolder> ListValuesType(KernelContext* ctx,
   return value_type;
 }
 
+Result<TypeHolder> MaxPrecisionDecimalType(KernelContext*,
+                                           const std::vector<TypeHolder>& args) {
+  std::shared_ptr<DataType> out_type_;
+  auto type_id = args[0].type->id();
+  if (type_id == Type::DECIMAL128 || type_id == Type::DECIMAL256) {
+    auto base_type_ = checked_cast<const DecimalType*>(args[0].type);
+    if (type_id == Type::DECIMAL128) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal128Type::Make(Decimal128Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    } else {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal256Type::Make(Decimal256Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    }
+  } else {
+    return Status::TypeError(
+        "A call to MaxPrecisionDecimalType was made with a non-DecimalType");

Review Comment:
   would you mind point out the type here?



##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -55,6 +55,7 @@ using internal::BinaryBitBlockCounter;
 using internal::BitBlockCount;
 using internal::BitmapReader;
 using internal::checked_cast;
+using internal::checked_pointer_cast;

Review Comment:
   why in header?



##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -1380,6 +1383,8 @@ Status CastBinaryDecimalArgs(DecimalPromotion promotion, std::vector<TypeHolder>
 ARROW_EXPORT
 Status CastDecimalArgs(TypeHolder* begin, size_t count);
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(std::shared_ptr<DataType> type);

Review Comment:
   Should this being exported?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2407309983

   I enumerated several here: https://github.com/apache/arrow/issues/35166#issuecomment-2336776704
   
   Clickhouse for instance just ignores precision.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2407350960

   Would you mind making this Ready for review?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2367116722

   :warning: GitHub issue #35166 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2415122098

   > Two problems with just validating afterward: First, I'd expect in reasonable cases for the validation to fail. A sum of 1m decimals of approximately the same size you'd expect to have 6 more digits of precision.
   
   It depends obviously if all decimals are of the same sign, and what their actual magnitude is. 
   
   > Second, just checking for overflow doesn't solve the underlying problem.
   
   In the example above, I used a `validate` call simply to show that the result was indeed erroneous. I didn't mean we should actually call validation afterwards. We should instead check for overflow at each individual aggregation step (for each add or multiply, for example). This is required _even_ if we were to bump the result's precision to the max.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2843508113

   > @khwilson I'm still not sure this is actually desirable (@zanmato1984 what do you think?).
   
   Most DBMSes do precision promotion for `sum` aggregation, and most promotions are arbitrary. So I think the idea of this issue/PR makes very much sense. I will look further into the implementation later.
    
   > What I think is definitely desirable is to check for overflow when doing sum/mean/product of decimals, regardless of which data type is selected for output.
   
   Agreed, checked versions are necessary.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2074666687


##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -365,11 +365,16 @@ struct ProductImpl : public ScalarAggregator {
   }
 
   Status Finalize(KernelContext*, Datum* out) override {
+    std::shared_ptr<DataType> out_type_ = this->out_type;
+    if (is_decimal(this->out_type->id())) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, WidenDecimalToMaxPrecision(this->out_type));
+    }
+

Review Comment:
   I can't remember exactly why I had to do this. I believe it's related to how the `Visit` function gets called for `SumLikeImpl`. Once the rebase is done I can take a look!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137679315


##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -259,8 +271,11 @@ struct GroupedReducingFactory {
 // Sum implementation
 
 template <typename Type>
-struct GroupedSumImpl : public GroupedReducingAggregator<Type, GroupedSumImpl<Type>> {
-  using Base = GroupedReducingAggregator<Type, GroupedSumImpl<Type>>;
+struct GroupedSumImpl
+    : public GroupedReducingAggregator<Type, GroupedSumImpl<Type>,
+                                       typename FindAccumulatorType<Type>::Type> {

Review Comment:
   Oh, good catch. Initially it was because I had added the `PromoteDecimal` as a template argument, which necessitated spelling out all the arguments. Can remove.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-3031423835

   I pushed several commits to address the latest comments from @pitrou and me. I'll merge after CI is green.
   
   @amoeba do you think we can include this in the 21.0.0 release?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2845440744

   That makes sense. From a sequencing stand point, should the checked versions come first or should we get this through first?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2957120982

   Awesome. Will change!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2957466314

   Done


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2956902740

   Hi Rossi! Thanks for the patch! This is essentially the change that I was considering, but it does make GetOutType non-static. I’m not sure of the downstream design consequences of that. My lean was that the refactoring from static to non-static might have other unknown-to-me consequences, but if you are OK with that change, happy to go with it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++][Compute] Increase precision of decimals in sum aggregates [arrow]

Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-3032767953

   After merging your PR, Conbench analyzed the 3 benchmarking runs that have been run so far on merge-commit bf56a956267e5e5432b394f6bc273ccc97424a74.
   
   There were no benchmark performance regressions. 🎉
   
   The [full Conbench report](https://github.com/apache/arrow/runs/45306976086) has more details.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++][Compute] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 merged PR #44184:
URL: https://github.com/apache/arrow/pull/44184


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2074337147


##########
cpp/src/arrow/compute/kernels/hash_aggregate.cc:
##########
@@ -564,7 +564,7 @@ struct GroupedReducingAggregator : public GroupedAggregator {
   template <typename T = Type>
   static enable_if_decimal<T, std::shared_ptr<DataType>> GetOutType(
       const std::shared_ptr<DataType>& in_type) {
-    return in_type;
+    return WidenDecimalToMaxPrecision(in_type).ValueOrDie();

Review Comment:
   I don't think we should widen the output type for all the subclasses of `GroupedReducingAggregator`. Promoting to max precision is reasonable for `sum` but is not necessarily the case for others.



##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -365,11 +365,16 @@ struct ProductImpl : public ScalarAggregator {
   }
 
   Status Finalize(KernelContext*, Datum* out) override {
+    std::shared_ptr<DataType> out_type_ = this->out_type;
+    if (is_decimal(this->out_type->id())) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, WidenDecimalToMaxPrecision(this->out_type));
+    }
+

Review Comment:
   This is hijacking the output type at the execution time. However we should be consistent between the function signature (the return type determined at the function resolution) and the value actually being returned.
   
   You should probably change the return type somewhere in https://github.com/apache/arrow/blob/067fd2a2c6e54d33b9ae8a3324f59bebe960d485/cpp/src/arrow/compute/kernels/aggregate_basic.cc#L1051-L1054 .
   (BTW, what you did for hash aggregates is correct - widening the precision to the actual output type rather than hijacking the return value at the last minute).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2869954243

   OK @zanmato1984 it took a bit given the way that everything gets set. Updates:
   
   * Removed all the Decimal32/64 logic (except in `WidenDecimalToMaxPrecision`
   * Introduced a `PromoteDecimal` template variable to determine whether a `SumLike` should actually promote the decimal
   * Reverted the Mean and Product promotions as we're not agreed on how to handle those.
   
   As for the mean and product, there's a few actions we could take:
   
   * Status quo: Input type is the same as output type
   * Promote both to maximum precision
   * Promote only _product_ to max precision: `abs(mean) <= max(abs(max), abs(min))` so not really _necessary_ for mean
   * Promote only _product_ to a double: this is what, e.g., duckdb does
   
   I think the best course of action is actually the fourth choice: Promote only product to double and leave mean the same as the input type. Thoughts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2841976994

   @khwilson I'm still not sure this is actually desirable (@zanmato1984 what do you think?).
   
   What I think is definitely desirable is to check for overflow when doing sum/mean/product of decimals, regardless of which data type is selected for output.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137674301


##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   So in the end, the effect of this change is *only* to the metadata. The previous implementation would simply store a `decimal(4, 2)` and call it a `decimal(3, 2)`. So this is the only change.
   
   As mentioned above, I was planning to turn to the checked variants (taking up the mantle of #37536 ) once this PR is through.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137659252


##########
docs/source/cpp/compute.rst:
##########
@@ -171,8 +171,8 @@ To avoid exhaustively listing supported types, the tables below use a number
 of general type categories:
 
 * "Numeric": Integer types (Int8, etc.) and Floating-point types (Float32,
-  Float64, sometimes Float16).  Some functions also accept Decimal128 and
-  Decimal256 input.
+  Float64, sometimes Float16).  Some functions also accept
+  Decimal128/256 input.

Review Comment:
   The initial change to this line was to say that "some functions also accept Decimal32/64/128/256 input" but that change was rolled back to make this PR focus on Decimal128/256. Hence the change here. But this seems fine.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2129530668


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -528,6 +533,28 @@ Status CastDecimalArgs(TypeHolder* begin, size_t count) {
   return Status::OK();
 }
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(

Review Comment:
   Might want to add some test cases for this function in `codegen_internal_test.cc`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2877649750

   Meanwhile this PR is well polished to be precise and focus on precision promotion for sum. I'm reviewing the implementation now. Thanks @khwilson .


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2409115768

   @mapleFU I believe this is done now. Some notes on the diff:
   
   * The hash aggregates had to be updated as well (missed them in the first pass)
   * I've also added in Decimal32/64 support the basic aggregates (sum, product, mean, min/max, index). However, there's quite a lot of missing support for these types still in compute (most notably in casts)
   * Docs are updated to reflect the change
   
   And a note that quite a few tests are failing for what appears to be the same reason as #41390. Happy to address them if you'd like.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1798549797


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -76,6 +76,26 @@ Result<TypeHolder> ListValuesType(KernelContext* ctx,
   return value_type;
 }
 
+Result<TypeHolder> MaxPrecisionDecimalType(KernelContext*,
+                                           const std::vector<TypeHolder>& args) {
+  std::shared_ptr<DataType> out_type_;
+  auto type_id = args[0].type->id();
+  if (type_id == Type::DECIMAL128 || type_id == Type::DECIMAL256) {
+    auto base_type_ = checked_cast<const DecimalType*>(args[0].type);
+    if (type_id == Type::DECIMAL128) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal128Type::Make(Decimal128Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    } else {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal256Type::Make(Decimal256Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    }
+  } else {
+    return Status::TypeError(
+        "A call to MaxPrecisionDecimalType was made with a non-DecimalType");

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2414437715

   I'm lukewarm about the approach here. Silently casting to the max precision discards metadata about the input; it also risks producing errors further down the line (if e.g. the max precision is deemed too large for other operations). It also doesn't automatically eliminate any potential overflow, for example:
   ```python
   >>> a = pa.array([789.3] * 20).cast(pa.decimal128(38, 35))
   >>> a
   <pyarrow.lib.Decimal128Array object at 0x7f0f103ca7a0>
   [
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440,
     789.29999999999995452526491135358810440
   ]
   >>> pc.sum(a)
   <pyarrow.Decimal128Scalar: Decimal('-1228.11834604692408266343214451664848480')>
   ```
   
   We should instead check that the result of an aggregate fits into the resulting Decimal type, while overflows currently pass silently:
   ```python
   >>> a = pa.array([123., 456., 789.]).cast(pa.decimal128(4, 1))
   >>> a
   <pyarrow.lib.Decimal128Array object at 0x7f0ed06261a0>
   [
     123.0,
     456.0,
     789.0
   ]
   >>> pc.sum(a)
   <pyarrow.Decimal128Scalar: Decimal('1368.0')>
   >>> pc.sum(a).validate(full=True)
   Traceback (most recent call last):
     ...
   ArrowInvalid: Decimal value 13680 does not fit in precision of decimal128(4, 1)
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2180374979


##########
docs/source/cpp/compute.rst:
##########
@@ -276,7 +275,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the precision for its width. For instance, an array

Review Comment:
   Or is my suggestion not correct? In any case, "the precision for its width" reads weird.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2180372067


##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -154,17 +154,29 @@ struct GroupedReducingAggregator : public GroupedAggregator {
   std::shared_ptr<DataType> out_type() const override { return out_type_; }
 
   template <typename T = Type>
-  static enable_if_t<!is_decimal_type<T>::value, std::shared_ptr<DataType>> GetOutType(
+  enable_if_t<!is_decimal_type<T>::value, Result<std::shared_ptr<DataType>>> GetOutType(
       const std::shared_ptr<DataType>& in_type) {
     return TypeTraits<AccType>::type_singleton();
   }
 
   template <typename T = Type>
-  static enable_if_decimal<T, std::shared_ptr<DataType>> GetOutType(
+  enable_if_decimal<T, Result<std::shared_ptr<DataType>>> GetOutType(
       const std::shared_ptr<DataType>& in_type) {
-    return in_type;
+    if (PromoteDecimal()) {
+      return WidenDecimalToMaxPrecision(in_type);
+    } else {
+      return in_type;
+    }
   }
 
+  /// If this returns true, then the aggregator will promote a decimal

Review Comment:
   Kind of a nit, but it might be better to avoid docstring notation (`///`) in `.cc` files because those are not APIs for which we want to expose a documentation.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2181949079


##########
docs/source/cpp/compute.rst:
##########
@@ -276,7 +275,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the precision for its width. For instance, an array
+  of ``decimal128(3, 2)`` will return a ``decimal128(3, 2)`` scalar.
+  The result is rounded away from zero.

Review Comment:
   ```suggestion
     precision and scale. The result is rounded away from zero.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1794464330


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -76,6 +76,26 @@ Result<TypeHolder> ListValuesType(KernelContext* ctx,
   return value_type;
 }
 
+Result<TypeHolder> MaxPrecisionDecimalType(KernelContext*,
+                                           const std::vector<TypeHolder>& args) {
+  std::shared_ptr<DataType> out_type_;
+  auto type_id = args[0].type->id();
+  if (type_id == Type::DECIMAL128 || type_id == Type::DECIMAL256) {
+    auto base_type_ = checked_cast<const DecimalType*>(args[0].type);
+    if (type_id == Type::DECIMAL128) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal128Type::Make(Decimal128Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    } else {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal256Type::Make(Decimal256Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    }
+  } else {
+    return Status::TypeError(
+        "A call to MaxPrecisionDecimalType was made with a non-DecimalType");

Review Comment:
   Simplified to just call the common `WidenDecimalToMaxPrecision` function which is just the identity on non-Decimal types



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2402404888

   Thanks for the review!
   
   By a user-side cast, do you mean that users should essentially do:
   
   ```sql
   select avg(cast(blah as decimal(big-precision)))
   ```
   
   instead of
   
   ```sql
   select avg(blah)
   ```
   
   or do you mean that this code should "inject" a cast on the "user" side?
   
   If you mean putting the cast onto the user, then I would think you'd want to add an error if the answer can't fit into the default precision, but that seems like it would be more disruptive (and out of step with how other systems handle decimal aggregates).
   
   If you mean "injecting" the cast on the user side, would that end up creating a copy of the array?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2415588449

   Nice! I'm excited for the checked variants of sum and product!
   
   With the integer overflow example, I only meant to point out that the compute module currently allows overflows, so I think it would be unexpected for `sum` to complain about an overflow _only_ if the underlying type was a decimal. But if the goal with #37090 is to _replace_ `sum` with a checked version, then the solution of erroring makes a lot of sense, and I'd be happy to implement it when #37536 gets merged. :-)
   
   Still, I do think that users would find it unexpected to get an error if the sum fit in the underlying storage since this is how all the databases I've used (and the four I surveyed in #35166) have operated.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2954279621

   OK, @zanmato1984 added the tests and updated docs.
   
   For the SFINAE things, a couple considerations:
   * Probably isn't super critical for performance, but...
   * In the grouped aggregators, `GetOutType` is `static` so creating class-level (e.g., `GroupedMeanImpl`) runtime differences would likely require refactoring `GetOutType`.
   
   Given the second bullet, I'd keep it as SFINAE for consistency between the two implementations.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2138018506


##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   There may be a misunderstanding, but I was talking about the situation where the Sum's result would overflow the input type (be it decimal128 or decimal256).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2845401668

   > Cool. I’m happy to do some more clean up or do a little work on creating the checked versions of the aggregates to push this along
   
   Thank you for the contribution and passion. How about we keep this PR focused solely on the precision promotion? And the potential checked version can be done in another PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2845458454

   I think they are well independent so feel free to start anytime. And thank you for willing to take this!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137063623


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -528,6 +533,28 @@ Status CastDecimalArgs(TypeHolder* begin, size_t count) {
   return Status::OK();
 }
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(
+    std::shared_ptr<DataType> type) {
+  DCHECK(is_decimal(type->id()));

Review Comment:
   Error checking here is weird. Why are we using asserts if we're also returning a `Result` in case of error? We should do one or the other.



##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -259,8 +271,11 @@ struct GroupedReducingFactory {
 // Sum implementation
 
 template <typename Type>
-struct GroupedSumImpl : public GroupedReducingAggregator<Type, GroupedSumImpl<Type>> {
-  using Base = GroupedReducingAggregator<Type, GroupedSumImpl<Type>>;
+struct GroupedSumImpl
+    : public GroupedReducingAggregator<Type, GroupedSumImpl<Type>,
+                                       typename FindAccumulatorType<Type>::Type> {

Review Comment:
   I'm curious, why was this change necessary?



##########
docs/source/cpp/compute.rst:
##########
@@ -171,8 +171,8 @@ To avoid exhaustively listing supported types, the tables below use a number
 of general type categories:
 
 * "Numeric": Integer types (Int8, etc.) and Floating-point types (Float32,
-  Float64, sometimes Float16).  Some functions also accept Decimal128 and
-  Decimal256 input.
+  Float64, sometimes Float16).  Some functions also accept
+  Decimal128/256 input.

Review Comment:
   More and more functions now also accept decimal32/decimal64.
   ```suggestion
     Float64, sometimes Float16).  Some functions also accept Decimal input.
   ```



##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   You don't have to test both `arr.sum()` and `pc.sum(arr)` everytime since the former is just calling the latter. This will make the tests more readable and easier to maintain.



##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];

Review Comment:
   Nit: make those const
   ```suggestion
       const auto& ty = init_types[i];
       const auto& out_ty = out_types[i];
   ```



##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   Ok, but can we also test the effects of the added precision? i.e. test a case that would fail with the previous behavior of not increasing precision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1794463683


##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -1380,6 +1383,8 @@ Status CastBinaryDecimalArgs(DecimalPromotion promotion, std::vector<TypeHolder>
 ARROW_EXPORT
 Status CastDecimalArgs(TypeHolder* begin, size_t count);
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(std::shared_ptr<DataType> type);

Review Comment:
   Done!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2406554981

   Generally this method is ok for me, but I'm not so familiar with the "common solutions" here. I'll dive into Presto/ClickHouse


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2414833049

   Two problems with just validating afterward: First, I'd expect in reasonable cases for the validation to fail. A sum of 1m decimals of approximately the same size you'd expect to have 6 more digits of precision. I assume this is why all the DBMSs I looked at increase the precision by default.
   
   Second, just checking for overflow doesn't solve the underlying problem. Consider:
   
   ```python
   a = pa.array([789.3] * 18).cast(pa.decimal128(38, 35))
   print(pc.sum(a))
   pc.sum(a).validate(full=True)  # passes
   ```
   
   In duckdb, they implement an intermediate check to make sure that there's not an internal overflow:
   
   ```python
   tab = pa.Table.from_pydict({"a": a})
   duckdb.query("select sum(a) from tab")
   # Traceback (most recent call last):
   #   File "<stdin>", line 1, in <module>
   # duckdb.duckdb.OutOfRangeException: Out of Range Error: Overflow in HUGEINT addition: 
   # 157859999999999990905052982270717620880 + 78929999999999995452526491135358810440
   ```
   
   Notably, this lack of overflow checking also applies to integer sums in arrow:
   
   ```python
   >>> pa.array([9223372036854775800] * 2, type=pa.int64())
   <pyarrow.lib.Int64Array object at 0x10c1d8b80>
   [
     9223372036854775800,
     9223372036854775800
   ]
   >>> pc.sum(pa.array([9223372036854775800] * 2, type=pa.int64()))
   <pyarrow.Int64Scalar: -16>
   >>> pc.sum(pa.array([9223372036854775800] * 2, type=pa.int64())).validate(full=True)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2415128474

   > Notably, this lack of overflow checking also applies to integer sums in arrow:
   
   Yes, and there's already a bug open for it: https://github.com/apache/arrow/issues/37090
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2134820205


##########
docs/source/cpp/compute.rst:
##########
@@ -264,7 +264,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the maximum precision for its width. For instance, an array

Review Comment:
   Good catch. Missed on the revision.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2148071405


##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2974528816

   OK, all in all:
   * @pitrou removed one of `pc.sum` and `arr.sum` in the python test
   * Added an example of an overflow in the sum tests
   * Made a small modification to the docs to clarify that current changes apply only to sums.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2869958414

   > As for the mean and product, there's a few actions we could take:
   >     * Status quo: Input type is the same as output type
   >     * Promote both to maximum precision
   >     * Promote only _product_ to max precision: `abs(mean) <= max(abs(max), abs(min))` so not really _necessary_ for mean
   >     * Promote only _product_ to a double: this is what, e.g., duckdb does
   
   "Promoting" to double is a demotion: it loses a lot of potential precision. The mean should certainly stay the same as the input type (and care should be taken to avoid overflows during intermediate computations, if at all possible).
   
   As for the product, it would also be better to not lose too much precision, and therefore stay in the decimal domain. But it sounds less common to compute the product of a bunch of decimals, anyway.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2182152519


##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   Yeah I was thinking so too. Just took `(13)` as an example to avoid the trouble of shifting `10+`s.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2074664481


##########
cpp/src/arrow/compute/kernels/hash_aggregate.cc:
##########
@@ -564,7 +564,7 @@ struct GroupedReducingAggregator : public GroupedAggregator {
   template <typename T = Type>
   static enable_if_decimal<T, std::shared_ptr<DataType>> GetOutType(
       const std::shared_ptr<DataType>& in_type) {
-    return in_type;
+    return WidenDecimalToMaxPrecision(in_type).ValueOrDie();

Review Comment:
   Yeah, different databases do different things. E.g., for a mean, duckdb converts to a double (which is also what the array aggregate does for non-decimal numeric types in arrow). Do you have a preference for what the type for the other aggregates _should_ be?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2956930276

   I think it would be less impactful in terms of design consequence if the changes are all in `.cc` file. So we should be good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137971547


##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   Ok, but this PR adds support for situations which were not supported before, so we test for them anyway, no?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137664477


##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   Sure, but as it was, this test tested both `arr.sum()` and `pc.sum(arr)` for each of its subcases _except_ this one. The correct change would be to remove all those checks, which seemed like a larger change than making it consistent.
   
   In the new test, I can just do one or the other, but the goal was to make it consistent with this test.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2874763959

   If I understand the code correctly, what happens for the mean for decimals is that it first computes the sum by using the underlying `+` of the Decimal type, then computes the count of values as a long, and does an integer division of the two. As seen in this commit, that `+` operator essentially ignores precision. I think this is a reasonable implementation except:
   
   * There should be a checked variant;
   * Potentially, you'd want to add some extra scale, in line with the binary division operation.
   
   For comparison, I checked how a few other DBMS's handle the mean:
   * postgresql: Drops all precision and scale information in mean
   * duckdb: Turns into a double
   * mariadb: Appears to just add 4 to scale and precision
   
   For product, it's not very common for databases to provide a product aggregator of any kind. I quickly looked at ANSI SQL, postgres, mysql, and clickhouse, none of which provide a product. The "standard" product implementation for such aggregates is `(2 * sum(col % 2) - 1) * exp(sum(ln(abs(col)))`, but that requires transforming into a double. Moreover, basically any column with at least 100 values will overflow a decimal's precision.
   
   So I'd propose for product either demoting to double (will give a semantically correct value but lose precision) or maxing out the width of the decimal type. Of course, there should also be a checked variant (but I'm deferring that until after this commit is complete).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-3027957444

   @pitrou did you get a chance to look at this again?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2407428127

   Sure!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2403086919

   No problem! Hope your travels were fun!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1798846238


##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -1440,6 +1443,14 @@ Status CastBinaryDecimalArgs(DecimalPromotion promotion, std::vector<TypeHolder>
 ARROW_EXPORT
 Status CastDecimalArgs(TypeHolder* begin, size_t count);
 
+/// Given a DataType, if it is a DecimalType, return a DecimalType with the same scale
+/// and the maximum precision for that DecimalType.
+///
+/// If it is _not_ a DecimalType, return a Status::TypeError

Review Comment:
   ```suggestion
   /// If it is _not_ a DecimalType, returns a Status::TypeError.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1796422657


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -76,6 +76,26 @@ Result<TypeHolder> ListValuesType(KernelContext* ctx,
   return value_type;
 }
 
+Result<TypeHolder> MaxPrecisionDecimalType(KernelContext*,
+                                           const std::vector<TypeHolder>& args) {
+  std::shared_ptr<DataType> out_type_;
+  auto type_id = args[0].type->id();
+  if (type_id == Type::DECIMAL128 || type_id == Type::DECIMAL256) {
+    auto base_type_ = checked_cast<const DecimalType*>(args[0].type);
+    if (type_id == Type::DECIMAL128) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal128Type::Make(Decimal128Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    } else {
+      ARROW_ASSIGN_OR_RAISE(out_type_, Decimal256Type::Make(Decimal256Type::kMaxPrecision,
+                                                            base_type_->scale()));
+    }
+  } else {
+    return Status::TypeError(
+        "A call to MaxPrecisionDecimalType was made with a non-DecimalType");

Review Comment:
   I mean, TypeError and add the type here. Since we've supported decimal32 and decimal64



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2853154231

   OK, I think I've correctly rebased (letting all the tests run now to figure out).
   
   I can remove the Decimal32/64 support once I check if I've rebased correctly.
   
   Will also look at the inline comments once the rebase is completed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2074681131


##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -365,11 +365,16 @@ struct ProductImpl : public ScalarAggregator {
   }
 
   Status Finalize(KernelContext*, Datum* out) override {
+    std::shared_ptr<DataType> out_type_ = this->out_type;
+    if (is_decimal(this->out_type->id())) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, WidenDecimalToMaxPrecision(this->out_type));
+    }
+

Review Comment:
   > The link you have is to a different commit, I believe. If you look at those lines in this PR they do change the type at resolution time.
   
   Ah, I see changing the output type in resolution time. Thanks for clarifying.
   
   Then the question is why changing the output type again during the execution, which I think you are figuring out :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2074659482


##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -365,11 +365,16 @@ struct ProductImpl : public ScalarAggregator {
   }
 
   Status Finalize(KernelContext*, Datum* out) override {
+    std::shared_ptr<DataType> out_type_ = this->out_type;
+    if (is_decimal(this->out_type->id())) {
+      ARROW_ASSIGN_OR_RAISE(out_type_, WidenDecimalToMaxPrecision(this->out_type));
+    }
+

Review Comment:
   The link you have is to a different commit, I believe. If you look at those lines in this PR they do change the type at resolution time.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2877634929

   Hi @khwilson , thanks for the update and the extensive research.
   
   > > Ideally, the scale should be "floating" just as in floating-point arithmetic, depending on the current running sum (the running sum can be very large if all data is positive, or very small if the data is centered around zero). It is then normalized to the original scale at the end. But of course that makes the algorithm more involved.
   > 
   > Yeah, this is quite complicated and essentially means you need to implement the floating point addition algorithm.
   
   +1 that this could be unrealistic for us to implement given the complicity.
   
   > > Either is fine to me.
   > 
   > Cool. @zanmato1984 do you have an opinion?
   
   I don't have an obvious preference on this. (And we don't have to jump to the conclusion too soon, do we?)
   
   I do have an opinion on the ideal case though. As I understand the arrow compute module, it should be a building block  of comprehensive data systems/applications. Therefore it should remain neutral on the application-specific behaviors, esp. the case that no one is obviously superior than others. That is, we probably should supply options for the desired behaviors, something like `enum PrecisionPolicy { PROMOTE_TO_MAX, DEMOTE_TO_DOUBLE, }`, and do the computation accordingly (as long as there's not too much engineering complexity). Of course this is an ultimate goal in the future and shouldn't be the concern of this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2875691410

   > If I understand the code correctly, what happens for the mean for decimals is that it first computes the sum by using the underlying `+` of the Decimal type, then computes the count of values as a long, and does an integer division of the two. As seen in this commit, that `+` operator essentially ignores precision. I think this is a reasonable implementation except:
   >     * There should be a checked variant;
   >     * Potentially, you'd want to add some extra scale, in line with the binary division operation.
   
   Ideally, the scale should be "floating" just as in floating-point arithmetic, depending on the current running sum (the running sum can be very large if all data is positive, or very small if the data is centered around zero). It is then normalized to the original scale at the end. But of course that makes the algorithm more involved.
   
   (that would also eliminate the need for a checked variant?)
   
   > So I'd propose for product either demoting to double (will give a semantically correct value but lose precision) or maxing out the width of the decimal type. Of course, there should also be a checked variant (but I'm deferring that until after this commit is complete).
   
   Either is fine to me. 
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2917055063

   Sorry @khwilson for being late. I wasn't able to make my time for this in last few weeks. I will look into it again in this or next week. Thanks for the patient, appreciate it!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137664477


##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   Sure, but as it was, this test tested both `arr.sum()` and `pc.sum(arr)` for each of its subcases _except_ this one. The correct change would be to remove all those checks, which seemed like a larger change than making it consistent.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2138010388


##########
cpp/src/arrow/compute/kernels/aggregate_test.cc:
##########
@@ -495,51 +495,67 @@ TEST_F(TestSumKernelRoundOff, Basics) {
 }
 
 TEST(TestDecimalSumKernel, SimpleSum) {
-  for (const auto& ty : {decimal128(3, 2), decimal256(3, 2)}) {
+  std::vector<std::shared_ptr<DataType>> init_types = {decimal128(3, 2),
+                                                       decimal256(3, 2)};
+  std::vector<std::shared_ptr<DataType>> out_types = {decimal128(38, 2),
+                                                      decimal256(76, 2)};
+
+  for (size_t i = 0; i < init_types.size(); ++i) {
+    auto& ty = init_types[i];
+    auto& out_ty = out_types[i];
+
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])")),
-        ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     Datum chunks =
         ChunkedArrayFromJSON(ty, {R"(["0.00", "1.01", "2.02", "3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
     chunks = ChunkedArrayFromJSON(
         ty, {R"(["0.00", "1.01", "2.02"])", "[]", R"(["3.03", "4.04", "5.05"])"});
-    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(ty, R"("15.15")")));
+    EXPECT_THAT(Sum(chunks), ResultWith(ScalarFromJSON(out_ty, R"("15.15")")));
 
     ScalarAggregateOptions options(/*skip_nulls=*/true, /*min_count=*/0);
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     EXPECT_THAT(Sum(ArrayFromJSON(ty, R"([null])"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
     chunks = ChunkedArrayFromJSON(ty, {});
-    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+    EXPECT_THAT(Sum(chunks, options), ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));
 
     EXPECT_THAT(
         Sum(ArrayFromJSON(ty, R"(["1.01", null, "3.03", null, "5.05", null, "7.07"])"),
             options),
-        ResultWith(ScalarFromJSON(ty, R"("16.16")")));
+        ResultWith(ScalarFromJSON(out_ty, R"("16.16")")));
 
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"("5.05")")),
-                ResultWith(ScalarFromJSON(ty, R"("5.05")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("5.05")")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)")),
-                ResultWith(ScalarFromJSON(ty, R"(null)")));
+                ResultWith(ScalarFromJSON(out_ty, R"(null)")));
     EXPECT_THAT(Sum(ScalarFromJSON(ty, R"(null)"), options),
-                ResultWith(ScalarFromJSON(ty, R"("0.00")")));
+                ResultWith(ScalarFromJSON(out_ty, R"("0.00")")));

Review Comment:
   I dont think it does. Those situations (decimal32 and 64 support) were removed for a future PR. All these cases were previously supported, we’re just changing the return types



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2119695323


##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -42,7 +42,8 @@ namespace {
 // Sum/Mean/Product implementation
 
 template <typename Type, typename Impl,
-          typename AccumulateType = typename FindAccumulatorType<Type>::Type>
+          typename AccumulateType = typename FindAccumulatorType<Type>::Type,
+          bool PromoteDecimal = false>

Review Comment:
   Ditto.



##########
cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc:
##########
@@ -187,10 +187,18 @@ struct SumLikeInit {
     return Status::OK();
   }
 
+  /// By default, we widen the decimal to max precision for SumLikes
+  /// However, this may not be the desired behaviour (see, e.g., MeanKernelInit)

Review Comment:
   These comments might be unnecessary now?



##########
docs/source/cpp/compute.rst:
##########
@@ -264,7 +264,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the maximum precision for its width. For instance, an array

Review Comment:
   This is for `mean` right? And we didn't promote for `mean` in this PR.



##########
cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc:
##########
@@ -157,7 +157,7 @@ struct NullSumImpl : public NullImpl<ArrowType> {
   }
 };
 
-template <template <typename> class KernelClass>
+template <template <typename> class KernelClass, bool PromoteDecimal = true>

Review Comment:
   Does it have to be a template argument? If this argument is not performance critical, we should make it a runtime one.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2402830848

   @khwilson Hey sorry for the delay from me here, I've been traveling a lot lately for work and have been at ASF Community Over Code this week. I promise i'll get to this soon. In the meantime, you're in the very capable hands of @mapleFU 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2945561960

   Thanks, Rossi! My turn to apologize on tardiness. Will address this weekend.
   
   On Thu, Jun 5, 2025 at 1:41 PM Rossi Sun ***@***.***> wrote:
   
   > ***@***.**** commented on this pull request.
   > ------------------------------
   >
   > In cpp/src/arrow/compute/kernels/codegen_internal.cc
   > <https://github.com/apache/arrow/pull/44184#discussion_r2129530668>:
   >
   > > @@ -528,6 +533,28 @@ Status CastDecimalArgs(TypeHolder* begin, size_t count) {
   >    return Status::OK();
   >  }
   >
   > +Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(
   >
   > Might want to add some test cases for this function in
   > codegen_internal_test.cc.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > <https://github.com/apache/arrow/pull/44184#pullrequestreview-2901370679>,
   > or unsubscribe
   > <https://github.com/notifications/unsubscribe-auth/AALU5EX7XE4NS3F6BSYIL5L3CB6MJAVCNFSM6AAAAABOVDOQEWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDSMBRGM3TANRXHE>
   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2877184646

   > Ideally, the scale should be "floating" just as in floating-point arithmetic, depending on the current running sum (the running sum can be very large if all data is positive, or very small if the data is centered around zero). It is then normalized to the original scale at the end. But of course that makes the algorithm more involved.
   
   Yeah, this is quite complicated and essentially means you need to implement the floating point addition algorithm.
   
   > Either is fine to me.
   
   Cool. @zanmato1984 do you have an opinion?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2134818764


##########
cpp/src/arrow/compute/kernels/aggregate_basic.inc.cc:
##########
@@ -157,7 +157,7 @@ struct NullSumImpl : public NullImpl<ArrowType> {
   }
 };
 
-template <template <typename> class KernelClass>
+template <template <typename> class KernelClass, bool PromoteDecimal = true>

Review Comment:
   In the end, each `Impl` ends up choosing whether it wants to promote the decimal, so the code size consequence should be minimal. OTOH, the performance implication either way should also be minimal. So I'm indifferent, which I guess augurs for a runtime check.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "wirable23 (via GitHub)" <gi...@apache.org>.
wirable23 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-3016636816

   Would this change be part of arrow 21?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2134828228


##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -42,7 +42,8 @@ namespace {
 // Sum/Mean/Product implementation
 
 template <typename Type, typename Impl,
-          typename AccumulateType = typename FindAccumulatorType<Type>::Type>
+          typename AccumulateType = typename FindAccumulatorType<Type>::Type,
+          bool PromoteDecimal = false>

Review Comment:
   Problem here is that `GetOutType` is `static` so creating a runtime check is a bit of a nuisance without a larger refactor. And given that, maybe it's better to be consistent between the two implementations?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2134828228


##########
cpp/src/arrow/compute/kernels/hash_aggregate_numeric.cc:
##########
@@ -42,7 +42,8 @@ namespace {
 // Sum/Mean/Product implementation
 
 template <typename Type, typename Impl,
-          typename AccumulateType = typename FindAccumulatorType<Type>::Type>
+          typename AccumulateType = typename FindAccumulatorType<Type>::Type,
+          bool PromoteDecimal = false>

Review Comment:
   Problem here is that `GetOutType` is `static` so creating a runtime check is a bit of a nuisance without a larger refactor



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2180376421


##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   Is the example wrong or is it just misleading? The previous sentence mentions that the sum will expand the precision to the max.



##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   Is the example wrong or is it just confusing? The previous sentence mentions that the sum will expand the precision to the max.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2180372884


##########
docs/source/cpp/compute.rst:
##########
@@ -276,7 +275,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the precision for its width. For instance, an array

Review Comment:
   ```suggestion
     scale and the max precision for its width. For instance, an array
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2181951079


##########
docs/source/cpp/compute.rst:
##########
@@ -276,7 +275,9 @@ the input to a single output value.
   is always -1, regardless of whether there are nulls in the input.
 
 * \(5) For decimal inputs, the resulting decimal will have the same
-  precision and scale. The result is rounded away from zero.
+  scale and the precision for its width. For instance, an array

Review Comment:
   Now fully reverted.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r1793554649


##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -55,6 +55,7 @@ using internal::BinaryBitBlockCounter;
 using internal::BitBlockCount;
 using internal::BitmapReader;
 using internal::checked_cast;
+using internal::checked_pointer_cast;

Review Comment:
   I was following the pattern of the `internal::checked_cast` on the line above. But I'm not wedded to including it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2182070469


##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   > Do you think it's better to move the promotion description (for sum) to an individual bullet, say (13), and append (13) to the "Notes" for sum, like: sum, ..., (9) (13).
   
   Nitpicky, but can we make the two notes contiguous for better readability? :) For example (9) (10)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++][Compute] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-3031643012

   CI is good, merging.
   
   Thanks a lot @khwilson for contributing this!
   
   @wirable23 Please wait for our release manager's response about into 21.0.0. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2182067470


##########
docs/source/cpp/compute.rst:
##########
@@ -294,8 +295,10 @@ the input to a single output value.
   the values to be pivoted. The output is a Struct with one field for each key
   in :member:`PivotOptions::key_names`.
 
-* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending on the
-  input type.
+* \(9) Output is Int64, UInt64, Float64, or Decimal128/256, depending
+  on the input type. For sums of decimals, the precision is increased to the maximum
+  precision for the type's width. For instance, an array of
+  ``decimal128(3, 2)`` will return a ``decimal128(38, 2)`` scalar.

Review Comment:
   Ahah, no, I misread 3 for 38. Sorry.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2917803097

   No problem! Thanks for the reviews!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "zanmato1984 (via GitHub)" <gi...@apache.org>.
zanmato1984 commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2957456844

   Oh @khwilson one more thing, please update the PR description based on the latest code changes. Thanks.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2134824192


##########
cpp/src/arrow/compute/kernels/codegen_internal.cc:
##########
@@ -528,6 +533,28 @@ Status CastDecimalArgs(TypeHolder* begin, size_t count) {
   return Status::OK();
 }
 
+Result<std::shared_ptr<DataType>> WidenDecimalToMaxPrecision(

Review Comment:
   Done.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2138007218


##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   Hobgoblins etc. ☺️ happy to remove these double checks as long as it’s the correct style moving forward! 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #44184:
URL: https://github.com/apache/arrow/pull/44184#discussion_r2137973459


##########
python/pyarrow/tests/test_compute.py:
##########
@@ -355,10 +355,55 @@ def test_sum_array(arrow_type):
 
     arr = pa.array([], type=arrow_type)
     assert arr.sum().as_py() is None  # noqa: E711
+    assert pc.sum(arr).as_py() is None  # noqa: E711

Review Comment:
   Well, I don't think it makes sense to add more spurious tests in the name of consistency, in any case :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in sum aggregates [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2914842858

   @zanmato1984 there appear to be a few linting issues I need to fix, but before tackling those you mentioned you had "more [suggestions] coming." Just a small poke!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2845380973

   Cool. I’m happy to do some more clean up or do a little work on creating the checked versions of the aggregates to push this along


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-35166: [C++] Increase precision of decimals in aggregate functions [arrow]

Posted by "khwilson (via GitHub)" <gi...@apache.org>.
khwilson commented on PR #44184:
URL: https://github.com/apache/arrow/pull/44184#issuecomment-2829020999

   Hey @pitrou i was thinking about this PR again recently. Would you still be interested in merging if I cleaned it up a bit?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org