You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "zeroshade (via GitHub)" <gi...@apache.org> on 2024/09/04 19:40:20 UTC
[PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
zeroshade opened a new pull request, #43957:
URL: https://github.com/apache/arrow/pull/43957
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
Widening the Decimal128/256 type to allow for bitwidths of 32 and 64 allows for more interoperability with other libraries and utilities which already support these types. This provides even more opportunities for zero-copy interactions between things such as libcudf and various databases.
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
This PR contains the basic C++ implementations for Decimal32/Decimal64 types, arrays, builders and scalars. It also includes the minimum necessary to get everything compiling and tests passing without also extending the acero kernels and parquet handling (both of which will be handled in follow-up PRs).
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
Yes, tests were extended where applicable to add decimal32/decimal64 cases.
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
Currently if a user is using `decimal(precision, scale)` rather than `decimal128(precision, scale)` they will get a `Decimal128Type` if the precision is <= 38 (max precision for Decimal128) and `Decimal256Type` if the precision is higher. Following the same pattern, this change means that using `decimal(precision, scale)` instead of the specific `decimal32`/`decimal64`/`decimal128`/`decimal256` functions results in the following functionality:
- for precisions [1 : 9] => `Decimal32Type`
- for precisions [10 : 18] => `Decimal64Type`
- for precisions [19 : 38] => `Decimal128Type`
- for precisions [39 : 76] => `Decimal256Type`
While many of our tests currently make the assumption that `decimal` with a low precision would be `Decimal128` and had to be updated, this may cause an initial surprise if users are making the same assumptions.
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763476858
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
I've simplified this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761442449
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
these can't be defined in the base generic class because they return themselves by reference (you can't initialize a reference of the derived type from the base type)
I'll try just defining them inline in the header
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761376230
##########
docs/source/status.rst:
##########
@@ -44,6 +44,10 @@ Data Types
+-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
| Float32/64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
+| Decimal32 | ✓ | | ✓ | | | | | | |
++-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
+| Decimal64 | ✓ | | ✓ | | | | | | |
++-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
Review Comment:
the Go impl was already merged in the arrow-go repo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763459645
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
Review Comment:
wouldn't that get lost by the `memcpy` right below it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2359187901
@pitrou can you take another pass here? I believe I've addressed all of your comments. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761414180
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
I agree with @pitrou that "smallest" is easier to remember than "least" but I'm fine with renaming it to "least" if that is preferred.
@felipecrv, @bkietz, @lidavidm thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767138767
##########
cpp/src/arrow/util/decimal_internal.h:
##########
@@ -442,8 +516,9 @@ struct RealTraits<float> {
static constexpr float two_to_192(float x) { return x == 0 ? 0 : kFloatInf; }
static constexpr int kMantissaBits = 24;
- // ceil(log10(2 ^ kMantissaBits))
- static constexpr int kMantissaDigits = 8;
+ // log10(2 ^ kMantissaBits) ~= 7.2, let's be conservative to ensure more accuracy
+ // with our conversions for Decimal values
Review Comment:
I reran the tests to confirm, and yea it's only the Decimal32 case which runs into this problem (and only for Float since the double case falls back to the approx algorithm).
In the float case, Decimal32 won't hit the condition to fall back on the approx algorithm as MaxPrecision == 9 and MantissaDigits is 8 (or 7 if I keep this change). In either case, using 7 for the mantissa digits is likely more accurate given that log10(2 ^ 24) is ~7.2 as opposed to doubles which uses 16 because log(2 ^ 53) ~= 15.9 so it makes sense to ceiling that to 16.
The only alternative would be to just have Decimal32 *always* go to the approx algorithm even for the float case despite maxprecision being larger than the mantissa digits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763398047
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,243 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
Review Comment:
It would be nice to add tests for the non-trivial methods: `ToString`, `FromString`, `ToReal`, `FromReal`, `FromBigEndian`.
(same for `Decimal64`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766971607
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
That said, if you don't want to debug these now, I would favor removing the hacks, adding temporary workarounds in the tests, and opening an issue together with value(s) reproducing the problem. Then this can be investigated (by you, me or @bkietz for example :-)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752259319
##########
cpp/src/arrow/type.cc:
##########
@@ -1459,6 +1478,42 @@ int32_t DecimalType::DecimalSize(int32_t precision) {
return static_cast<int32_t>(std::ceil((precision / 8.0) * std::log2(10) + 1));
}
+// ----------------------------------------------------------------------
+// Decimal32 type
+
+Decimal32Type::Decimal32Type(int32_t precision, int32_t scale)
+ : DecimalType(type_id, 4, precision, scale) {
+ ARROW_CHECK_GE(precision, kMinPrecision);
+ ARROW_CHECK_LE(precision, kMaxPrecision);
Review Comment:
made these changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751047267
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal32Type;
Review Comment:
Makes sense, i'll fix that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2347130817
I'll give until tomorrow afternoon. If there's no further comments / objections, i'll merge this.
Thanks everyone!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766813001
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1230,65 +1275,72 @@ TYPED_TEST(TestDecimalToRealDouble, LargeValues) {
}
TYPED_TEST(TestDecimalToRealDouble, Precision) {
- // 2**63 + 2**11 (exactly representable in a double's 53 bits of precision)
- CheckDecimalToReal<TypeParam, double>("9223372036854777856", 0, 9.223372036854778e+18);
- CheckDecimalToReal<TypeParam, double>("-9223372036854777856", 0,
- -9.223372036854778e+18);
- // 2**64 - 2**11 (exactly representable in a double)
- CheckDecimalToReal<TypeParam, double>("18446744073709549568", 0, 1.844674407370955e+19);
- CheckDecimalToReal<TypeParam, double>("-18446744073709549568", 0,
- -1.844674407370955e+19);
- // 2**64 + 2**11 (exactly representable in a double)
- CheckDecimalToReal<TypeParam, double>("18446744073709555712", 0,
- 1.8446744073709556e+19);
- CheckDecimalToReal<TypeParam, double>("-18446744073709555712", 0,
- -1.8446744073709556e+19);
- // Almost 10**38 (minus 2**73)
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 0,
- 9.999999999999998e+37);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 0,
- -9.999999999999998e+37);
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 10,
- 9.999999999999998e+27);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 10,
- -9.999999999999998e+27);
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", -10,
- 9.999999999999998e+47);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", -10,
- -9.999999999999998e+47);
+ if constexpr (TypeParam::kMaxPrecision >= 19) {
+ // 2**63 + 2**11 (exactly representable in a double's 53 bits of precision)
+ CheckDecimalToReal<TypeParam, double>("9223372036854777856", 0,
+ 9.223372036854778e+18);
+ CheckDecimalToReal<TypeParam, double>("-9223372036854777856", 0,
+ -9.223372036854778e+18);
+ // 2**64 - 2**11 (exactly representable in a double)
+ CheckDecimalToReal<TypeParam, double>("18446744073709549568", 0,
+ 1.844674407370955e+19);
+ CheckDecimalToReal<TypeParam, double>("-18446744073709549568", 0,
+ -1.844674407370955e+19);
+ // 2**64 + 2**11 (exactly representable in a double)
+ CheckDecimalToReal<TypeParam, double>("18446744073709555712", 0,
+ 1.8446744073709556e+19);
+ CheckDecimalToReal<TypeParam, double>("-18446744073709555712", 0,
+ -1.8446744073709556e+19);
+
+ // Almost 10**38 (minus 2**73)
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 0,
+ 9.999999999999998e+37);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 0,
+ -9.999999999999998e+37);
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 10,
+ 9.999999999999998e+27);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 10,
+ -9.999999999999998e+27);
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", -10,
+ 9.999999999999998e+47);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", -10,
+ -9.999999999999998e+47);
+ }
// Integers are always exact
auto scale = TypeParam::kMaxScale - 1;
std::string seven = "7.";
seven.append(scale, '0');
CheckDecimalToReal<TypeParam, double>(seven, scale, 7.0);
CheckDecimalToReal<TypeParam, double>("-" + seven, scale, -7.0);
-
- CheckDecimalToReal<TypeParam, double>("99999999999999999999.0000000000000000", 16,
- 99999999999999999999.0);
- CheckDecimalToReal<TypeParam, double>("-99999999999999999999.0000000000000000", 16,
- -99999999999999999999.0);
+ if constexpr (TypeParam::kMaxPrecision >= 20) {
+ CheckDecimalToReal<TypeParam, double>("99999999999999999999.0000000000000000", 16,
+ 99999999999999999999.0);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999999999.0000000000000000", 16,
+ -99999999999999999999.0);
+ }
// Small fractions are within one ULP
CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.9", 1, 9999999.9);
CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.9", 1, -9999999.9);
- CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.999999999999999", 15,
- 9999999.999999999999999);
- CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.999999999999999", 15,
- -9999999.999999999999999);
-
- // Large fractions are within 2^-52
- constexpr double epsilon = 2.220446049250313080847263336181640625e-16; // 2^-52
- CheckDecimalToRealWithinEpsilon<TypeParam, double>(
- "112334829348925.99070703983306884765625", 23, epsilon,
- 112334829348925.99070703983306884765625);
- CheckDecimalToRealWithinEpsilon<TypeParam, double>(
- "1.987748987892758765582589910934859345", 36, epsilon,
- 1.987748987892758765582589910934859345);
+ if constexpr (TypeParam::kMaxPrecision >= 23) {
+ CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.999999999999999", 15,
+ 9999999.999999999999999);
+ CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.999999999999999", 15,
+ -9999999.999999999999999);
+ // Large fractions are within 2^-52
+ constexpr double epsilon = 2.220446049250313080847263336181640625e-16; // 2^-52
+ CheckDecimalToRealWithinEpsilon<TypeParam, double>(
+ "112334829348925.99070703983306884765625", 23, epsilon,
+ 112334829348925.99070703983306884765625);
+ CheckDecimalToRealWithinEpsilon<TypeParam, double>(
+ "1.987748987892758765582589910934859345", 36, epsilon,
+ 1.987748987892758765582589910934859345);
+ }
}
#endif // __MINGW32__
-TEST(Decimal128Test, TestFromBigEndian) {
+TEST(Decimal32Test, TestFromBigEndian) {
Review Comment:
It seems a number of tests are mostly or exactly identical between the Decimal tests, perhaps you can write a generic version and call it for each concrete test type?
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -125,7 +137,7 @@ class DecimalFromStringTest : public ::testing::Test {
"00a", "1e1a", "0.00123D/3", "1.23eA8", "1.23E+3A", "-1.23E--5",
"1.2345E+++07"}) {
ARROW_SCOPED_TRACE("invalid_value = '", invalid_value, "'");
- ASSERT_RAISES(Invalid, Decimal128::FromString(invalid_value));
+ ASSERT_RAISES(Invalid, DecimalType::FromString(invalid_value));
Review Comment:
:+1:
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1389,6 +1435,838 @@ std::vector<CType> GetRandomNumbers(int32_t size) {
return ret;
}
+TEST(Decimal32Test, Multiply) {
+ ASSERT_EQ(Decimal32(60501), Decimal32(301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(-301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(301) * Decimal32(-201));
+
+ ASSERT_EQ(Decimal32(60501), Decimal32(-301) * Decimal32(-201));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ Decimal32 result = Decimal32(x) * Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) * y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Divide) {
+ ASSERT_EQ(Decimal32(66), Decimal32(20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(-20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(20100) / Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(66), Decimal32(-20100) / Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) / Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) / y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Rescale) {
+ ASSERT_OK_AND_EQ(Decimal32(11100), Decimal32(111).Rescale(0, 2));
+ ASSERT_OK_AND_EQ(Decimal32(111), Decimal32(11100).Rescale(2, 0));
+ ASSERT_OK_AND_EQ(Decimal32(5), Decimal32(500000).Rescale(6, 1));
+ ASSERT_OK_AND_EQ(Decimal32(500000), Decimal32(5).Rescale(1, 6));
+ ASSERT_RAISES(Invalid, Decimal32(555555).Rescale(6, 1));
+
+ // Test some random numbers.
+ for (auto original_scale : GetRandomNumbers<Int8Type>(16)) {
+ for (auto value : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 unscaled_value = Decimal32(value);
+ Decimal32 scaled_value = unscaled_value;
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 4;
+ new_scale++, scaled_value *= Decimal32(10)) {
+ ASSERT_OK_AND_EQ(scaled_value, unscaled_value.Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(unscaled_value, scaled_value.Rescale(new_scale, original_scale));
+ }
+ }
+ }
+
+ for (auto original_scale : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 value(1);
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 8;
+ new_scale++, value *= Decimal32(10)) {
+ Decimal32 negative_value = value * -1;
+ ASSERT_OK_AND_EQ(value, Decimal32(1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(negative_value, Decimal32(-1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(Decimal32(1), value.Rescale(new_scale, original_scale));
+ ASSERT_OK_AND_EQ(Decimal32(-1), negative_value.Rescale(new_scale, original_scale));
+ }
+ }
+}
+
+TEST(Decimal32Test, Mod) {
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) % Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) % y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Sign) {
+ ASSERT_EQ(1, Decimal32(999999).Sign());
+ ASSERT_EQ(-1, Decimal32(-999999).Sign());
+ ASSERT_EQ(1, Decimal32(0).Sign());
+}
+
+TEST(Decimal32Test, GetWholeAndFraction) {
+ Decimal32 value("123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
+
+ value.GetWholeAndFraction(1, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(12345, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(6, out);
+
+ value.GetWholeAndFraction(5, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(1, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(23456, out);
+
+ value.GetWholeAndFraction(7, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(0, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+}
+
+TEST(Decimal32Test, GetWholeAndFractionNegative) {
+ Decimal32 value("-123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
+
+ value.GetWholeAndFraction(1, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-12345, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-6, out);
+
+ value.GetWholeAndFraction(5, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-1, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-23456, out);
+
+ value.GetWholeAndFraction(7, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(0, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-123456, out);
+}
+
+TEST(Decimal32Test, IncreaseScale) {
+ Decimal32 result;
+ int32_t out;
+
+ result = Decimal32("1234").IncreaseScaleBy(0);
Review Comment:
Same here and below: create helper functions to make these tests more concise and readable.
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +110,11 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (std::is_base_of_v<BasicDecimal32, DecimalType> &&
+ std::is_same_v<Real, double>) {
+ return DecimalType::FromReal(static_cast<float>(real), precision, scale);
Review Comment:
I think what happens is that the algorithm below assumes the destination Decimal has more mantissa bits than the source Real. This is false for the double->Decimal64 conversion. I think what we should do in that case is simply fall back to `FromPositiveRealApprox`.
Something like (untested :-)):
```c++
if constexpr (kMaxPrecision <= kMantissaDigits) {
return Derived::FromPositiveRealApprox(real, precision, scale);
}
```
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1170,40 +1206,49 @@ TYPED_TEST(TestDecimalToRealFloat, LargeValues) {
}
TYPED_TEST(TestDecimalToRealFloat, Precision) {
Review Comment:
Should also expand the parametrization of `TestDecimalToReal` above?
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +110,11 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (std::is_base_of_v<BasicDecimal32, DecimalType> &&
+ std::is_same_v<Real, double>) {
+ return DecimalType::FromReal(static_cast<float>(real), precision, scale);
Review Comment:
This is lowering the mantissa's precision to 24 bits even though Decimal32 has 32 bits of precision, is it right?
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -896,22 +917,30 @@ TYPED_TEST_SUITE(TestDecimalFromRealFloat, DecimalTypes);
TYPED_TEST(TestDecimalFromRealFloat, SuccessConversion) {
Review Comment:
I think we'd like to expand the parametrization for `TestDecimalFromReal` above as well.
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
Hmm, why?
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1389,6 +1435,838 @@ std::vector<CType> GetRandomNumbers(int32_t size) {
return ret;
}
+TEST(Decimal32Test, Multiply) {
+ ASSERT_EQ(Decimal32(60501), Decimal32(301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(-301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(301) * Decimal32(-201));
+
+ ASSERT_EQ(Decimal32(60501), Decimal32(-301) * Decimal32(-201));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ Decimal32 result = Decimal32(x) * Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) * y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Divide) {
+ ASSERT_EQ(Decimal32(66), Decimal32(20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(-20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(20100) / Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(66), Decimal32(-20100) / Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) / Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) / y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Rescale) {
+ ASSERT_OK_AND_EQ(Decimal32(11100), Decimal32(111).Rescale(0, 2));
+ ASSERT_OK_AND_EQ(Decimal32(111), Decimal32(11100).Rescale(2, 0));
+ ASSERT_OK_AND_EQ(Decimal32(5), Decimal32(500000).Rescale(6, 1));
+ ASSERT_OK_AND_EQ(Decimal32(500000), Decimal32(5).Rescale(1, 6));
+ ASSERT_RAISES(Invalid, Decimal32(555555).Rescale(6, 1));
+
+ // Test some random numbers.
+ for (auto original_scale : GetRandomNumbers<Int8Type>(16)) {
+ for (auto value : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 unscaled_value = Decimal32(value);
+ Decimal32 scaled_value = unscaled_value;
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 4;
+ new_scale++, scaled_value *= Decimal32(10)) {
+ ASSERT_OK_AND_EQ(scaled_value, unscaled_value.Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(unscaled_value, scaled_value.Rescale(new_scale, original_scale));
+ }
+ }
+ }
+
+ for (auto original_scale : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 value(1);
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 8;
+ new_scale++, value *= Decimal32(10)) {
+ Decimal32 negative_value = value * -1;
+ ASSERT_OK_AND_EQ(value, Decimal32(1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(negative_value, Decimal32(-1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(Decimal32(1), value.Rescale(new_scale, original_scale));
+ ASSERT_OK_AND_EQ(Decimal32(-1), negative_value.Rescale(new_scale, original_scale));
+ }
+ }
+}
+
+TEST(Decimal32Test, Mod) {
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) % Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) % y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Sign) {
+ ASSERT_EQ(1, Decimal32(999999).Sign());
+ ASSERT_EQ(-1, Decimal32(-999999).Sign());
+ ASSERT_EQ(1, Decimal32(0).Sign());
+}
+
+TEST(Decimal32Test, GetWholeAndFraction) {
+ Decimal32 value("123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
Review Comment:
Looks like we want a helper function to avoid repeating these snippets:
```c++
template <typename DecimalType>
void AssertGetWholeAndFraction(
DecimalType value, int32_t scale, std::pair<int32_t> expected_whole_and_fraction);
AssertGetWholeAndFraction(value, 0, {123456, 0});
AssertGetWholeAndFraction(value, 1, {12345, 6});
...
```
##########
cpp/src/arrow/util/decimal_internal.h:
##########
@@ -442,8 +516,9 @@ struct RealTraits<float> {
static constexpr float two_to_192(float x) { return x == 0 ? 0 : kFloatInf; }
static constexpr int kMantissaBits = 24;
- // ceil(log10(2 ^ kMantissaBits))
- static constexpr int kMantissaDigits = 8;
+ // log10(2 ^ kMantissaBits) ~= 7.2, let's be conservative to ensure more accuracy
+ // with our conversions for Decimal values
Review Comment:
Did it come up in some of the tests otherwise?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761350590
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
Review Comment:
you're right, that comment is for the decimal64/128/256 not for 32, i'll remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761378106
##########
cpp/src/arrow/testing/gtest_util.h:
##########
@@ -171,7 +171,8 @@ using PrimitiveArrowTypes =
using TemporalArrowTypes =
::testing::Types<Date32Type, Date64Type, TimestampType, Time32Type, Time64Type>;
-using DecimalArrowTypes = ::testing::Types<Decimal128Type, Decimal256Type>;
+using DecimalArrowTypes =
+ ::testing::Types</*Decimal32Type, Decimal64Type,*/ Decimal128Type, Decimal256Type>;
Review Comment:
yes, i didn't want to make this already large PR even larger. I'll implement the cast kernels and so on as a follow-up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761552917
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
It doesn't make sense, especially if the decimal precision remains the same (which it probably does?).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761447575
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
You could use [CRTP](https://en.wikipedia.org/wiki/Curiously_recurring_template_pattern) :-) But, yes, defining them inline is fine as well...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763412533
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,72 @@ struct DecimalGenerator {
}
};
+template <typename DecimalType>
+struct SmallDecimalGenerator {
+ using DecimalBuilderType = typename TypeTraits<DecimalType>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+ using IntegerType = typename DecimalValue::ValueType;
+ using IntArrowType = typename CTypeTraits<IntegerType>::ArrowType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static IntegerType MaxDecimalInteger(int32_t digits) {
+ return static_cast<IntegerType>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger =
+ std::is_same_v<DecimalType, Decimal32Type> ? 9 : 18;
+ static_assert(
+ kMaxDigitsInInteger >= DecimalType::kByteWidth,
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto digits_to_generate = decimal_type.precision();
+ auto values = checked_pointer_cast<typename TypeTraits<IntArrowType>::ArrayType>(
+ rng_->Numeric<IntArrowType>(size, -1 * MaxDecimalInteger(digits_to_generate),
+ MaxDecimalInteger(digits_to_generate),
+ null_probability, alignment, memory_pool));
+
Review Comment:
good point. updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2331116181
> Following the same pattern, this change means that using `decimal(precision, scale)` instead of the specific `decimal32`/`decimal64`/`decimal128`/`decimal256` functions results in the following functionality
I'm afraid this may massively break user code. I would suggest another approach:
* deprecate the `decimal()` factory while keeping its current behavior of always returning at least `decimal128`
* introduce a new `smallest_decimal()` factory that is documented to return the smallest possible type, and explicitly makes no guarantees about the stability of the return type
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1754715223
##########
cpp/src/arrow/engine/substrait/expression_internal.cc:
##########
@@ -951,25 +951,28 @@ struct ScalarToProtoImpl {
Status Visit(const MonthIntervalScalar& s) { return NotImplemented(s); }
Status Visit(const DayTimeIntervalScalar& s) { return NotImplemented(s); }
- Status Visit(const Decimal128Scalar& s) {
+ template <typename T,
+ typename = internal::EnableIfIsOneOf<T, Decimal32Scalar, Decimal64Scalar,
+ Decimal128Scalar, Decimal256Scalar>>
+ Status Visit(const T& s) {
+ using TypeClass = typename T::TypeClass;
Review Comment:
```suggestion
template <typename T, typename TypeClass = typename T::TypeClass>
enable_if_decimal<TypeClass, Status> Visit(const T& s) {
```
##########
cpp/src/arrow/engine/substrait/expression_internal.cc:
##########
@@ -951,25 +951,28 @@ struct ScalarToProtoImpl {
Status Visit(const MonthIntervalScalar& s) { return NotImplemented(s); }
Status Visit(const DayTimeIntervalScalar& s) { return NotImplemented(s); }
- Status Visit(const Decimal128Scalar& s) {
+ template <typename T,
+ typename = internal::EnableIfIsOneOf<T, Decimal32Scalar, Decimal64Scalar,
+ Decimal128Scalar, Decimal256Scalar>>
+ Status Visit(const T& s) {
+ using TypeClass = typename T::TypeClass;
+ using ValueType = typename T::ValueType;
+
auto decimal = std::make_unique<Lit::Decimal>();
- auto decimal_type = checked_cast<const Decimal128Type*>(s.type.get());
+ auto decimal_type = checked_cast<const TypeClass*>(s.type.get());
decimal->set_precision(decimal_type->precision());
decimal->set_scale(decimal_type->scale());
decimal->set_value(reinterpret_cast<const char*>(s.value.native_endian_bytes()),
- sizeof(Decimal128));
+ sizeof(ValueType));
#if !ARROW_LITTLE_ENDIAN
std::reverse(decimal->mutable_value()->begin(), decimal->mutable_value()->end());
#endif
lit_->set_allocated_decimal(decimal.release());
return Status::OK();
}
- // Need support for parameterized UDTs
- Status Visit(const Decimal256Scalar& s) { return NotImplemented(s); }
Review Comment:
:+1:
##########
cpp/src/arrow/compare.cc:
##########
@@ -750,6 +750,18 @@ class TypeEqualsVisitor {
return Status::OK();
}
+ Status Visit(const Decimal32Type& left) {
+ const auto& right = checked_cast<const Decimal32Type&>(right_);
+ result_ = left.precision() == right.precision() && left.scale() == right.scale();
+ return Status::OK();
+ }
Review Comment:
```suggestion
Status Visit(const DecimalType& left) {
const auto& right = checked_cast<const DecimalType&>(right_);
result_ = left.byte_width() == right.byte_width() && left.precision() == right.precision() && left.scale() == right.scale();
return Status::OK();
}
```
##########
cpp/src/arrow/array/diff.cc:
##########
@@ -707,7 +707,11 @@ class MakeFormatterImpl {
template <typename T>
enable_if_decimal<T, Status> Visit(const T&) {
impl_ = [](const Array& array, int64_t index, std::ostream* os) {
- if constexpr (T::type_id == Type::DECIMAL128) {
+ if constexpr (T::type_id == Type::DECIMAL32) {
Review Comment:
```suggestion
const auto& decimal_array =
checked_cast<const typename TypeTraits<T>::ArrayType&>(array);
*os << decimal_array.FormatValue(index);
```
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -314,6 +314,20 @@ class SchemaWriter {
writer_->Int(type.list_size());
}
+ void WriteTypeMetadata(const Decimal32Type& type) {
Review Comment:
I think we don't need to unpack to specific widths
```suggestion
void WriteTypeMetadata(const DecimalType& type) {
```
##########
cpp/src/arrow/compute/kernels/hash_aggregate.cc:
##########
@@ -892,8 +894,10 @@ struct GroupedVarStdImpl : public GroupedAggregator {
// float/double/int64/decimal: calculate `m2` (sum((X-mean)^2)) with
// `two pass algorithm` (see aggregate_var_std.cc)
template <typename T = Type>
- enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4), Status> ConsumeImpl(
- const ExecSpan& batch) {
+ enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4) ||
+ (!is_integer_type<T>::value && sizeof(CType) == 4),
Review Comment:
I'd prefer to see this last clause be more explicit, IIUC you could also write
```suggestion
std::is_same_v<CType, Decimal32>,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1754671465
##########
cpp/src/arrow/array/builder_dict.h:
##########
@@ -298,20 +298,11 @@ class DictionaryBuilderBase : public ArrayBuilder {
return Append(std::string_view(value, length));
}
- /// \brief Append a decimal (only for Decimal128Type)
- template <typename T1 = T>
- enable_if_decimal128<T1, Status> Append(const Decimal128& value) {
- uint8_t data[16];
- value.ToBytes(data);
- return Append(data, 16);
- }
-
- /// \brief Append a decimal (only for Decimal128Type)
- template <typename T1 = T>
- enable_if_decimal256<T1, Status> Append(const Decimal256& value) {
- uint8_t data[32];
- value.ToBytes(data);
- return Append(data, 32);
+ /// \brief Append a decimal (only for Decimal32/64/128/256 Type)
+ template <typename T1 = T, typename CType = typename TypeTraits<T1>::CType>
+ enable_if_decimal<T1, Status> Append(const CType& value) {
+ auto bytes = value.ToBytes();
+ return Append(bytes.data(), bytes.size());
Review Comment:
[argh](https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/50577346#L1534)
```suggestion
return Append(bytes.data(), static_cast<int64_t>(bytes.size()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752707143
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,263 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ static void SwapEndian(const uint32_t* old_data, uint32_t* const new_data,
+ const int64_t length);
Review Comment:
updated using the suggestion, let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766976289
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1170,40 +1206,49 @@ TYPED_TEST(TestDecimalToRealFloat, LargeValues) {
}
TYPED_TEST(TestDecimalToRealFloat, Precision) {
Review Comment:
ah, i missed that one!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767191246
##########
cpp/src/arrow/util/decimal_internal.h:
##########
@@ -442,8 +516,9 @@ struct RealTraits<float> {
static constexpr float two_to_192(float x) { return x == 0 ? 0 : kFloatInf; }
static constexpr int kMantissaBits = 24;
- // ceil(log10(2 ^ kMantissaBits))
- static constexpr int kMantissaDigits = 8;
+ // log10(2 ^ kMantissaBits) ~= 7.2, let's be conservative to ensure more accuracy
+ // with our conversions for Decimal values
Review Comment:
I've switched this back to 8 and just unconditionally send Decimal32 to the approx algorithm. Let me know if this is sufficient or if I should put it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2349114555
Sure, @pitrou I'll wait for your review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761375789
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,257 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal32 from one scale to another
+ Result<Decimal32> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal32 out;
+ auto dstatus = BasicDecimal32::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
+ return Status::Invalid("Invalid cast from Decimal32 value to ", sizeof(T),
+ "byte integer");
+ }
+ return static_cast<T>(value_);
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Status ToInteger(T* out) const {
+ return ToInteger<T>().Value(out);
+ }
+
+ /// \brief Convert to a floating-point number (scaled)
+ float ToFloat(int32_t scale) const;
+ /// \brief Convert to a floating-point number (scaled)
+ double ToDouble(int32_t scale) const;
+
+ /// \brief Convert to a floating-point number (scaled)
+ template <typename T, typename = std::enable_if_t<std::is_floating_point_v<T>>>
+ T ToReal(int32_t scale) const {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "Unexpected floating-point type");
+ if constexpr (std::is_same_v<T, float>) {
+ return ToFloat(scale);
+ } else {
+ return ToDouble(scale);
+ }
+ }
+
+ ARROW_FRIEND_EXPORT friend std::ostream& operator<<(std::ostream& os,
+ const Decimal32& decimal);
+
+ private:
+ /// Converts internal error code to Status
+ Status ToArrowStatus(DecimalStatus dstatus) const;
+};
+
+class ARROW_EXPORT Decimal64 : public BasicDecimal64 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal64::BasicDecimal64;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal64 from a BasicDecimal64
+ constexpr Decimal64(const BasicDecimal64& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal64(value) {}
+
+ explicit Decimal64(const BasicDecimal32& value) noexcept
+ : BasicDecimal64(static_cast<int64_t>(value.value())) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal64(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal64 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal64() noexcept : BasicDecimal64() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal64, Decimal64>> Divide(const Decimal64& divisor) const {
+ std::pair<Decimal64, Decimal64> result;
+ auto dstatus = BasicDecimal64::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal64 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal64> FromString(std::string_view s);
+ static Result<Decimal64> FromString(const std::string& s);
+ static Result<Decimal64> FromString(const char* s);
+
+ static Result<Decimal64> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal64> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal64> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal64 from one scale to another
+ Result<Decimal64> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal64 out;
+ auto dstatus = BasicDecimal64::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
Review Comment:
you're right, these checks aren't needed for decimal64 or decimal32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761361053
##########
cpp/src/arrow/type.cc:
##########
@@ -480,7 +494,7 @@ Result<std::shared_ptr<DataType>> MaybeMergeNumericTypes(
ARROW_ASSIGN_OR_RAISE(const int32_t precision,
MaxDecimalDigitsForInteger(other_type->id()));
ARROW_ASSIGN_OR_RAISE(const auto promoted_decimal,
- DecimalType::Make(promoted_type->id(), precision, 0));
+ DecimalType::Make(promoted_type->id(), precision - 1, 0));
Review Comment:
The values we use for `MaxDecimalDigitsForInteger` is actually 1 + numeric_limits<T>::digits10 (i.e. for int32/uint32 we return 10 instead of 9, for int64 we return 19 instead of 18). As such we were passing a precision that was higher than the maximum precision allowed for Decimal32 and Decimal64 here (9 and 18 respectively).
it felt safer to me to put the -1 here than to change the value returned in `MaxDecimalDigitsForInteger` (which is used elsewhere and caused other complications in tests)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763451885
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
you're right, this should be 4 not 8 so that we avoid an undefined shift by 32 below this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763368294
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,72 @@ struct DecimalGenerator {
}
};
+template <typename DecimalType>
+struct SmallDecimalGenerator {
+ using DecimalBuilderType = typename TypeTraits<DecimalType>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+ using IntegerType = typename DecimalValue::ValueType;
+ using IntArrowType = typename CTypeTraits<IntegerType>::ArrowType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static IntegerType MaxDecimalInteger(int32_t digits) {
+ return static_cast<IntegerType>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger =
+ std::is_same_v<DecimalType, Decimal32Type> ? 9 : 18;
+ static_assert(
+ kMaxDigitsInInteger >= DecimalType::kByteWidth,
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto digits_to_generate = decimal_type.precision();
+ auto values = checked_pointer_cast<typename TypeTraits<IntArrowType>::ArrayType>(
+ rng_->Numeric<IntArrowType>(size, -1 * MaxDecimalInteger(digits_to_generate),
+ MaxDecimalInteger(digits_to_generate),
+ null_probability, alignment, memory_pool));
+
Review Comment:
Note that we could just reuse the buffers, or perhaps even return `values.View(decimal_type)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763377104
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -521,6 +726,23 @@ std::string Decimal128::ToString(int32_t scale) const {
return str;
}
+static inline void ShiftAndAdd(std::string_view input, uint32_t* out) {
Review Comment:
This looks like a copy-paste job but I don't understand why it's useful. Converting a string to decimal32 or decimal64 shouldn't require multiple passes, so this should definitely be simplified.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761363940
##########
cpp/src/arrow/type_test.cc:
##########
@@ -1241,15 +1245,15 @@ TEST_F(TestUnifySchemas, Decimal) {
CheckPromoteTo(decimal256(3, -2), decimal256(5, -2), decimal256(5, -2), options);
// int32() is essentially decimal128(10, 0)
- CheckPromoteTo(int32(), decimal128(3, 2), decimal128(12, 2), options);
- CheckPromoteTo(int32(), decimal128(3, -2), decimal128(10, 0), options);
- CheckPromoteTo(int64(), decimal128(38, 37), decimal256(56, 37), options);
+ CheckPromoteTo(int32(), decimal128(3, 2), decimal128(11, 2), options);
+ CheckPromoteTo(int32(), decimal128(3, -2), decimal128(9, 0), options);
+ CheckPromoteTo(int64(), decimal128(38, 37), decimal256(55, 37), options);
CheckUnifyFailsTypeError(decimal256(1, 0), decimal128(1, 0), options);
options.promote_numeric_width = true;
CheckPromoteTo(decimal128(3, 2), decimal256(5, 2), decimal256(5, 2), options);
- CheckPromoteTo(int32(), decimal128(38, 37), decimal256(47, 37), options);
+ CheckPromoteTo(int32(), decimal128(38, 37), decimal256(46, 37), options);
CheckUnifyFailsInvalid(decimal128(38, 10), decimal256(76, 5), options);
Review Comment:
it gets tested in `array_test.cc` (though indirectly). i'll add some explicit tests for it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1765545771
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -342,6 +477,56 @@ struct Decimal128RealConversion
} // namespace
+Decimal32::Decimal32(const std::string& str) : Decimal32() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal32> Decimal32::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal32> Decimal32::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal32::ToFloat(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal32::ToDouble(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal32::ToIntegerString() const { return std::to_string(value_); }
+
+Decimal32::operator int64_t() const { return static_cast<int64_t>(value_); }
+
+Decimal32::operator Decimal64() const { return Decimal64(static_cast<int64_t>(value_)); }
+
+Decimal64::Decimal64(const std::string& str) : Decimal64() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal64> Decimal64::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal64RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal64> Decimal64::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal64RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal64::ToFloat(int32_t scale) const {
+ return Decimal64RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal64::ToDouble(int32_t scale) const {
+ return Decimal64RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal64::ToIntegerString() const { return std::to_string(value_); }
Review Comment:
updated to use `internal::StringFormatter` like the other decimal types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763394823
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
Review Comment:
How about something like:
```suggestion
int32_t result = is_negative ? 0xffffffff : 0;
```
which should automically get you the required sign extension?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1773722574
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -697,8 +943,150 @@ Status DecimalFromString(const char* type_name, std::string_view s, Decimal* out
return Status::OK();
}
+template <typename DecimalClass>
+Status SimpleDecimalFromString(const char* type_name, std::string_view s,
+ DecimalClass* out, int32_t* precision, int32_t* scale) {
+ if (s.empty()) {
+ return Status::Invalid("Empty string cannot be converted to ", type_name);
+ }
+
+ DecimalComponents dec;
+ if (!ParseDecimalComponents(s.data(), s.size(), &dec)) {
+ return Status::Invalid("The string '", s, "' is not a valid ", type_name, " number");
+ }
+
+ // count number of significant digits (without leading zeros)
+ size_t first_non_zero = dec.whole_digits.find_first_not_of('0');
+ size_t significant_digits = dec.fractional_digits.size();
+ if (first_non_zero != std::string::npos) {
+ significant_digits += dec.whole_digits.size() - first_non_zero;
+ }
+ int32_t parsed_precision = static_cast<int32_t>(significant_digits);
+
+ int32_t parsed_scale = 0;
+ if (dec.has_exponent) {
+ auto adjusted_exponent = dec.exponent;
+ parsed_scale =
+ -adjusted_exponent + static_cast<int32_t>(dec.fractional_digits.size());
+ } else {
+ parsed_scale = static_cast<int32_t>(dec.fractional_digits.size());
+ }
+
+ if (out != nullptr) {
+ if constexpr (std::is_same_v<Decimal32, DecimalClass>) {
+ uint32_t value{0};
+ ShiftAndAdd(dec.whole_digits, &value);
+ ShiftAndAdd(dec.fractional_digits, &value);
+ if (value > static_cast<uint32_t>(
+ std::numeric_limits<typename DecimalClass::ValueType>::max())) {
+ return Status::Invalid("The string '", s, "' cannot be represented as ",
+ type_name);
+ }
+
+ *out = DecimalClass(value);
+ if (dec.sign == '-') {
+ out->Negate();
+ }
+ } else {
Review Comment:
following the suggestion from https://github.com/apache/arrow/pull/43957#discussion_r1771711880, i was able to eliminate the branch there entirely. Now the `Decimal32` case just uses the `uint64_t` variant anyways, performs the overflow check and then constructs the decimal value from the `uint64_t` by downcasting.
So the codepaths have been unified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1773729468
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -92,6 +93,12 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
constexpr int kMantissaBits = RealTraits<Real>::kMantissaBits;
constexpr int kMantissaDigits = RealTraits<Real>::kMantissaDigits;
+ // to avoid precision and rounding issues, we'll unconditionally
+ // throw Decimal32 to the approx algorithm instead.
Review Comment:
Issue opened and referenced here (#44216)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1745945528
##########
cpp/src/arrow/testing/gtest_util.h:
##########
@@ -171,7 +171,8 @@ using PrimitiveArrowTypes =
using TemporalArrowTypes =
::testing::Types<Date32Type, Date64Type, TimestampType, Time32Type, Time64Type>;
-using DecimalArrowTypes = ::testing::Types<Decimal128Type, Decimal256Type>;
+using DecimalArrowTypes =
+ ::testing::Types</*Decimal32Type, Decimal64Type,*/ Decimal128Type, Decimal256Type>;
Review Comment:
These are commented out because we didn't implement casting for the new decimal types. This is mentioned in the issue as check boxes to do rather than as an entirely separate issue currently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2340101293
Can we expedite the format change in a separate PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2341014419
@pitrou I did that here: https://github.com/apache/arrow/pull/43976#pullrequestreview-2284369034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1754704921
##########
cpp/src/parquet/arrow/test_util.h:
##########
@@ -142,7 +142,11 @@ template <int32_t byte_width>
static void random_decimals(int64_t n, uint32_t seed, int32_t precision, uint8_t* out) {
auto gen = ::arrow::random::RandomArrayGenerator(seed);
std::shared_ptr<Array> decimals;
- if constexpr (byte_width == 16) {
+ if constexpr (byte_width == 4) {
+ decimals = gen.Decimal32(::arrow::decimal32(precision, 0), n);
+ } else if constexpr (byte_width == 8) {
+ decimals = gen.Decimal64(::arrow::decimal32(precision, 0), n);
Review Comment:
nice catch, I must have missed that / been a bit over-eager with my updates.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1765548463
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -521,6 +726,23 @@ std::string Decimal128::ToString(int32_t scale) const {
return str;
}
+static inline void ShiftAndAdd(std::string_view input, uint32_t* out) {
Review Comment:
simplified this for decimal32 (decimal64 is able to just use the existing implementation used for the other types by just using a size of 1)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2347133590
I plan to review this, could you wait for it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761527214
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto remaining_digits = decimal_type.precision();
+ auto values = checked_pointer_cast<Int32Array>(rng_->Int32(
+ size, -1 * MaxDecimalInteger(remaining_digits),
+ MaxDecimalInteger(remaining_digits), null_probability, alignment, memory_pool));
+
+ DecimalBuilderType builder(type_, memory_pool, alignment);
+ ABORT_NOT_OK(builder.Reserve(size));
+
+ for (int64_t i = 0; i < size; ++i) {
+ if (values->IsValid(i)) {
+ builder.UnsafeAppend(DecimalValue{values->Value(i)});
+ } else {
+ builder.UnsafeAppendNull();
+ }
+ }
+ std::shared_ptr<Array> array;
+ ABORT_NOT_OK(builder.Finish(&array));
+ return array;
+ }
+};
+
+template <>
+struct DecimalGenerator<Decimal64Type> {
Review Comment:
I was able to create a single generic `SmallDecimalGenerator<T>` so this worked out :)
##########
cpp/src/arrow/type_test.cc:
##########
@@ -1241,15 +1245,15 @@ TEST_F(TestUnifySchemas, Decimal) {
CheckPromoteTo(decimal256(3, -2), decimal256(5, -2), decimal256(5, -2), options);
// int32() is essentially decimal128(10, 0)
- CheckPromoteTo(int32(), decimal128(3, 2), decimal128(12, 2), options);
- CheckPromoteTo(int32(), decimal128(3, -2), decimal128(10, 0), options);
- CheckPromoteTo(int64(), decimal128(38, 37), decimal256(56, 37), options);
+ CheckPromoteTo(int32(), decimal128(3, 2), decimal128(11, 2), options);
+ CheckPromoteTo(int32(), decimal128(3, -2), decimal128(9, 0), options);
+ CheckPromoteTo(int64(), decimal128(38, 37), decimal256(55, 37), options);
CheckUnifyFailsTypeError(decimal256(1, 0), decimal128(1, 0), options);
options.promote_numeric_width = true;
CheckPromoteTo(decimal128(3, 2), decimal256(5, 2), decimal256(5, 2), options);
- CheckPromoteTo(int32(), decimal128(38, 37), decimal256(47, 37), options);
+ CheckPromoteTo(int32(), decimal128(38, 37), decimal256(46, 37), options);
CheckUnifyFailsInvalid(decimal128(38, 10), decimal256(76, 5), options);
Review Comment:
tests added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761537759
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
I tried using CRTP, still gives a compiler error that it can't initialize a reference to the derived class from the base class :smile:. probably because `*this` is still the type of the base class, not the derived
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766941722
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
With decimal32 and decimal64 I needed to reduce the `mul_step` by 1 in order to eliminate off-by-one rounding errors. I didn't want to unconditionally add extra operations to decimal128/256 by lowering their `mul_step` so I only did it for decimal32/64
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766945492
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
That's several things that need to be arbitrarily lowered, and sounds a bit unexpected.
Random hacks like this are a bad smell, especially if there's no detailed explanation other that "it works better".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763380995
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -697,8 +919,150 @@ Status DecimalFromString(const char* type_name, std::string_view s, Decimal* out
return Status::OK();
}
+template <typename DecimalClass>
+Status SimpleDecimalFromString(const char* type_name, std::string_view s,
+ DecimalClass* out, int32_t* precision, int32_t* scale) {
+ if (s.empty()) {
+ return Status::Invalid("Empty string cannot be converted to ", type_name);
+ }
+
+ DecimalComponents dec;
+ if (!ParseDecimalComponents(s.data(), s.size(), &dec)) {
+ return Status::Invalid("The string '", s, "' is not a valid ", type_name, " number");
+ }
+
+ // count number of significant digits (without leading zeros)
+ size_t first_non_zero = dec.whole_digits.find_first_not_of('0');
+ size_t significant_digits = dec.fractional_digits.size();
+ if (first_non_zero != std::string::npos) {
+ significant_digits += dec.whole_digits.size() - first_non_zero;
+ }
+ int32_t parsed_precision = static_cast<int32_t>(significant_digits);
+
+ int32_t parsed_scale = 0;
+ if (dec.has_exponent) {
+ auto adjusted_exponent = dec.exponent;
+ parsed_scale =
+ -adjusted_exponent + static_cast<int32_t>(dec.fractional_digits.size());
+ } else {
+ parsed_scale = static_cast<int32_t>(dec.fractional_digits.size());
+ }
+
+ if (out != nullptr) {
+ if constexpr (std::is_same_v<Decimal32, DecimalClass>) {
Review Comment:
Basically, we should just be computing `ParseValue<uint64_t>(dec.whole_digits) * (10^dec.exponent) + ParseValue<uint64_t>(dec.fractional_digits)`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763383059
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -342,6 +477,56 @@ struct Decimal128RealConversion
} // namespace
+Decimal32::Decimal32(const std::string& str) : Decimal32() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal32> Decimal32::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal32> Decimal32::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal32::ToFloat(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal32::ToDouble(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal32::ToIntegerString() const { return std::to_string(value_); }
Review Comment:
We should be using `FormatChars` to avoid locale-specific behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752151777
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
That was my thought too, we just need to make sure we call it out in the changelog to ensure anyone who reads the changelog that was relying on code like that would be aware.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752036013
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
By safe I mean: all the things someone assumed when they wrote an `enable_if_decimal` still hold. And you're saying you checked all of them with `GetSumType` being the only one where assumptions didn't hold well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1747615412
##########
cpp/src/arrow/type.cc:
##########
@@ -3301,8 +3356,18 @@ std::shared_ptr<Field> field(std::string name, std::shared_ptr<DataType> type,
}
std::shared_ptr<DataType> decimal(int32_t precision, int32_t scale) {
- return precision <= Decimal128Type::kMaxPrecision ? decimal128(precision, scale)
- : decimal256(precision, scale);
+ return precision <= Decimal32Type::kMaxPrecision ? decimal32(precision, scale)
Review Comment:
the updated version does exactly that, added a deprecation notice and created a new function
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761537759
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
I tried using CRTP, still gives a compiler error that it can't initialize a reference to the derived class from the base class :smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761527824
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
Review Comment:
renamed
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
Review Comment:
renamed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763478613
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
Review Comment:
Ah! it would only get lost if `kMaxDecimalBytes == length` in which case we were just copying all the bytes anyways and the sign extension would already be there. Good point!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1765544427
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
+ return Decimal32(value);
+ }
+
+ int32_t final_value = -1 * (is_negative && length < kMaxDecimalBytes);
+ final_value = SafeLeftShift(final_value, bits_offset * CHAR_BIT);
+ final_value |= value;
+ return Decimal32(final_value);
+}
+
+Status Decimal32::ToArrowStatus(DecimalStatus dstatus) const {
+ return arrow::ToArrowStatus(dstatus, 32);
+}
+
+std::ostream& operator<<(std::ostream& os, const Decimal32& decimal) {
+ os << decimal.ToIntegerString();
+ return os;
+}
+
+Result<Decimal64> Decimal64::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
Review Comment:
updated and fixed
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,243 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
Review Comment:
tests added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1765545979
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -342,6 +477,56 @@ struct Decimal128RealConversion
} // namespace
+Decimal32::Decimal32(const std::string& str) : Decimal32() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal32> Decimal32::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal32> Decimal32::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal32::ToFloat(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal32::ToDouble(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal32::ToIntegerString() const { return std::to_string(value_); }
Review Comment:
updated to use `internal::StringFormatter` like the other decimal types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761349571
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
Review Comment:
Isn't the purpose of this assert to validate that the precision *matches* `kByteWidth` as opposed to just verifying that the precision is equal to the max digits?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761340509
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -132,7 +142,13 @@ struct GetSumType<T, enable_if_integer<T>> {
};
template <typename T>
-struct GetSumType<T, enable_if_decimal<T>> {
+struct GetSumType<T, enable_if_decimal32<T>> {
+ using SumType = arrow::Decimal64;
Review Comment:
See discussion [here](https://github.com/apache/arrow/pull/43957#discussion_r1750962486) it's to avoid overflows, we do the same for the sumtype for other integer types, which all have a Sum type of 64-bit integers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763383435
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -342,6 +477,56 @@ struct Decimal128RealConversion
} // namespace
+Decimal32::Decimal32(const std::string& str) : Decimal32() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal32> Decimal32::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal32> Decimal32::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal32RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal32::ToFloat(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal32::ToDouble(int32_t scale) const {
+ return Decimal32RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal32::ToIntegerString() const { return std::to_string(value_); }
+
+Decimal32::operator int64_t() const { return static_cast<int64_t>(value_); }
+
+Decimal32::operator Decimal64() const { return Decimal64(static_cast<int64_t>(value_)); }
+
+Decimal64::Decimal64(const std::string& str) : Decimal64() {
+ *this = FromString(str).ValueOrDie();
+}
+
+Result<Decimal64> Decimal64::FromReal(float x, int32_t precision, int32_t scale) {
+ return Decimal64RealConversion::FromReal(x, precision, scale);
+}
+
+Result<Decimal64> Decimal64::FromReal(double x, int32_t precision, int32_t scale) {
+ return Decimal64RealConversion::FromReal(x, precision, scale);
+}
+
+float Decimal64::ToFloat(int32_t scale) const {
+ return Decimal64RealConversion::ToReal<float>(*this, scale);
+}
+
+double Decimal64::ToDouble(int32_t scale) const {
+ return Decimal64RealConversion::ToReal<double>(*this, scale);
+}
+
+std::string Decimal64::ToIntegerString() const { return std::to_string(value_); }
Review Comment:
Also `FormatChars` here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761925456
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
Review Comment:
fair enough, simplified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761336406
##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -336,8 +336,8 @@ struct ProductImpl : public ScalarAggregator {
internal::VisitArrayValuesInline<ArrowType>(
data,
[&](typename TypeTraits<ArrowType>::CType value) {
- this->product =
- MultiplyTraits<AccType>::Multiply(*out_type, this->product, value);
+ this->product = MultiplyTraits<AccType>::Multiply(
+ *out_type, this->product, static_cast<ProductType>(value));
Review Comment:
Because the constructors for Decimal32/Decimal64 are marked `explicit` I needed to explicitly add casts there to satisfy the templates for conversions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763395937
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
+ return Decimal32(value);
+ }
+
+ int32_t final_value = -1 * (is_negative && length < kMaxDecimalBytes);
+ final_value = SafeLeftShift(final_value, bits_offset * CHAR_BIT);
+ final_value |= value;
+ return Decimal32(final_value);
+}
+
+Status Decimal32::ToArrowStatus(DecimalStatus dstatus) const {
+ return arrow::ToArrowStatus(dstatus, 32);
+}
+
+std::ostream& operator<<(std::ostream& os, const Decimal32& decimal) {
+ os << decimal.ToIntegerString();
+ return os;
+}
+
+Result<Decimal64> Decimal64::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
Review Comment:
Same comments as in `Decimal32::FromBigEndian` obviously.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2332319891
@pitrou @bkietz @wgtmac I've updated this based on the suggestion, created a `smallest_decimal` function and added a deprecated message to the docstring for `decimal`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1779479216
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
I also think maybe int32 or `span<char, 4>` better here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1771691664
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1336,43 +1406,42 @@ TEST(Decimal128Test, TestFromBigEndian) {
// convert to big endian
std::reverse(native_endian.begin(), native_endian.end());
#endif
- ASSERT_OK_AND_EQ(complement, Decimal128::FromBigEndian(native_endian.data(), 16));
+ ASSERT_OK_AND_EQ(complement, TypeParam::FromBigEndian(native_endian.data(),
+ TypeParam::kByteWidth));
- value <<= 8;
- value += Decimal128(start);
+ value <<= 2;
+ value += TypeParam(start);
}
}
}
-TEST(Decimal128Test, TestFromBigEndianBadLength) {
- ASSERT_RAISES(Invalid, Decimal128::FromBigEndian(0, -1));
- ASSERT_RAISES(Invalid, Decimal128::FromBigEndian(0, 17));
+TYPED_TEST(TestBasicDecimalFunctionality, TestFromBigEndianBadLength) {
+ ASSERT_RAISES(Invalid, TypeParam::FromBigEndian(0, -1));
+ ASSERT_RAISES(Invalid, TypeParam::FromBigEndian(0, TypeParam::kByteWidth + 1));
}
-TEST(Decimal128Test, TestToInteger) {
- Decimal128 value1("1234");
- int32_t out1;
+TYPED_TEST(TestBasicDecimalFunctionality, TestToInteger) {
+ if constexpr (std::is_same_v<TypeParam, Decimal256>) {
Review Comment:
`Decimal256` isn't in `BasicFunctionalityDecimalTypes`, so is this condition useful? Or were you planning to add back `Decimal256` into the type list?
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -697,8 +943,150 @@ Status DecimalFromString(const char* type_name, std::string_view s, Decimal* out
return Status::OK();
}
+template <typename DecimalClass>
+Status SimpleDecimalFromString(const char* type_name, std::string_view s,
+ DecimalClass* out, int32_t* precision, int32_t* scale) {
+ if (s.empty()) {
+ return Status::Invalid("Empty string cannot be converted to ", type_name);
+ }
+
+ DecimalComponents dec;
+ if (!ParseDecimalComponents(s.data(), s.size(), &dec)) {
+ return Status::Invalid("The string '", s, "' is not a valid ", type_name, " number");
+ }
+
+ // count number of significant digits (without leading zeros)
+ size_t first_non_zero = dec.whole_digits.find_first_not_of('0');
+ size_t significant_digits = dec.fractional_digits.size();
+ if (first_non_zero != std::string::npos) {
+ significant_digits += dec.whole_digits.size() - first_non_zero;
+ }
+ int32_t parsed_precision = static_cast<int32_t>(significant_digits);
+
+ int32_t parsed_scale = 0;
+ if (dec.has_exponent) {
+ auto adjusted_exponent = dec.exponent;
+ parsed_scale =
+ -adjusted_exponent + static_cast<int32_t>(dec.fractional_digits.size());
+ } else {
+ parsed_scale = static_cast<int32_t>(dec.fractional_digits.size());
+ }
+
+ if (out != nullptr) {
+ if constexpr (std::is_same_v<Decimal32, DecimalClass>) {
+ uint32_t value{0};
+ ShiftAndAdd(dec.whole_digits, &value);
+ ShiftAndAdd(dec.fractional_digits, &value);
+ if (value > static_cast<uint32_t>(
+ std::numeric_limits<typename DecimalClass::ValueType>::max())) {
+ return Status::Invalid("The string '", s, "' cannot be represented as ",
+ type_name);
+ }
+
+ *out = DecimalClass(value);
+ if (dec.sign == '-') {
+ out->Negate();
+ }
+ } else {
Review Comment:
The two codepaths are almost identical except for `uint32_t` vs. `uint64_t`, you can probably reconcile them (perhaps something like `make_unsigned_t<typename DecimalClass::DigitType>`?)
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -92,6 +93,12 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
constexpr int kMantissaBits = RealTraits<Real>::kMantissaBits;
constexpr int kMantissaDigits = RealTraits<Real>::kMantissaDigits;
+ // to avoid precision and rounding issues, we'll unconditionally
+ // throw Decimal32 to the approx algorithm instead.
Review Comment:
Can we open a GH issue to improve this and reference it here?
(for example, a very simple solution would be to convert to Decimal64 and convert down to Decimal32 afterwards)
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +116,10 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (kMaxPrecision <= kMantissaDigits) {
Review Comment:
```suggestion
// The algorithm below requires the destination decimal type
// to be strictly more precise than the source float type
// (see `kSafeMulByTenTo` calculation).
if constexpr (kMaxPrecision <= kMantissaDigits) {
```
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -521,6 +750,23 @@ std::string Decimal128::ToString(int32_t scale) const {
return str;
}
+static inline void ShiftAndAdd(std::string_view input, uint32_t* out) {
+ const uint32_t len =
+ static_cast<uint32_t>(std::min(kInt32DecimalDigits + 1, input.size()));
+ if (len == 0) {
+ return;
+ }
+
+ uint32_t value = 0;
+ ARROW_CHECK(internal::ParseValue<UInt32Type>(input.data(), len, &value));
+
+ uint64_t tmp = *out;
+ tmp *= kUInt32PowersOfTen[len];
+ tmp += value;
+
+ *out = static_cast<uint32_t>(tmp & 0xFFFFFFFFU);
Review Comment:
So we're just losing bits if there's an overflow here? I'll note that `SimpleDecimalFromString` checks for overflows on other ops.
How about:
1. remove the `uint32_t`-output variant of this function
2. for Decimal32, call the `uint64_t` variant and reuse the existing limits check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1744654309
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -1116,6 +1168,10 @@ ArrayKernelExec GeneratePhysicalNumeric(detail::GetTypeId get_id) {
template <template <typename... Args> class Generator, typename... Args>
ArrayKernelExec GenerateDecimalToDecimal(detail::GetTypeId get_id) {
switch (get_id.id) {
+ // case Type::DECIMAL32:
+ // return Generator<Decimal32Type, Args...>::Exec;
+ // case Type::DECIMAL64:
+ // return Generator<Decimal64Type, Args...>::Exec;
Review Comment:
(why are these commented out?)
##########
cpp/src/arrow/array/builder_dict.h:
##########
@@ -298,6 +298,22 @@ class DictionaryBuilderBase : public ArrayBuilder {
return Append(std::string_view(value, length));
}
+ /// \brief Append a decimal (only for Decimal32Type)
Review Comment:
We can do all this with `if constexpr` now right? (Though this works fine too)
##########
cpp/src/arrow/type.cc:
##########
@@ -3301,8 +3356,18 @@ std::shared_ptr<Field> field(std::string name, std::shared_ptr<DataType> type,
}
std::shared_ptr<DataType> decimal(int32_t precision, int32_t scale) {
Review Comment:
The docstring for this function needs updating
##########
cpp/src/arrow/testing/gtest_util.h:
##########
@@ -171,7 +171,8 @@ using PrimitiveArrowTypes =
using TemporalArrowTypes =
::testing::Types<Date32Type, Date64Type, TimestampType, Time32Type, Time64Type>;
-using DecimalArrowTypes = ::testing::Types<Decimal128Type, Decimal256Type>;
+using DecimalArrowTypes =
+ ::testing::Types</*Decimal32Type, Decimal64Type,*/ Decimal128Type, Decimal256Type>;
Review Comment:
Ditto here. (Should we file issues to come back to these?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1745679965
##########
cpp/src/arrow/type.cc:
##########
@@ -3301,8 +3356,18 @@ std::shared_ptr<Field> field(std::string name, std::shared_ptr<DataType> type,
}
std::shared_ptr<DataType> decimal(int32_t precision, int32_t scale) {
- return precision <= Decimal128Type::kMaxPrecision ? decimal128(precision, scale)
- : decimal256(precision, scale);
+ return precision <= Decimal32Type::kMaxPrecision ? decimal32(precision, scale)
Review Comment:
Is it a breaking change? Users may get different data type and array type when precision is smaller than 18.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751059011
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
what is the `PhysicalType` here used for compared to the `T` itself? (I was just following the existing pattern for `Decimal128/Decimal256` when I did this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766946256
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +110,11 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (std::is_base_of_v<BasicDecimal32, DecimalType> &&
+ std::is_same_v<Real, double>) {
+ return DecimalType::FromReal(static_cast<float>(real), precision, scale);
Review Comment:
It's the right condition logically, though :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767126099
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
It's not really that unexpected honestly. The algorithm explicitly states that it can cause off-by-1 rounding errors because of the interleaved multiplication and rounded division that we're doing.
> // NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
In the case where we need this, we using a precision of 16 for Decimal64 which is "almost" the full precision of 18 and thus hitting the aforementioned "off-by-one error due to rounding" that is mentioned in the existing comment. By forcing the multiplication step to 1 instead of 2 in this case, we eliminate the off-by-one error (though we'd likely still run into it if we were using 17 or 18 precision which would already be using the minimum multiplication step of 1).
That said, the issue that @bkietz filed to possibly replace a lot of this logic with fewer steps (using multiplications by 5 etc.) could potentially alleviate this issue directly also.
If we really don't like this and are okay with the off-by-one rounding error mentioned in the comment, then I can adjust the tests to simply not test this case which has the off-by-one error for Decimal64.
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752947614
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
I added suggestions here https://github.com/apache/arrow/pull/43957#discussion_r1752947006
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763360419
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
Review Comment:
Speaking of which, are there any tests for summing decimal32 and decimal64?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763389919
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
Review Comment:
We guard against `length > kMaxDecimalBytes` above, so this is always 0?
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
Why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763391139
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
+ return Decimal32(value);
+ }
+
+ int32_t final_value = -1 * (is_negative && length < kMaxDecimalBytes);
+ final_value = SafeLeftShift(final_value, bits_offset * CHAR_BIT);
+ final_value |= value;
Review Comment:
This looks... complicated?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763455692
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
i've added some comments to better explain this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763448468
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
Review Comment:
I can get rid of the `max` call and just use `length - kMaxDecimalBytes` then as this would always be `length - kMaxDecimalBytes`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763455692
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
i've added some comments to better explain this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761373979
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,257 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal32 from one scale to another
+ Result<Decimal32> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal32 out;
+ auto dstatus = BasicDecimal32::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
+ return Status::Invalid("Invalid cast from Decimal32 value to ", sizeof(T),
+ "byte integer");
+ }
Review Comment:
you're right, this can't happen for decimal32
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761353151
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto remaining_digits = decimal_type.precision();
Review Comment:
inherited from the original implementation for decimal128/256, i'll rename the variable to `digits_to_generate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761602539
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
Review Comment:
It's just a sanity check of the random generation algorithm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1773720710
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -521,6 +750,23 @@ std::string Decimal128::ToString(int32_t scale) const {
return str;
}
+static inline void ShiftAndAdd(std::string_view input, uint32_t* out) {
+ const uint32_t len =
+ static_cast<uint32_t>(std::min(kInt32DecimalDigits + 1, input.size()));
+ if (len == 0) {
+ return;
+ }
+
+ uint32_t value = 0;
+ ARROW_CHECK(internal::ParseValue<UInt32Type>(input.data(), len, &value));
+
+ uint64_t tmp = *out;
+ tmp *= kUInt32PowersOfTen[len];
+ tmp += value;
+
+ *out = static_cast<uint32_t>(tmp & 0xFFFFFFFFU);
Review Comment:
`SimpleDecimalFromString` has the same overflow check for both Decimal32 and Decimal64 though?
that said, removing the `uint32_t` variant and just using the `uint64_t` variant does simplify this, so I'll do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2332009931
> > Following the same pattern, this change means that using `decimal(precision, scale)` instead of the specific `decimal32`/`decimal64`/`decimal128`/`decimal256` functions results in the following functionality
>
> I'm afraid this may massively break user code. I would suggest another approach:
>
> * deprecate the `decimal()` factory while keeping its current behavior of always returning at least `decimal128`
> * introduce a new `smallest_decimal()` factory that is documented to return the smallest possible type, and explicitly makes no guarantees about the stability of the return type
I just have the same concern. +1 on the proposed workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751077202
##########
cpp/src/arrow/compute/kernels/aggregate_var_std.cc:
##########
@@ -46,15 +46,18 @@ struct VarStdState {
double ToDouble(T value) const {
return static_cast<double>(value);
}
+ double ToDouble(const Decimal32& value) const { return value.ToDouble(decimal_scale); }
+ double ToDouble(const Decimal64& value) const { return value.ToDouble(decimal_scale); }
double ToDouble(const Decimal128& value) const { return value.ToDouble(decimal_scale); }
double ToDouble(const Decimal256& value) const { return value.ToDouble(decimal_scale); }
// float/double/int64/decimal: calculate `m2` (sum((X-mean)^2)) with `two pass
// algorithm`
// https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass_algorithm
template <typename T = ArrowType>
- enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4)> Consume(
- const ArraySpan& array) {
+ enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4) ||
+ (!is_integer_type<T>::value && sizeof(CType) == 4)>
Review Comment:
fixed to have the SumType be Decimal64 now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752947006
##########
cpp/src/arrow/type_fwd.h:
##########
@@ -512,9 +528,28 @@ std::shared_ptr<DataType> fixed_size_binary(int32_t byte_width);
///
/// If the precision is greater than 38, a Decimal256Type is returned,
/// otherwise a Decimal128Type.
+///
+/// Deprecated: prefer `smallest_decimal` instead.
Review Comment:
`smallest_decimal` is soo0o long. `best_decimal`, `least_decimal`, `fast_decimal`...?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752054170
##########
cpp/src/arrow/engine/substrait/expression_internal.cc:
##########
@@ -951,6 +951,38 @@ struct ScalarToProtoImpl {
Status Visit(const MonthIntervalScalar& s) { return NotImplemented(s); }
Status Visit(const DayTimeIntervalScalar& s) { return NotImplemented(s); }
+ Status Visit(const Decimal32Scalar& s) {
Review Comment:
This is also a place where we'd benefit from a generic
##########
cpp/src/arrow/array/util.cc:
##########
@@ -152,6 +152,42 @@ class ArrayDataEndianSwapper {
return Status::OK();
}
+ Status Visit(const Decimal32Type& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[1]->size(), pool_));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ // NOTE: data_->length not trusted (see warning above)
+ const int64_t length = data_->buffers[1]->size() / Decimal32Type::kByteWidth;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = bit_util::FromBigEndian(data[i]);
+#else
+ new_data[i] = bit_util::FromLittleEndian(data[i]);
+#endif
Review Comment:
If it's not too annoying: I think the copy-pasta here has reached the critical mass where it'd be worthwhile to make Visit generic for decimal types, probably assisted by extracting `{Decimal32,Decimal64,Decimal128,Decimal256}::SwapEndian()`
##########
cpp/src/arrow/type.cc:
##########
@@ -1459,6 +1478,42 @@ int32_t DecimalType::DecimalSize(int32_t precision) {
return static_cast<int32_t>(std::ceil((precision / 8.0) * std::log2(10) + 1));
}
+// ----------------------------------------------------------------------
+// Decimal32 type
+
+Decimal32Type::Decimal32Type(int32_t precision, int32_t scale)
+ : DecimalType(type_id, 4, precision, scale) {
+ ARROW_CHECK_GE(precision, kMinPrecision);
+ ARROW_CHECK_LE(precision, kMaxPrecision);
Review Comment:
Would it be worthwhile to extract
```
template <typename D>
Status ValidateDecimalPrecision(int32_t precision) {
if (precision < D::kMinPrecision || precision > D::kMaxPrecision) {
return Status::Invalid("Decimal precision out of range [", int32_t(D::kMinPrecision),
", ", int32_t(D::kMaxPrecision), "]: ", precision);
}
return Status::Ok();
}
```
?
Then we could write
```suggestion
ARROW_CHECK_OK(ValidatePrecisionScale<Decimal32Type>());
```
And in the `::Make` functions below
```
Result<std::shared_ptr<DataType>> Decimal32Type::Make(int32_t precision, int32_t scale) {
RETURN_NOT_OK(ValidatePrecisionScale<Decimal32Type>(precision));
return std::make_shared<Decimal32Type>(precision, scale);
}
```
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
It's a potentially breaking change for user code like:
```
template <typename T>
enable_if_decimal_value<T> SetToMagic(T* ptr) {
constexpr char MAGIC[256] = ...;
if constexpr (std::is_same_v<T, Decimal128>) {
memcpy(ptr, &MAGIC, sizeof(Decimal128));
} else {
static_assert(std::is_same_v<T, Decimal256>); // error for Decimal32
memcpy(ptr, &MAGIC, sizeof(MAGIC));
}
}
```
I think that risk is outweighed by the greater weirdness of not including the new types however
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
"narrowest_decimal"?
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
I wonder if `std::array<char, 4>` or `span<char, 4>` would be a more doable replacement
##########
cpp/src/arrow/type_fwd.h:
##########
@@ -512,9 +528,28 @@ std::shared_ptr<DataType> fixed_size_binary(int32_t byte_width);
///
/// If the precision is greater than 38, a Decimal256Type is returned,
/// otherwise a Decimal128Type.
+///
+/// Deprecated: prefer `smallest_decimal` instead.
Review Comment:
We have an attribute macro which should provide compiler warnings when the deprecated function is used
```suggestion
///
/// Deprecated: prefer `smallest_decimal` instead.
ARROW_DEPRECATED("Deprecated in 18.0. Use `smallest_decimal` instead")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752268424
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
The issue comes down to `visit_data_inline.h` which uses `std::string_view` to call these `GetViewType` functions, so I don't think we could replace these `std::string_view` usages without significant refactors elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751080158
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
In theory, because of how the templates work, as long as the value types have the proper methods implemented (operators and methods) then it *should* be safe, with the exception of dealing with the overflows that needed to be managed with the `GetSumType` and so on which we've already caught
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761566440
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
By the way, the reason we upcast integers when summing them is to emulate what DB engines typically do. I don't think DB engines upcast decimals when summing them.
See e.g. https://www.postgresql.org/docs/current/functions-aggregate.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761553319
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -132,7 +142,13 @@ struct GetSumType<T, enable_if_integer<T>> {
};
template <typename T>
-struct GetSumType<T, enable_if_decimal<T>> {
+struct GetSumType<T, enable_if_decimal32<T>> {
+ using SumType = arrow::Decimal64;
Review Comment:
Decimal is **not** an integer type. The fact that it's coded as an integer is irrelevant.
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -132,7 +142,13 @@ struct GetSumType<T, enable_if_integer<T>> {
};
template <typename T>
-struct GetSumType<T, enable_if_decimal<T>> {
+struct GetSumType<T, enable_if_decimal32<T>> {
+ using SumType = arrow::Decimal64;
Review Comment:
Decimal is **NOT** an integer type. The fact that it's coded as an integer is irrelevant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761447575
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
You could use [CRTP](https://en.wikipedia.org/wiki/Curiously_recurring_template_pattern) :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1760857841
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
"least_decimal" is the best of these 3 suggestions ("best" and "fast" do not convey the right meaning), but "smallest" is easier to remember than "least" IMHO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761866741
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -132,7 +142,13 @@ struct GetSumType<T, enable_if_integer<T>> {
};
template <typename T>
-struct GetSumType<T, enable_if_decimal<T>> {
+struct GetSumType<T, enable_if_decimal32<T>> {
+ using SumType = arrow::Decimal64;
Review Comment:
that's a good point and I agree, i'll put it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1753456305
##########
cpp/src/parquet/arrow/test_util.h:
##########
@@ -142,7 +142,11 @@ template <int32_t byte_width>
static void random_decimals(int64_t n, uint32_t seed, int32_t precision, uint8_t* out) {
auto gen = ::arrow::random::RandomArrayGenerator(seed);
std::shared_ptr<Array> decimals;
- if constexpr (byte_width == 16) {
+ if constexpr (byte_width == 4) {
+ decimals = gen.Decimal32(::arrow::decimal32(precision, 0), n);
+ } else if constexpr (byte_width == 8) {
+ decimals = gen.Decimal64(::arrow::decimal32(precision, 0), n);
Review Comment:
I mean when bitwidth==8, why it's a decimal32 here 🤔?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767127531
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1230,65 +1275,72 @@ TYPED_TEST(TestDecimalToRealDouble, LargeValues) {
}
TYPED_TEST(TestDecimalToRealDouble, Precision) {
- // 2**63 + 2**11 (exactly representable in a double's 53 bits of precision)
- CheckDecimalToReal<TypeParam, double>("9223372036854777856", 0, 9.223372036854778e+18);
- CheckDecimalToReal<TypeParam, double>("-9223372036854777856", 0,
- -9.223372036854778e+18);
- // 2**64 - 2**11 (exactly representable in a double)
- CheckDecimalToReal<TypeParam, double>("18446744073709549568", 0, 1.844674407370955e+19);
- CheckDecimalToReal<TypeParam, double>("-18446744073709549568", 0,
- -1.844674407370955e+19);
- // 2**64 + 2**11 (exactly representable in a double)
- CheckDecimalToReal<TypeParam, double>("18446744073709555712", 0,
- 1.8446744073709556e+19);
- CheckDecimalToReal<TypeParam, double>("-18446744073709555712", 0,
- -1.8446744073709556e+19);
- // Almost 10**38 (minus 2**73)
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 0,
- 9.999999999999998e+37);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 0,
- -9.999999999999998e+37);
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 10,
- 9.999999999999998e+27);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 10,
- -9.999999999999998e+27);
- CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", -10,
- 9.999999999999998e+47);
- CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", -10,
- -9.999999999999998e+47);
+ if constexpr (TypeParam::kMaxPrecision >= 19) {
+ // 2**63 + 2**11 (exactly representable in a double's 53 bits of precision)
+ CheckDecimalToReal<TypeParam, double>("9223372036854777856", 0,
+ 9.223372036854778e+18);
+ CheckDecimalToReal<TypeParam, double>("-9223372036854777856", 0,
+ -9.223372036854778e+18);
+ // 2**64 - 2**11 (exactly representable in a double)
+ CheckDecimalToReal<TypeParam, double>("18446744073709549568", 0,
+ 1.844674407370955e+19);
+ CheckDecimalToReal<TypeParam, double>("-18446744073709549568", 0,
+ -1.844674407370955e+19);
+ // 2**64 + 2**11 (exactly representable in a double)
+ CheckDecimalToReal<TypeParam, double>("18446744073709555712", 0,
+ 1.8446744073709556e+19);
+ CheckDecimalToReal<TypeParam, double>("-18446744073709555712", 0,
+ -1.8446744073709556e+19);
+
+ // Almost 10**38 (minus 2**73)
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 0,
+ 9.999999999999998e+37);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 0,
+ -9.999999999999998e+37);
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", 10,
+ 9.999999999999998e+27);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", 10,
+ -9.999999999999998e+27);
+ CheckDecimalToReal<TypeParam, double>("99999999999999978859343891977453174784", -10,
+ 9.999999999999998e+47);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999978859343891977453174784", -10,
+ -9.999999999999998e+47);
+ }
// Integers are always exact
auto scale = TypeParam::kMaxScale - 1;
std::string seven = "7.";
seven.append(scale, '0');
CheckDecimalToReal<TypeParam, double>(seven, scale, 7.0);
CheckDecimalToReal<TypeParam, double>("-" + seven, scale, -7.0);
-
- CheckDecimalToReal<TypeParam, double>("99999999999999999999.0000000000000000", 16,
- 99999999999999999999.0);
- CheckDecimalToReal<TypeParam, double>("-99999999999999999999.0000000000000000", 16,
- -99999999999999999999.0);
+ if constexpr (TypeParam::kMaxPrecision >= 20) {
+ CheckDecimalToReal<TypeParam, double>("99999999999999999999.0000000000000000", 16,
+ 99999999999999999999.0);
+ CheckDecimalToReal<TypeParam, double>("-99999999999999999999.0000000000000000", 16,
+ -99999999999999999999.0);
+ }
// Small fractions are within one ULP
CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.9", 1, 9999999.9);
CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.9", 1, -9999999.9);
- CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.999999999999999", 15,
- 9999999.999999999999999);
- CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.999999999999999", 15,
- -9999999.999999999999999);
-
- // Large fractions are within 2^-52
- constexpr double epsilon = 2.220446049250313080847263336181640625e-16; // 2^-52
- CheckDecimalToRealWithinEpsilon<TypeParam, double>(
- "112334829348925.99070703983306884765625", 23, epsilon,
- 112334829348925.99070703983306884765625);
- CheckDecimalToRealWithinEpsilon<TypeParam, double>(
- "1.987748987892758765582589910934859345", 36, epsilon,
- 1.987748987892758765582589910934859345);
+ if constexpr (TypeParam::kMaxPrecision >= 23) {
+ CheckDecimalToRealWithinOneULP<TypeParam, double>("9999999.999999999999999", 15,
+ 9999999.999999999999999);
+ CheckDecimalToRealWithinOneULP<TypeParam, double>("-9999999.999999999999999", 15,
+ -9999999.999999999999999);
+ // Large fractions are within 2^-52
+ constexpr double epsilon = 2.220446049250313080847263336181640625e-16; // 2^-52
+ CheckDecimalToRealWithinEpsilon<TypeParam, double>(
+ "112334829348925.99070703983306884765625", 23, epsilon,
+ 112334829348925.99070703983306884765625);
+ CheckDecimalToRealWithinEpsilon<TypeParam, double>(
+ "1.987748987892758765582589910934859345", 36, epsilon,
+ 1.987748987892758765582589910934859345);
+ }
}
#endif // __MINGW32__
-TEST(Decimal128Test, TestFromBigEndian) {
+TEST(Decimal32Test, TestFromBigEndian) {
Review Comment:
added generic versions for most of them. the LeftShift/RightShift and Negate tests are sufficiently different in their values that it's harder to make them generic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767195342
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -156,7 +162,9 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
// NOTE: if `precision` is the full precision then the algorithm will
// lose the last digit. If `precision` is almost the full precision,
// there can be an off-by-one error due to rounding.
- const int mul_step = std::max(1, kMaxPrecision - precision);
+ constexpr int is_dec32_or_dec64 =
+ DecimalType::kByteWidth <= BasicDecimal64::kByteWidth;
+ const int mul_step = std::max(1, kMaxPrecision - precision - is_dec32_or_dec64);
Review Comment:
I've adjusted the tests to simply not hit this case anymore with the off-by-one rounding issue for decimal64.
Let me know if you prefer that to the above hack with explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763451885
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
Review Comment:
you're right, this should be 4 not 8 so that we avoid an undefined shift by 32 below this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763458374
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
+ return Decimal32(value);
+ }
+
+ int32_t final_value = -1 * (is_negative && length < kMaxDecimalBytes);
+ final_value = SafeLeftShift(final_value, bits_offset * CHAR_BIT);
+ final_value |= value;
Review Comment:
I adapted this from the `Decimal128::FromBigEndian` but now that I'm re-reading it, am I wrong to think that I could just do `FromBigEndian` on the int32_t? (not `uint32_t`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761533842
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
+
+ /// \brief Divide this number by the divisor and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \param[out] result the quotient
+ /// \param[out] remainder the remainder after the division
+ DecimalStatus Divide(const BasicDecimal32& divisor, BasicDecimal32* result,
+ BasicDecimal32* remainder) const;
+
+ /// \brief In-place division
+ BasicDecimal32& operator/=(const BasicDecimal32& right);
+ /// \brief Bitwise "or" between two BasicDecimal32s
+ BasicDecimal32& operator|=(const BasicDecimal32& right);
+ /// \brief Bitwise "and" between two BasicDecimal32s
+ BasicDecimal32& operator&=(const BasicDecimal32& right);
+ /// \brief Shift left by the given number of bits.
+ BasicDecimal32& operator<<=(uint32_t bits);
+
+ BasicDecimal32 operator<<(uint32_t bits) const {
+ auto res = *this;
+ res <<= bits;
+ return res;
+ }
+
+ /// \brief Shift right by the given number of bits.
+ ///
+ /// Negative values will sign-extend
+ BasicDecimal32& operator>>=(uint32_t bits);
+
+ BasicDecimal32 operator>>(uint32_t bits) const {
+ auto res = *this;
+ res >>= bits;
+ return res;
+ }
+
+ /// \brief Convert BasicDecimal32 from one scale to another
+ DecimalStatus Rescale(int32_t original_scale, int32_t new_scale,
+ BasicDecimal32* out) const;
+
+ void GetWholeAndFraction(int scale, BasicDecimal32* whole,
+ BasicDecimal32* fraction) const;
+
+ /// \brief Scale up.
+ BasicDecimal32 IncreaseScaleBy(int32_t increase_by) const;
+
+ /// \brief Scale down.
+ ///
+ /// - If 'round' is true, the right-most digits are dropped and the result value is
+ /// rounded up (+1 for +ve, -1 for -ve) based on the value of the dropped digits
+ /// (>= 10^reduce_by / 2).
+ /// - If 'round' is false, the right-most digits are simply dropped.
+ BasicDecimal32 ReduceScaleBy(int32_t reduce_by, bool round = true) const;
+
+ /// \brief Whether this number fits in the given precision
+ ///
+ /// Return true if the number of significant digits is less or equal to 'precision'.
+ bool FitsInPrecision(int32_t precision) const;
+
+ /// \brief Get the maximum valid unscaled decimal value.
+ static const BasicDecimal32& GetMaxValue();
+ /// \brief Get the maximum valid unscaled decimal value for the given precision.
+ static BasicDecimal32 GetMaxValue(int32_t precision);
+
+ /// \brief Get the maximum decimal value (is not a valid value).
+ static constexpr BasicDecimal32 GetMaxSentinel() {
+ return BasicDecimal32(std::numeric_limits<int32_t>::max());
+ }
+
+ /// \brief Get the minimum decimal value (is not a valid value).
+ static constexpr BasicDecimal32 GetMinSentinel() {
+ return BasicDecimal32(std::numeric_limits<int32_t>::min());
+ }
+
+ /// \brief Scale multiplier for a given scale value.
+ static const BasicDecimal32& GetScaleMultiplier(int32_t scale);
+ /// \brief Half-scale multiplier for a given scale value.
+ static const BasicDecimal32& GetHalfScaleMultiplier(int32_t scale);
+
+ explicit operator BasicDecimal64() const;
+};
+
+ARROW_EXPORT bool operator<(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator<=(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator>(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator>=(const BasicDecimal32& left, const BasicDecimal32& right);
+
+ARROW_EXPORT BasicDecimal32 operator-(const BasicDecimal32& self);
+ARROW_EXPORT BasicDecimal32 operator+(const BasicDecimal32& left,
+ const BasicDecimal32& right);
+ARROW_EXPORT BasicDecimal32 operator-(const BasicDecimal32& left,
+ const BasicDecimal32& right);
+ARROW_EXPORT BasicDecimal32 operator*(const BasicDecimal32& left,
+ const BasicDecimal32& right);
Review Comment:
If i try making them generic, it prevents implicit casting for these operators, causing compiler errors for things like `val < 0` as it can't work out whether to cast both to Decimal32/Decimal64 or cast both to int/int64 to determine which operator to call.
I also had to move these above their respective class definitions so that the inlined versions of `Abs` / `Negate` etc could use the `<` operator in them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761603065
##########
cpp/src/arrow/type.cc:
##########
@@ -480,7 +494,7 @@ Result<std::shared_ptr<DataType>> MaybeMergeNumericTypes(
ARROW_ASSIGN_OR_RAISE(const int32_t precision,
MaxDecimalDigitsForInteger(other_type->id()));
ARROW_ASSIGN_OR_RAISE(const auto promoted_decimal,
- DecimalType::Make(promoted_type->id(), precision, 0));
+ DecimalType::Make(promoted_type->id(), precision - 1, 0));
Review Comment:
Ah! Got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763429117
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -697,8 +919,150 @@ Status DecimalFromString(const char* type_name, std::string_view s, Decimal* out
return Status::OK();
}
+template <typename DecimalClass>
+Status SimpleDecimalFromString(const char* type_name, std::string_view s,
+ DecimalClass* out, int32_t* precision, int32_t* scale) {
+ if (s.empty()) {
+ return Status::Invalid("Empty string cannot be converted to ", type_name);
+ }
+
+ DecimalComponents dec;
+ if (!ParseDecimalComponents(s.data(), s.size(), &dec)) {
+ return Status::Invalid("The string '", s, "' is not a valid ", type_name, " number");
+ }
+
+ // count number of significant digits (without leading zeros)
+ size_t first_non_zero = dec.whole_digits.find_first_not_of('0');
+ size_t significant_digits = dec.fractional_digits.size();
+ if (first_non_zero != std::string::npos) {
+ significant_digits += dec.whole_digits.size() - first_non_zero;
+ }
+ int32_t parsed_precision = static_cast<int32_t>(significant_digits);
+
+ int32_t parsed_scale = 0;
+ if (dec.has_exponent) {
+ auto adjusted_exponent = dec.exponent;
+ parsed_scale =
+ -adjusted_exponent + static_cast<int32_t>(dec.fractional_digits.size());
+ } else {
+ parsed_scale = static_cast<int32_t>(dec.fractional_digits.size());
+ }
+
+ if (out != nullptr) {
+ if constexpr (std::is_same_v<Decimal32, DecimalClass>) {
Review Comment:
the `ShiftAndAdd` function handles the math for shifting `dec.whole_digits` before adding `dec.fractional_digits`
but if we want to just do that here explicitly rather than having a 32-bit overload for `ShiftAndAdd` i can do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767127900
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1389,6 +1435,838 @@ std::vector<CType> GetRandomNumbers(int32_t size) {
return ret;
}
+TEST(Decimal32Test, Multiply) {
+ ASSERT_EQ(Decimal32(60501), Decimal32(301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(-301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(301) * Decimal32(-201));
+
+ ASSERT_EQ(Decimal32(60501), Decimal32(-301) * Decimal32(-201));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ Decimal32 result = Decimal32(x) * Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) * y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Divide) {
+ ASSERT_EQ(Decimal32(66), Decimal32(20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(-20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(20100) / Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(66), Decimal32(-20100) / Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) / Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) / y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Rescale) {
+ ASSERT_OK_AND_EQ(Decimal32(11100), Decimal32(111).Rescale(0, 2));
+ ASSERT_OK_AND_EQ(Decimal32(111), Decimal32(11100).Rescale(2, 0));
+ ASSERT_OK_AND_EQ(Decimal32(5), Decimal32(500000).Rescale(6, 1));
+ ASSERT_OK_AND_EQ(Decimal32(500000), Decimal32(5).Rescale(1, 6));
+ ASSERT_RAISES(Invalid, Decimal32(555555).Rescale(6, 1));
+
+ // Test some random numbers.
+ for (auto original_scale : GetRandomNumbers<Int8Type>(16)) {
+ for (auto value : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 unscaled_value = Decimal32(value);
+ Decimal32 scaled_value = unscaled_value;
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 4;
+ new_scale++, scaled_value *= Decimal32(10)) {
+ ASSERT_OK_AND_EQ(scaled_value, unscaled_value.Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(unscaled_value, scaled_value.Rescale(new_scale, original_scale));
+ }
+ }
+ }
+
+ for (auto original_scale : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 value(1);
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 8;
+ new_scale++, value *= Decimal32(10)) {
+ Decimal32 negative_value = value * -1;
+ ASSERT_OK_AND_EQ(value, Decimal32(1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(negative_value, Decimal32(-1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(Decimal32(1), value.Rescale(new_scale, original_scale));
+ ASSERT_OK_AND_EQ(Decimal32(-1), negative_value.Rescale(new_scale, original_scale));
+ }
+ }
+}
+
+TEST(Decimal32Test, Mod) {
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) % Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) % y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Sign) {
+ ASSERT_EQ(1, Decimal32(999999).Sign());
+ ASSERT_EQ(-1, Decimal32(-999999).Sign());
+ ASSERT_EQ(1, Decimal32(0).Sign());
+}
+
+TEST(Decimal32Test, GetWholeAndFraction) {
+ Decimal32 value("123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
Review Comment:
added a helper via lambda like in other tests
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1389,6 +1435,838 @@ std::vector<CType> GetRandomNumbers(int32_t size) {
return ret;
}
+TEST(Decimal32Test, Multiply) {
+ ASSERT_EQ(Decimal32(60501), Decimal32(301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(-301) * Decimal32(201));
+
+ ASSERT_EQ(Decimal32(-60501), Decimal32(301) * Decimal32(-201));
+
+ ASSERT_EQ(Decimal32(60501), Decimal32(-301) * Decimal32(-201));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ Decimal32 result = Decimal32(x) * Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) * y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Divide) {
+ ASSERT_EQ(Decimal32(66), Decimal32(20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(-20100) / Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-66), Decimal32(20100) / Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(66), Decimal32(-20100) / Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) / Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) / y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Rescale) {
+ ASSERT_OK_AND_EQ(Decimal32(11100), Decimal32(111).Rescale(0, 2));
+ ASSERT_OK_AND_EQ(Decimal32(111), Decimal32(11100).Rescale(2, 0));
+ ASSERT_OK_AND_EQ(Decimal32(5), Decimal32(500000).Rescale(6, 1));
+ ASSERT_OK_AND_EQ(Decimal32(500000), Decimal32(5).Rescale(1, 6));
+ ASSERT_RAISES(Invalid, Decimal32(555555).Rescale(6, 1));
+
+ // Test some random numbers.
+ for (auto original_scale : GetRandomNumbers<Int8Type>(16)) {
+ for (auto value : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 unscaled_value = Decimal32(value);
+ Decimal32 scaled_value = unscaled_value;
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 4;
+ new_scale++, scaled_value *= Decimal32(10)) {
+ ASSERT_OK_AND_EQ(scaled_value, unscaled_value.Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(unscaled_value, scaled_value.Rescale(new_scale, original_scale));
+ }
+ }
+ }
+
+ for (auto original_scale : GetRandomNumbers<Int16Type>(16)) {
+ Decimal32 value(1);
+ for (int32_t new_scale = original_scale; new_scale < original_scale + 8;
+ new_scale++, value *= Decimal32(10)) {
+ Decimal32 negative_value = value * -1;
+ ASSERT_OK_AND_EQ(value, Decimal32(1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(negative_value, Decimal32(-1).Rescale(original_scale, new_scale));
+ ASSERT_OK_AND_EQ(Decimal32(1), value.Rescale(new_scale, original_scale));
+ ASSERT_OK_AND_EQ(Decimal32(-1), negative_value.Rescale(new_scale, original_scale));
+ }
+ }
+}
+
+TEST(Decimal32Test, Mod) {
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(301));
+
+ ASSERT_EQ(Decimal32(234), Decimal32(20100) % Decimal32(-301));
+
+ ASSERT_EQ(Decimal32(-234), Decimal32(-20100) % Decimal32(-301));
+
+ // Test some random numbers.
+ for (auto x : GetRandomNumbers<Int32Type>(16)) {
+ for (auto y : GetRandomNumbers<Int32Type>(16)) {
+ if (y == 0) {
+ continue;
+ }
+
+ Decimal32 result = Decimal32(x) % Decimal32(y);
+ ASSERT_EQ(Decimal32(static_cast<int64_t>(x) % y), result)
+ << " x: " << x << " y: " << y;
+ }
+ }
+}
+
+TEST(Decimal32Test, Sign) {
+ ASSERT_EQ(1, Decimal32(999999).Sign());
+ ASSERT_EQ(-1, Decimal32(-999999).Sign());
+ ASSERT_EQ(1, Decimal32(0).Sign());
+}
+
+TEST(Decimal32Test, GetWholeAndFraction) {
+ Decimal32 value("123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
+
+ value.GetWholeAndFraction(1, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(12345, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(6, out);
+
+ value.GetWholeAndFraction(5, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(1, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(23456, out);
+
+ value.GetWholeAndFraction(7, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(0, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(123456, out);
+}
+
+TEST(Decimal32Test, GetWholeAndFractionNegative) {
+ Decimal32 value("-123456");
+ Decimal32 whole;
+ Decimal32 fraction;
+ int32_t out;
+
+ value.GetWholeAndFraction(0, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-123456, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(0, out);
+
+ value.GetWholeAndFraction(1, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-12345, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-6, out);
+
+ value.GetWholeAndFraction(5, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(-1, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-23456, out);
+
+ value.GetWholeAndFraction(7, &whole, &fraction);
+ ASSERT_OK(whole.ToInteger(&out));
+ ASSERT_EQ(0, out);
+ ASSERT_OK(fraction.ToInteger(&out));
+ ASSERT_EQ(-123456, out);
+}
+
+TEST(Decimal32Test, IncreaseScale) {
+ Decimal32 result;
+ int32_t out;
+
+ result = Decimal32("1234").IncreaseScaleBy(0);
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2371852920
ping @pitrou for another round :) we're getting closer! haha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766939055
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +110,11 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (std::is_base_of_v<BasicDecimal32, DecimalType> &&
+ std::is_same_v<Real, double>) {
+ return DecimalType::FromReal(static_cast<float>(real), precision, scale);
Review Comment:
the example you give would essentially be the same condition (though more concise) which would always affect double->Decimal32 conversions. That said, using `FromPositiveRealApprox` is likely better than a blanket drop to float. I'll give this a try and make sure the tests still pass for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766932408
##########
cpp/src/arrow/util/decimal_internal.h:
##########
@@ -442,8 +516,9 @@ struct RealTraits<float> {
static constexpr float two_to_192(float x) { return x == 0 ? 0 : kFloatInf; }
static constexpr int kMantissaBits = 24;
- // ceil(log10(2 ^ kMantissaBits))
- static constexpr int kMantissaDigits = 8;
+ // log10(2 ^ kMantissaBits) ~= 7.2, let's be conservative to ensure more accuracy
+ // with our conversions for Decimal values
Review Comment:
Yea, we had a persistent off-by-one rounding issue with Decimal32 and Decimal64 for the `FromReal` tests which was fixed by this, and had no detrimental effect on Decimal128/256
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1766941181
##########
cpp/src/arrow/util/decimal_internal.h:
##########
@@ -442,8 +516,9 @@ struct RealTraits<float> {
static constexpr float two_to_192(float x) { return x == 0 ? 0 : kFloatInf; }
static constexpr int kMantissaBits = 24;
- // ceil(log10(2 ^ kMantissaBits))
- static constexpr int kMantissaDigits = 8;
+ // log10(2 ^ kMantissaBits) ~= 7.2, let's be conservative to ensure more accuracy
+ // with our conversions for Decimal values
Review Comment:
As I mentioned elsewhere, float->Decimal32 probably needs to fall back on the approx algorithm.
I'm a bit surprised for Decimal64, though. Is the RoundedRightShift right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761371756
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
I was following the pattern that was there for decimal128 and 256. I'll try inlining these
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1745888761
##########
cpp/src/arrow/type.cc:
##########
@@ -3301,8 +3356,18 @@ std::shared_ptr<Field> field(std::string name, std::shared_ptr<DataType> type,
}
std::shared_ptr<DataType> decimal(int32_t precision, int32_t scale) {
- return precision <= Decimal128Type::kMaxPrecision ? decimal128(precision, scale)
- : decimal256(precision, scale);
+ return precision <= Decimal32Type::kMaxPrecision ? decimal32(precision, scale)
Review Comment:
This is a breaking change, and personally I'd prefer we deprecate this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751083012
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
Doing a little digging, the reason why we can't change it to `int32_t` here is because all of the decimal types are treated as `fixed_size_binary` rather than `numeric` in the rest of the codebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752241450
##########
cpp/src/arrow/array/util.cc:
##########
@@ -152,6 +152,42 @@ class ArrayDataEndianSwapper {
return Status::OK();
}
+ Status Visit(const Decimal32Type& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[1]->size(), pool_));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ // NOTE: data_->length not trusted (see warning above)
+ const int64_t length = data_->buffers[1]->size() / Decimal32Type::kByteWidth;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = bit_util::FromBigEndian(data[i]);
+#else
+ new_data[i] = bit_util::FromLittleEndian(data[i]);
+#endif
Review Comment:
Made this change, let me know what you think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752256148
##########
cpp/src/arrow/engine/substrait/expression_internal.cc:
##########
@@ -951,6 +951,38 @@ struct ScalarToProtoImpl {
Status Visit(const MonthIntervalScalar& s) { return NotImplemented(s); }
Status Visit(const DayTimeIntervalScalar& s) { return NotImplemented(s); }
+ Status Visit(const Decimal32Scalar& s) {
Review Comment:
Converted to generic
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1753322432
##########
cpp/src/parquet/arrow/test_util.h:
##########
@@ -142,7 +142,11 @@ template <int32_t byte_width>
static void random_decimals(int64_t n, uint32_t seed, int32_t precision, uint8_t* out) {
auto gen = ::arrow::random::RandomArrayGenerator(seed);
std::shared_ptr<Array> decimals;
- if constexpr (byte_width == 16) {
+ if constexpr (byte_width == 4) {
+ decimals = gen.Decimal32(::arrow::decimal32(precision, 0), n);
+ } else if constexpr (byte_width == 8) {
+ decimals = gen.Decimal64(::arrow::decimal32(precision, 0), n);
Review Comment:
why ::arrow::decimal32?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1753486228
##########
cpp/src/parquet/arrow/test_util.h:
##########
@@ -142,7 +142,11 @@ template <int32_t byte_width>
static void random_decimals(int64_t n, uint32_t seed, int32_t precision, uint8_t* out) {
auto gen = ::arrow::random::RandomArrayGenerator(seed);
std::shared_ptr<Array> decimals;
- if constexpr (byte_width == 16) {
+ if constexpr (byte_width == 4) {
+ decimals = gen.Decimal32(::arrow::decimal32(precision, 0), n);
+ } else if constexpr (byte_width == 8) {
+ decimals = gen.Decimal64(::arrow::decimal32(precision, 0), n);
Review Comment:
ah...good catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #43957:
URL: https://github.com/apache/arrow/pull/43957#issuecomment-2329835655
:warning: GitHub issue #43956 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1746083477
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -969,11 +1009,17 @@ Result<std::shared_ptr<DataType>> GetDecimal(const RjObject& json_type) {
bit_width = maybe_bit_width.ValueOrDie();
}
- if (bit_width == 128) {
- return decimal128(precision, scale);
- } else if (bit_width == 256) {
- return decimal256(precision, scale);
+ switch (bit_width) {
+ case 32:
+ return decimal32(precision, scale);
+ case 64:
+ return decimal64(precision, scale);
+ case 128:
+ return decimal128(precision, scale);
+ case 256:
+ return decimal256(precision, scale);
}
+
return Status::Invalid("Only 128 bit and 256 Decimals are supported. Received",
bit_width);
Review Comment:
Could you update this message too?
##########
cpp/src/arrow/c/bridge.cc:
##########
@@ -1253,6 +1253,10 @@ struct SchemaImporter {
type_ = decimal128(prec_scale[0], prec_scale[1]);
} else if (prec_scale[2] == 256) {
type_ = decimal256(prec_scale[0], prec_scale[1]);
+ } else if (prec_scale[2] == 32) {
+ type_ = decimal32(prec_scale[0], prec_scale[1]);
+ } else if (prec_scale[2] == 64) {
+ type_ = decimal64(prec_scale[0], prec_scale[1]);
Review Comment:
Can we use 32 -> 64 -> 128 -> 256 order here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1753422284
##########
cpp/src/parquet/arrow/test_util.h:
##########
@@ -142,7 +142,11 @@ template <int32_t byte_width>
static void random_decimals(int64_t n, uint32_t seed, int32_t precision, uint8_t* out) {
auto gen = ::arrow::random::RandomArrayGenerator(seed);
std::shared_ptr<Array> decimals;
- if constexpr (byte_width == 16) {
+ if constexpr (byte_width == 4) {
+ decimals = gen.Decimal32(::arrow::decimal32(precision, 0), n);
+ } else if constexpr (byte_width == 8) {
+ decimals = gen.Decimal64(::arrow::decimal32(precision, 0), n);
Review Comment:
presumably to disambiguate `parquet::arrow` and `arrow` namespaces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761042655
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
Why not `Decimal32Type`?
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -132,7 +142,13 @@ struct GetSumType<T, enable_if_integer<T>> {
};
template <typename T>
-struct GetSumType<T, enable_if_decimal<T>> {
+struct GetSumType<T, enable_if_decimal32<T>> {
+ using SumType = arrow::Decimal64;
Review Comment:
Why? This makes things slightly irregular and more difficult to predict for the user.
##########
cpp/src/arrow/compute/kernels/aggregate_basic.cc:
##########
@@ -336,8 +336,8 @@ struct ProductImpl : public ScalarAggregator {
internal::VisitArrayValuesInline<ArrowType>(
data,
[&](typename TypeTraits<ArrowType>::CType value) {
- this->product =
- MultiplyTraits<AccType>::Multiply(*out_type, this->product, value);
+ this->product = MultiplyTraits<AccType>::Multiply(
+ *out_type, this->product, static_cast<ProductType>(value));
Review Comment:
FTR, why was it necessary to add these casts?
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
Review Comment:
This can be massively simplified to:
```suggestion
static_assert(kMaxDigitsInInteger >= Decimal32Type::kMaxPrecision);
```
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
This is an internal header, no user code should ever use anything from here.
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,257 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal32 from one scale to another
+ Result<Decimal32> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal32 out;
+ auto dstatus = BasicDecimal32::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
+ return Status::Invalid("Invalid cast from Decimal32 value to ", sizeof(T),
+ "byte integer");
+ }
Review Comment:
Can this even happen?
##########
cpp/src/arrow/type_test.cc:
##########
@@ -1241,15 +1245,15 @@ TEST_F(TestUnifySchemas, Decimal) {
CheckPromoteTo(decimal256(3, -2), decimal256(5, -2), decimal256(5, -2), options);
// int32() is essentially decimal128(10, 0)
- CheckPromoteTo(int32(), decimal128(3, 2), decimal128(12, 2), options);
- CheckPromoteTo(int32(), decimal128(3, -2), decimal128(10, 0), options);
- CheckPromoteTo(int64(), decimal128(38, 37), decimal256(56, 37), options);
+ CheckPromoteTo(int32(), decimal128(3, 2), decimal128(11, 2), options);
+ CheckPromoteTo(int32(), decimal128(3, -2), decimal128(9, 0), options);
+ CheckPromoteTo(int64(), decimal128(38, 37), decimal256(55, 37), options);
CheckUnifyFailsTypeError(decimal256(1, 0), decimal128(1, 0), options);
options.promote_numeric_width = true;
CheckPromoteTo(decimal128(3, 2), decimal256(5, 2), decimal256(5, 2), options);
- CheckPromoteTo(int32(), decimal128(38, 37), decimal256(47, 37), options);
+ CheckPromoteTo(int32(), decimal128(38, 37), decimal256(46, 37), options);
CheckUnifyFailsInvalid(decimal128(38, 10), decimal256(76, 5), options);
Review Comment:
Can you add a couple tests that `smallest_decimal` returns the right type for a given precision?
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
Review Comment:
1. why not define those in the base generic class?
2. since the implementation of each method is very short, perhaps inline it in the header here?
##########
cpp/src/arrow/type.cc:
##########
@@ -1459,20 +1478,51 @@ int32_t DecimalType::DecimalSize(int32_t precision) {
return static_cast<int32_t>(std::ceil((precision / 8.0) * std::log2(10) + 1));
}
+template <typename D>
+Status ValidateDecimalPrecision(int32_t precision) {
+ if (precision < D::kMinPrecision || precision > D::kMaxPrecision) {
+ return Status::Invalid("Decimal precision out of range [", int32_t(D::kMinPrecision),
+ ", ", int32_t(D::kMaxPrecision), "]: ", precision);
+ }
+ return Status::OK();
+}
+
Review Comment:
Make this `static` or anonymous?
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
Review Comment:
This comment is outdated, remove it?
##########
docs/source/status.rst:
##########
@@ -44,6 +44,10 @@ Data Types
+-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
| Float32/64 | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
+-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
+| Decimal32 | ✓ | | ✓ | | | | | | |
++-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
+| Decimal64 | ✓ | | ✓ | | | | | | |
++-------------------+-------+-------+-------+----+-------+-------+-------+-------+-----------+
Review Comment:
Go isn't part of this PR, is it?
##########
cpp/src/arrow/type.cc:
##########
@@ -480,7 +494,7 @@ Result<std::shared_ptr<DataType>> MaybeMergeNumericTypes(
ARROW_ASSIGN_OR_RAISE(const int32_t precision,
MaxDecimalDigitsForInteger(other_type->id()));
ARROW_ASSIGN_OR_RAISE(const auto promoted_decimal,
- DecimalType::Make(promoted_type->id(), precision, 0));
+ DecimalType::Make(promoted_type->id(), precision - 1, 0));
Review Comment:
Why this change?
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
Review Comment:
`BaseType` sounds like a base class, perhaps call it `DigitType` or simply `Digit`.
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -969,12 +988,18 @@ Result<std::shared_ptr<DataType>> GetDecimal(const RjObject& json_type) {
bit_width = maybe_bit_width.ValueOrDie();
}
- if (bit_width == 128) {
- return decimal128(precision, scale);
- } else if (bit_width == 256) {
- return decimal256(precision, scale);
+ switch (bit_width) {
+ case 32:
+ return decimal32(precision, scale);
+ case 64:
+ return decimal64(precision, scale);
+ case 128:
+ return decimal128(precision, scale);
+ case 256:
+ return decimal256(precision, scale);
}
- return Status::Invalid("Only 128 bit and 256 Decimals are supported. Received",
+
+ return Status::Invalid("Only 32/64/128/256-bit Decimals are supported. Received",
Review Comment:
Let's add the required space at the end
```suggestion
return Status::Invalid("Only 32/64/128/256-bit Decimals are supported. Received ",
```
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto remaining_digits = decimal_type.precision();
+ auto values = checked_pointer_cast<Int32Array>(rng_->Int32(
+ size, -1 * MaxDecimalInteger(remaining_digits),
+ MaxDecimalInteger(remaining_digits), null_probability, alignment, memory_pool));
+
+ DecimalBuilderType builder(type_, memory_pool, alignment);
+ ABORT_NOT_OK(builder.Reserve(size));
+
+ for (int64_t i = 0; i < size; ++i) {
+ if (values->IsValid(i)) {
+ builder.UnsafeAppend(DecimalValue{values->Value(i)});
+ } else {
+ builder.UnsafeAppendNull();
+ }
+ }
+ std::shared_ptr<Array> array;
+ ABORT_NOT_OK(builder.Finish(&array));
+ return array;
+ }
+};
+
+template <>
+struct DecimalGenerator<Decimal64Type> {
Review Comment:
Hmm, can you perhaps refactor those two specialisations for `Decimal32Type` and `Decimal64Type` into a single one (perhaps `SmallDecimalGenerator<T>`)?
##########
cpp/src/arrow/testing/random.cc:
##########
@@ -343,8 +343,119 @@ struct DecimalGenerator {
}
};
+template <>
+struct DecimalGenerator<Decimal32Type> {
+ using DecimalBuilderType = typename TypeTraits<Decimal32Type>::BuilderType;
+ using DecimalValue = typename DecimalBuilderType::ValueType;
+
+ std::shared_ptr<DataType> type_;
+ RandomArrayGenerator* rng_;
+
+ static int32_t MaxDecimalInteger(int32_t digits) {
+ return static_cast<int32_t>(std::ceil(std::pow(10.0, digits))) - 1;
+ }
+
+ std::shared_ptr<Array> MakeRandomArray(int64_t size, double null_probability,
+ int64_t alignment, MemoryPool* memory_pool) {
+ static constexpr int32_t kMaxDigitsInInteger = 9;
+ static constexpr int kNumIntegers = Decimal32Type::kByteWidth / 4;
+ static_assert(
+ kNumIntegers == (Decimal32Type::kMaxPrecision + kMaxDigitsInInteger - 1) /
+ (kMaxDigitsInInteger + 1),
+ "inconsistent decimal metadata: kMaxPrecision doesn't match kByteWidth");
+
+ // First generate separate random values for individual components:
+ // boolean sign (including null-ness), and uint64 "digits" in big endian order.
+ const auto& decimal_type = checked_cast<const DecimalType&>(*type_);
+
+ auto remaining_digits = decimal_type.precision();
Review Comment:
Why "remaining"?
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
Review Comment:
There's already a ton of "basic" types here, can you name it perhaps `SmallBasicDecimal`?
##########
cpp/src/arrow/testing/gtest_util.h:
##########
@@ -171,7 +171,8 @@ using PrimitiveArrowTypes =
using TemporalArrowTypes =
::testing::Types<Date32Type, Date64Type, TimestampType, Time32Type, Time64Type>;
-using DecimalArrowTypes = ::testing::Types<Decimal128Type, Decimal256Type>;
+using DecimalArrowTypes =
+ ::testing::Types</*Decimal32Type, Decimal64Type,*/ Decimal128Type, Decimal256Type>;
Review Comment:
But it's going to be a separate PR, right?
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
+ int32_t CountLeadingBinaryZeros() const;
+
+ constexpr uint64_t low_bits() const { return static_cast<uint64_t>(value_); }
+
+ protected:
+ BaseType value_;
+};
+
+class BasicDecimal64;
+
+class ARROW_EXPORT BasicDecimal32 : public BasicDecimal<int32_t> {
+ public:
+ using BasicDecimal<int32_t>::BasicDecimal;
+ using ValueType = int32_t;
+
+ /// \brief Negate the current value (in-place)
+ BasicDecimal32& Negate();
+ /// \brief Absolute value (in-place)
+ BasicDecimal32& Abs();
+ /// \brief Absolute value
+ static BasicDecimal32 Abs(const BasicDecimal32& left);
+ /// \brief Add a number to this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator+=(const BasicDecimal32& right);
+ /// \brief Subtract a number from this one. The result is truncated to 32 bits.
+ BasicDecimal32& operator-=(const BasicDecimal32& right);
+ /// \brief Multiply this number by another. The result is truncated to 32 bits.
+ BasicDecimal32& operator*=(const BasicDecimal32& right);
+
+ /// \brief Divide this number by the divisor and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \param[out] result the quotient
+ /// \param[out] remainder the remainder after the division
+ DecimalStatus Divide(const BasicDecimal32& divisor, BasicDecimal32* result,
+ BasicDecimal32* remainder) const;
+
+ /// \brief In-place division
+ BasicDecimal32& operator/=(const BasicDecimal32& right);
+ /// \brief Bitwise "or" between two BasicDecimal32s
+ BasicDecimal32& operator|=(const BasicDecimal32& right);
+ /// \brief Bitwise "and" between two BasicDecimal32s
+ BasicDecimal32& operator&=(const BasicDecimal32& right);
+ /// \brief Shift left by the given number of bits.
+ BasicDecimal32& operator<<=(uint32_t bits);
+
+ BasicDecimal32 operator<<(uint32_t bits) const {
+ auto res = *this;
+ res <<= bits;
+ return res;
+ }
+
+ /// \brief Shift right by the given number of bits.
+ ///
+ /// Negative values will sign-extend
+ BasicDecimal32& operator>>=(uint32_t bits);
+
+ BasicDecimal32 operator>>(uint32_t bits) const {
+ auto res = *this;
+ res >>= bits;
+ return res;
+ }
+
+ /// \brief Convert BasicDecimal32 from one scale to another
+ DecimalStatus Rescale(int32_t original_scale, int32_t new_scale,
+ BasicDecimal32* out) const;
+
+ void GetWholeAndFraction(int scale, BasicDecimal32* whole,
+ BasicDecimal32* fraction) const;
+
+ /// \brief Scale up.
+ BasicDecimal32 IncreaseScaleBy(int32_t increase_by) const;
+
+ /// \brief Scale down.
+ ///
+ /// - If 'round' is true, the right-most digits are dropped and the result value is
+ /// rounded up (+1 for +ve, -1 for -ve) based on the value of the dropped digits
+ /// (>= 10^reduce_by / 2).
+ /// - If 'round' is false, the right-most digits are simply dropped.
+ BasicDecimal32 ReduceScaleBy(int32_t reduce_by, bool round = true) const;
+
+ /// \brief Whether this number fits in the given precision
+ ///
+ /// Return true if the number of significant digits is less or equal to 'precision'.
+ bool FitsInPrecision(int32_t precision) const;
+
+ /// \brief Get the maximum valid unscaled decimal value.
+ static const BasicDecimal32& GetMaxValue();
+ /// \brief Get the maximum valid unscaled decimal value for the given precision.
+ static BasicDecimal32 GetMaxValue(int32_t precision);
+
+ /// \brief Get the maximum decimal value (is not a valid value).
+ static constexpr BasicDecimal32 GetMaxSentinel() {
+ return BasicDecimal32(std::numeric_limits<int32_t>::max());
+ }
+
+ /// \brief Get the minimum decimal value (is not a valid value).
+ static constexpr BasicDecimal32 GetMinSentinel() {
+ return BasicDecimal32(std::numeric_limits<int32_t>::min());
+ }
+
+ /// \brief Scale multiplier for a given scale value.
+ static const BasicDecimal32& GetScaleMultiplier(int32_t scale);
+ /// \brief Half-scale multiplier for a given scale value.
+ static const BasicDecimal32& GetHalfScaleMultiplier(int32_t scale);
+
+ explicit operator BasicDecimal64() const;
+};
+
+ARROW_EXPORT bool operator<(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator<=(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator>(const BasicDecimal32& left, const BasicDecimal32& right);
+ARROW_EXPORT bool operator>=(const BasicDecimal32& left, const BasicDecimal32& right);
+
+ARROW_EXPORT BasicDecimal32 operator-(const BasicDecimal32& self);
+ARROW_EXPORT BasicDecimal32 operator+(const BasicDecimal32& left,
+ const BasicDecimal32& right);
+ARROW_EXPORT BasicDecimal32 operator-(const BasicDecimal32& left,
+ const BasicDecimal32& right);
+ARROW_EXPORT BasicDecimal32 operator*(const BasicDecimal32& left,
+ const BasicDecimal32& right);
Review Comment:
Similarly, these ones can probably be 1) made generic 2) be inlined in this header
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,257 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal32 from one scale to another
+ Result<Decimal32> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal32 out;
+ auto dstatus = BasicDecimal32::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
+ return Status::Invalid("Invalid cast from Decimal32 value to ", sizeof(T),
+ "byte integer");
+ }
+ return static_cast<T>(value_);
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Status ToInteger(T* out) const {
+ return ToInteger<T>().Value(out);
+ }
+
+ /// \brief Convert to a floating-point number (scaled)
+ float ToFloat(int32_t scale) const;
+ /// \brief Convert to a floating-point number (scaled)
+ double ToDouble(int32_t scale) const;
+
+ /// \brief Convert to a floating-point number (scaled)
+ template <typename T, typename = std::enable_if_t<std::is_floating_point_v<T>>>
+ T ToReal(int32_t scale) const {
+ static_assert(std::is_same_v<T, float> || std::is_same_v<T, double>,
+ "Unexpected floating-point type");
+ if constexpr (std::is_same_v<T, float>) {
+ return ToFloat(scale);
+ } else {
+ return ToDouble(scale);
+ }
+ }
+
+ ARROW_FRIEND_EXPORT friend std::ostream& operator<<(std::ostream& os,
+ const Decimal32& decimal);
+
+ private:
+ /// Converts internal error code to Status
+ Status ToArrowStatus(DecimalStatus dstatus) const;
+};
+
+class ARROW_EXPORT Decimal64 : public BasicDecimal64 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal64::BasicDecimal64;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal64 from a BasicDecimal64
+ constexpr Decimal64(const BasicDecimal64& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal64(value) {}
+
+ explicit Decimal64(const BasicDecimal32& value) noexcept
+ : BasicDecimal64(static_cast<int64_t>(value.value())) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal64(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal64 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal64() noexcept : BasicDecimal64() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal64, Decimal64>> Divide(const Decimal64& divisor) const {
+ std::pair<Decimal64, Decimal64> result;
+ auto dstatus = BasicDecimal64::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal64 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal64* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal64> FromString(std::string_view s);
+ static Result<Decimal64> FromString(const std::string& s);
+ static Result<Decimal64> FromString(const char* s);
+
+ static Result<Decimal64> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal64> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal64> FromBigEndian(const uint8_t* data, int32_t length);
+
+ /// \brief Convert Decimal64 from one scale to another
+ Result<Decimal64> Rescale(int32_t original_scale, int32_t new_scale) const {
+ Decimal64 out;
+ auto dstatus = BasicDecimal64::Rescale(original_scale, new_scale, &out);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return out;
+ }
+
+ /// \brief Convert to a signed integer
+ template <typename T, typename = internal::EnableIfIsOneOf<T, int32_t, int64_t>>
+ Result<T> ToInteger() const {
+ constexpr auto min_value = std::numeric_limits<T>::min();
+ constexpr auto max_value = std::numeric_limits<T>::max();
+ const auto& self = *this;
+ if (self < min_value || self > max_value) {
Review Comment:
This will issue pointless comparisons if T is `int64_t`.
##########
cpp/src/arrow/util/basic_decimal.h:
##########
@@ -166,6 +167,317 @@ class ARROW_EXPORT GenericBasicDecimal {
}
};
+template <typename BaseType>
+class ARROW_EXPORT BasicDecimal {
+ public:
+ static_assert(
+ std::is_same_v<BaseType, int32_t> || std::is_same_v<BaseType, int64_t>,
+ "for bitwidths larger than 64 bits use BasicDecimal128 and BasicDecimal256");
+
+ static constexpr int kMaxPrecision = std::numeric_limits<BaseType>::digits10;
+ static constexpr int kMaxScale = kMaxPrecision;
+ static constexpr int kBitWidth = sizeof(BaseType) * CHAR_BIT;
+ static constexpr int kByteWidth = sizeof(BaseType);
+
+ /// \brief Empty constructor creates a decimal with a value of 0.
+ constexpr BasicDecimal() noexcept : value_(0) {}
+
+ /// \brief Create a decimal from any integer not wider than 64 bits.
+ template <typename T,
+ typename = typename std::enable_if<
+ std::is_integral<T>::value && (sizeof(T) <= sizeof(int64_t)), T>::type>
+ constexpr BasicDecimal(T value) noexcept // NOLINT(runtime/explicit)
+ : value_(static_cast<BaseType>(value)) {}
+
+ /// \brief Create a decimal from an array of bytes.
+ ///
+ /// Bytes are assumed to be in native-endian byte order.
+ explicit BasicDecimal(const uint8_t* bytes) { memcpy(&value_, bytes, sizeof(value_)); }
+
+ const uint8_t* native_endian_bytes() const {
+ return reinterpret_cast<const uint8_t*>(&value_);
+ }
+
+ uint8_t* mutable_native_endian_bytes() { return reinterpret_cast<uint8_t*>(&value_); }
+
+ /// \brief Return the raw bytes of the value in native-endian byte order.
+ std::array<uint8_t, kByteWidth> ToBytes() const {
+ std::array<uint8_t, kByteWidth> out{{0}};
+ memcpy(out.data(), &value_, kByteWidth);
+ return out;
+ }
+
+ /// \brief Copy the raw bytes of the value in native-endian byte order
+ void ToBytes(uint8_t* out) const { memcpy(out, &value_, kByteWidth); }
+
+ /// \brief Return 1 if positive or 0, -1 if strictly negative
+ int64_t Sign() const { return 1 | (value_ >> (sizeof(kBitWidth) - 1)); }
+
+ bool IsNegative() const { return value_ < 0; }
+
+ explicit operator bool() const { return value_ != 0; }
+
+ friend bool operator==(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ == right.value_;
+ }
+
+ friend bool operator!=(const BasicDecimal& left, const BasicDecimal& right) {
+ return left.value_ != right.value_;
+ }
+
+ BaseType value() const { return value_; }
+
+ /// \brief count hte number of leading binary zeroes.
Review Comment:
```suggestion
/// \brief count the number of leading binary zeroes.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1757833095
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
Alright. Can we name it something shorter? I have at least 3 suggestions:
- `best_decimal`
- `fast_decimal`
- `least_decimal`
Some of these naming conventions exist in the C++ standard library https://en.cppreference.com/w/cpp/types/integer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761338731
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
See the discussion [here](https://github.com/apache/arrow/pull/43957#discussion_r1750962486)
We do the same for `int32` accumulators having their Sum type and Accumulator type being upcasted to Int64, it makes sense that the accumulated sum of Decimal32 should be Decimal64 to avoid overflows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763363483
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
Review Comment:
Not yet, I was going to cover that when I do the rest of the Kernel additions for decimal32/decimal64 in a follow-up PR.
These were only added here to make everything compile due to adding the new types to the generalized `enable_if_decimal...` template helpers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767126648
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -896,22 +917,30 @@ TYPED_TEST_SUITE(TestDecimalFromRealFloat, DecimalTypes);
TYPED_TEST(TestDecimalFromRealFloat, SuccessConversion) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761566440
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
By the way, the reason we upcast integers when summing them is to emulate what DB engines typically do. I don't think DB engines upcast decimals when summing them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763465041
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
+ if (bits_offset == 8) {
+ return Decimal32(value);
+ }
+
+ int32_t final_value = -1 * (is_negative && length < kMaxDecimalBytes);
+ final_value = SafeLeftShift(final_value, bits_offset * CHAR_BIT);
+ final_value |= value;
Review Comment:
Ah, I can't just blindly do `FromBigEndian` because i'm not guaranteed that `length` is exactly 4 bytes, hence why this shifting logic is needed for negative values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1763448468
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -742,6 +1106,78 @@ static inline uint64_t UInt64FromBigEndian(const uint8_t* bytes, int32_t length)
return ::arrow::bit_util::FromBigEndian(result);
}
+Result<Decimal32> Decimal32::FromBigEndian(const uint8_t* bytes, int32_t length) {
+ static constexpr int32_t kMinDecimalBytes = 1;
+ static constexpr int32_t kMaxDecimalBytes = 4;
+
+ if (ARROW_PREDICT_FALSE(length < kMinDecimalBytes || length > kMaxDecimalBytes)) {
+ return Status::Invalid("Length of byte array passed to Decimal32::FromBigEndian was ",
+ length, ", but must be between ", kMinDecimalBytes, " and ",
+ kMaxDecimalBytes);
+ }
+
+ const bool is_negative = static_cast<int8_t>(bytes[0]) < 0;
+
+ uint32_t result = 0;
+ memcpy(reinterpret_cast<uint8_t*>(&result) + kMaxDecimalBytes - length, bytes, length);
+
+ const auto value = bit_util::FromBigEndian(result);
+ const int32_t bits_offset = std::max(0, length - kMaxDecimalBytes);
Review Comment:
I read this wrong, you're right. this would always be 0 which isn't what we want, oy. it should be reversed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1767111668
##########
cpp/src/arrow/util/decimal.cc:
##########
@@ -109,6 +110,11 @@ struct DecimalRealConversion : public BaseDecimalRealConversion {
return OverflowError(real, precision, scale);
}
+ if constexpr (std::is_base_of_v<BasicDecimal32, DecimalType> &&
+ std::is_same_v<Real, double>) {
+ return DecimalType::FromReal(static_cast<float>(real), precision, scale);
Review Comment:
Also that haha
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1761864918
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal64Type;
Review Comment:
That's a good point. we aren't changing the precision when we do this, so if it is necessary to avoid overflow, the type would need to be cast to increase the precision etc. I'll change it back to be the appropriate types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1773713451
##########
cpp/src/arrow/util/decimal_test.cc:
##########
@@ -1336,43 +1406,42 @@ TEST(Decimal128Test, TestFromBigEndian) {
// convert to big endian
std::reverse(native_endian.begin(), native_endian.end());
#endif
- ASSERT_OK_AND_EQ(complement, Decimal128::FromBigEndian(native_endian.data(), 16));
+ ASSERT_OK_AND_EQ(complement, TypeParam::FromBigEndian(native_endian.data(),
+ TypeParam::kByteWidth));
- value <<= 8;
- value += Decimal128(start);
+ value <<= 2;
+ value += TypeParam(start);
}
}
}
-TEST(Decimal128Test, TestFromBigEndianBadLength) {
- ASSERT_RAISES(Invalid, Decimal128::FromBigEndian(0, -1));
- ASSERT_RAISES(Invalid, Decimal128::FromBigEndian(0, 17));
+TYPED_TEST(TestBasicDecimalFunctionality, TestFromBigEndianBadLength) {
+ ASSERT_RAISES(Invalid, TypeParam::FromBigEndian(0, -1));
+ ASSERT_RAISES(Invalid, TypeParam::FromBigEndian(0, TypeParam::kByteWidth + 1));
}
-TEST(Decimal128Test, TestToInteger) {
- Decimal128 value1("1234");
- int32_t out1;
+TYPED_TEST(TestBasicDecimalFunctionality, TestToInteger) {
+ if constexpr (std::is_same_v<TypeParam, Decimal256>) {
Review Comment:
that's a holdover from before I decided not to add it to the `BasicFunctionalityDecimalTypes`, i'll remove it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752117354
##########
cpp/src/arrow/type_fwd.h:
##########
@@ -512,9 +528,28 @@ std::shared_ptr<DataType> fixed_size_binary(int32_t byte_width);
///
/// If the precision is greater than 38, a Decimal256Type is returned,
/// otherwise a Decimal128Type.
+///
+/// Deprecated: prefer `smallest_decimal` instead.
Review Comment:
I completely forgot about that! thanks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752151777
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
That was my thought too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "bkietz (via GitHub)" <gi...@apache.org>.
bkietz commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752668339
##########
cpp/src/arrow/array/builder_dict.h:
##########
@@ -298,20 +298,12 @@ class DictionaryBuilderBase : public ArrayBuilder {
return Append(std::string_view(value, length));
}
- /// \brief Append a decimal (only for Decimal128Type)
- template <typename T1 = T>
- enable_if_decimal128<T1, Status> Append(const Decimal128& value) {
- uint8_t data[16];
- value.ToBytes(data);
- return Append(data, 16);
- }
-
- /// \brief Append a decimal (only for Decimal128Type)
- template <typename T1 = T>
- enable_if_decimal256<T1, Status> Append(const Decimal256& value) {
- uint8_t data[32];
+ /// \brief Append a decimal (only for Decimal32/64/128/256 Type)
+ template <typename T1 = T, typename CType = typename TypeTraits<T1>::CType>
+ enable_if_decimal<T1, Status> Append(const CType& value) {
+ uint8_t data[T1::kByteWidth];
value.ToBytes(data);
- return Append(data, 32);
+ return Append(data, T1::kByteWidth);
Review Comment:
Should be equivalent, little bit nicer:
```suggestion
auto bytes = value.ToBytes();
return Append(bytes.data(), bytes.size());
```
##########
cpp/src/arrow/util/decimal.h:
##########
@@ -31,6 +31,263 @@
namespace arrow {
+class Decimal64;
+
+/// Represents a signed 32-bit decimal value in two's complement.
+/// Calulations wrap around and overflow is ignored.
+/// The max decimal precision that can be safely represented is
+/// 9 significant digits.
+///
+/// The implementation is split into two parts :
+///
+/// 1. BasicDecimal32
+/// - can be safely compiled to IR without references to libstdc++
+/// 2. Decimal32
+/// - has additional functionality on top of BasicDecimal32 to deal with
+/// strings and streams
+class ARROW_EXPORT Decimal32 : public BasicDecimal32 {
+ public:
+ /// \cond FALSE
+ // (need to avoid a duplicate definition in sphinx)
+ using BasicDecimal32::BasicDecimal32;
+ /// \endcond
+
+ /// \brief constructor creates a Decimal32 from a BasicDecimal32
+ constexpr Decimal32(const BasicDecimal32& value) noexcept // NOLINT runtime/explicit
+ : BasicDecimal32(value) {}
+
+ /// \brief Parse the number from a base 10 string representation
+ explicit Decimal32(const std::string& value);
+
+ /// \brief Empty constructor creates a Decimal32 with a value of 0
+ /// this is required for some older compilers
+ constexpr Decimal32() noexcept : BasicDecimal32() {}
+
+ /// \brief Divide this number by right and return the result.
+ ///
+ /// This operation is not destructive.
+ /// The answer rounds to zero. Signs work like:
+ /// 21 / 5 -> 4, 1
+ /// -21 / 5 -> -4, -1
+ /// 21 / -5 -> -4, 1
+ /// -21 / -5 -> 4, -1
+ /// \param[in] divisor the number to divide by
+ /// \return the pair of the quotient and the remainder
+ Result<std::pair<Decimal32, Decimal32>> Divide(const Decimal32& divisor) const {
+ std::pair<Decimal32, Decimal32> result;
+ auto dstatus = BasicDecimal32::Divide(divisor, &result.first, &result.second);
+ ARROW_RETURN_NOT_OK(ToArrowStatus(dstatus));
+ return result;
+ }
+
+ /// \brief Convert the Decimal32 value to a base 10 decimal string with the given scale
+ std::string ToString(int32_t scale) const;
+
+ /// \brief Convert the value to an integer string
+ std::string ToIntegerString() const;
+
+ /// \brief Cast this value to an int64_t
+ explicit operator int64_t() const;
+
+ explicit operator Decimal64() const;
+
+ /// \brief Convert a decimal string to a Decimal value, optionally including
+ /// precision and scale if they're passed in and not null.
+ static Status FromString(std::string_view s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const std::string& s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Status FromString(const char* s, Decimal32* out, int32_t* precision,
+ int32_t* scale = NULLPTR);
+ static Result<Decimal32> FromString(std::string_view s);
+ static Result<Decimal32> FromString(const std::string& s);
+ static Result<Decimal32> FromString(const char* s);
+
+ static Result<Decimal32> FromReal(double real, int32_t precision, int32_t scale);
+ static Result<Decimal32> FromReal(float real, int32_t precision, int32_t scale);
+
+ /// \brief Convert from a big-endian byte representation. The length must be
+ /// between 1 and 4
+ /// \return error statis if the length is an invalid value
+ static Result<Decimal32> FromBigEndian(const uint8_t* data, int32_t length);
+
+ static void SwapEndian(const uint32_t* old_data, uint32_t* const new_data,
+ const int64_t length);
Review Comment:
This makes the usage in util.cc a bit nicer, but as a method on Decimal itself it might cause confusion. I was imagining
```suggestion
Decimal32 SwapEndian() const;
```
Then util.cc would look like
```c++
template <typename T>
enable_if_decimal<T, Status> Visit(const T& type) {
using value_type = typename TypeTraits<T>::CType;
auto data = data_->buffers[1]->span_as<value_type>();
auto new_data = new_buffer->mutable_data_as<value_type>();
for (value_type v : data) {
*new_data++ = v.SwapEndian();
}
```
... actually now that I think of it, it'd also be pretty straightforward to just use ToBytes instead of adding SwapEndian:
```c++
template <typename T>
enable_if_decimal<T, Status> Visit(const T& type) {
using value_type = typename TypeTraits<T>::CType;
auto data = data_->buffers[1]->span_as<value_type>();
auto new_data = new_buffer->mutable_data_as<value_type>();
for (value_type v : data) {
auto bytes = v.ToBytes();
std::reverse(bytes.begin(), bytes.end());
memcpy(new_data++, &bytes, sizeof(bytes));
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1752944892
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
Yep. Some templates made the bad decision of overloading on the PhysicalType so integers can't be used as PhysicalType for other types that are not the numeric ones. This is specially problematic when trying to improve the support for half-float.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1750917414
##########
cpp/src/arrow/array/builder_dict.h:
##########
@@ -298,6 +298,22 @@ class DictionaryBuilderBase : public ArrayBuilder {
return Append(std::string_view(value, length));
}
+ /// \brief Append a decimal (only for Decimal32Type)
+ template <typename T1 = T>
+ enable_if_decimal32<T1, Status> Append(const Decimal32& value) {
+ uint8_t data[4];
+ value.ToBytes(data);
+ return Append(data, 4);
+ }
+
+ /// \brief Append a decimal (only for Decimal64Type)
+ template <typename T1 = T>
+ enable_if_decimal64<T1, Status> Append(const Decimal64& value) {
Review Comment:
These two can be passed by value as they super small. So even if these functions weren't inlined...
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -140,6 +140,30 @@ struct GetViewType<Type, enable_if_t<is_base_binary_type<Type>::value ||
static T LogicalValue(PhysicalType value) { return value; }
};
+template <>
+struct GetViewType<Decimal32Type> {
+ using T = Decimal32;
+ using PhysicalType = std::string_view;
Review Comment:
Creating an issue to figure out how to change these to integers later. It's tricky because a lot of kernels have things that would get very confused if you set `int32_t` here.
##########
cpp/src/arrow/compute/kernels/hash_aggregate.cc:
##########
@@ -892,8 +894,10 @@ struct GroupedVarStdImpl : public GroupedAggregator {
// float/double/int64/decimal: calculate `m2` (sum((X-mean)^2)) with
// `two pass algorithm` (see aggregate_var_std.cc)
template <typename T = Type>
- enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4), Status> ConsumeImpl(
- const ExecSpan& batch) {
+ enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4) ||
+ (!is_integer_type<T>::value && sizeof(CType) == 4),
Review Comment:
Similar concerns like in a previous comment about this leading to overflow.
##########
cpp/src/arrow/compute/kernels/codegen_internal.h:
##########
@@ -224,7 +258,9 @@ using enable_if_not_floating_value = enable_if_t<!std::is_floating_point<T>::val
template <typename T, typename R = T>
using enable_if_decimal_value =
- enable_if_t<std::is_same<Decimal128, T>::value || std::is_same<Decimal256, T>::value,
+ enable_if_t<std::is_same<Decimal32, T>::value || std::is_same<Decimal64, T>::value ||
+ std::is_same<Decimal128, T>::value ||
+ std::is_same<Decimal256, T>::value,
Review Comment:
Hard to make sure extending an existing predicate like this is safe without reviewing every use-case.
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
Has there been discussion about this namin already?
##########
cpp/src/arrow/ipc/metadata_internal.cc:
##########
@@ -278,13 +278,20 @@ Status ConcreteTypeFromFlatbuffer(flatbuf::Type type, const void* type_data,
return Status::OK();
case flatbuf::Type::Decimal: {
auto dec_type = static_cast<const flatbuf::Decimal*>(type_data);
- if (dec_type->bitWidth() == 128) {
- return Decimal128Type::Make(dec_type->precision(), dec_type->scale()).Value(out);
- } else if (dec_type->bitWidth() == 256) {
- return Decimal256Type::Make(dec_type->precision(), dec_type->scale()).Value(out);
- } else {
- return Status::Invalid("Library only supports 128-bit or 256-bit decimal values");
+ switch (dec_type->bitWidth()) {
+ case 32:
+ return Decimal32Type::Make(dec_type->precision(), dec_type->scale()).Value(out);
+ case 64:
+ return Decimal64Type::Make(dec_type->precision(), dec_type->scale()).Value(out);
+ case 128:
+ return Decimal128Type::Make(dec_type->precision(), dec_type->scale())
+ .Value(out);
+ case 256:
+ return Decimal256Type::Make(dec_type->precision(), dec_type->scale())
+ .Value(out);
}
+ return Status::Invalid(
+ "Library supports 32-bit, 64-bit, 128-bit or 256-bit decimal values");
Review Comment:
```suggestion
"Library only supports 32/64/128/256-bit decimal values");
```
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal32Type;
Review Comment:
This accumulator type should probably be `Decimal64Type` but I wonder if the code that uses this can adjust the precision accordingly: estimate a reasonable `s'` to hold the accumulation of `n` `decimal32(p, s)` values.
##########
cpp/src/arrow/integration/json_internal.cc:
##########
@@ -969,13 +1009,20 @@ Result<std::shared_ptr<DataType>> GetDecimal(const RjObject& json_type) {
bit_width = maybe_bit_width.ValueOrDie();
}
- if (bit_width == 128) {
- return decimal128(precision, scale);
- } else if (bit_width == 256) {
- return decimal256(precision, scale);
+ switch (bit_width) {
+ case 32:
+ return decimal32(precision, scale);
+ case 64:
+ return decimal64(precision, scale);
+ case 128:
+ return decimal128(precision, scale);
+ case 256:
+ return decimal256(precision, scale);
}
- return Status::Invalid("Only 128 bit and 256 Decimals are supported. Received",
- bit_width);
+
+ return Status::Invalid(
+ "Only 32 bit, 64 bit, 128 bit and 256 bit Decimals are supported. Received",
Review Comment:
```suggestion
"Only 32/64/128/256-bit Decimals are supported. Received",
```
##########
cpp/src/arrow/compute/kernels/aggregate_var_std.cc:
##########
@@ -46,15 +46,18 @@ struct VarStdState {
double ToDouble(T value) const {
return static_cast<double>(value);
}
+ double ToDouble(const Decimal32& value) const { return value.ToDouble(decimal_scale); }
+ double ToDouble(const Decimal64& value) const { return value.ToDouble(decimal_scale); }
double ToDouble(const Decimal128& value) const { return value.ToDouble(decimal_scale); }
double ToDouble(const Decimal256& value) const { return value.ToDouble(decimal_scale); }
// float/double/int64/decimal: calculate `m2` (sum((X-mean)^2)) with `two pass
// algorithm`
// https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Two-pass_algorithm
template <typename T = ArrowType>
- enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4)> Consume(
- const ArraySpan& array) {
+ enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4) ||
+ (!is_integer_type<T>::value && sizeof(CType) == 4)>
Review Comment:
Probably a bad idea to sum `decimal32` values with this loop that doesn't handle overflow. Related to my note above.
##########
cpp/src/arrow/compute/kernels/aggregate_internal.h:
##########
@@ -52,6 +52,16 @@ struct FindAccumulatorType<I, enable_if_floating_point<I>> {
using Type = DoubleType;
};
+template <typename I>
+struct FindAccumulatorType<I, enable_if_decimal32<I>> {
+ using Type = Decimal32Type;
Review Comment:
This is specially necessary if the scale of the decimal32 is big because that eats a lot of the available bits in 32-bit integers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751044470
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -667,8 +667,10 @@ static ScalarVector GetScalars() {
std::make_shared<BinaryViewScalar>(hello),
std::make_shared<FixedSizeBinaryScalar>(
hello, fixed_size_binary(static_cast<int32_t>(hello->size()))),
- std::make_shared<Decimal128Scalar>(Decimal128(10), decimal(16, 4)),
- std::make_shared<Decimal256Scalar>(Decimal256(10), decimal(76, 38)),
+ std::make_shared<Decimal32Scalar>(Decimal32(10), smallest_decimal(7, 4)),
+ std::make_shared<Decimal64Scalar>(Decimal64(10), smallest_decimal(12, 4)),
+ std::make_shared<Decimal128Scalar>(Decimal128(10), smallest_decimal(20, 4)),
+ std::make_shared<Decimal256Scalar>(Decimal256(10), smallest_decimal(76, 38)),
Review Comment:
`smallest_decimal` was the suggestion by @pitrou, no one made any objections and I doubt he's particularly tied to the name if you have a suggestion for a better one. What are your thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-43956: [C++][Format] Add initial Decimal32/Decimal64 implementations [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #43957:
URL: https://github.com/apache/arrow/pull/43957#discussion_r1751077410
##########
cpp/src/arrow/compute/kernels/hash_aggregate.cc:
##########
@@ -892,8 +894,10 @@ struct GroupedVarStdImpl : public GroupedAggregator {
// float/double/int64/decimal: calculate `m2` (sum((X-mean)^2)) with
// `two pass algorithm` (see aggregate_var_std.cc)
template <typename T = Type>
- enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4), Status> ConsumeImpl(
- const ExecSpan& batch) {
+ enable_if_t<is_floating_type<T>::value || (sizeof(CType) > 4) ||
+ (!is_integer_type<T>::value && sizeof(CType) == 4),
Review Comment:
fixed to have SumType for Decimal32 be Decimal64 now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org