You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@arrow.apache.org by gi...@apache.org on 2026/10/06 05:07:11 UTC
(arrow-rs) 01/01: fix(arrow-cast): round decimal to float conversions once (#10509)
This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch gh-readonly-queue/main/pr-10509-c5e390031a1443f97355ed80411d6885e14642ff
in repository https://gitbox.apache.org/repos/asf/arrow-rs.git
commit cb3bec2362d8a1fb98ca1114f6bd0626ce8ec428
Author: David López <da...@gmail.com>
AuthorDate: Tue Oct 6 05:06:52 2026 +0000
fix(arrow-cast): round decimal to float conversions once (#10509)
# Which issue does this PR close?
- Closes #10508.
# Rationale for this change
`cast(decimal, Float64)` and `cast(decimal, Float32)` could return a
float that is not the nearest
one to the decimal's exact value — see the issue for the full analysis
and the failing values.
The short version is two independent double roundings.
`single_decimal_to_float_lossy` computed
`f(x) / 10_f64.powi(scale)`, where `f(x)` rounds the unscaled integer
once it exceeds 2^53 and
`10_f64.powi(scale)` is itself 1 ULP off for some scales a
`DECIMAL(38,s)` can declare; dividing two
wrongly rounded operands does not give the correctly rounded quotient.
Separately, the `Float32`
target computed the `f64` result and narrowed it with `as f32`, rounding
a second time — a decimal
just above an `f32` midpoint can collapse onto that midpoint in `f64`,
and round-half-even then sends
it the wrong way, so fixing the `f64` path alone leaves those cases
unchanged.
The expected result is the float nearest the decimal's exact value: what
parsing the decimal's own
text gives, and what Java's `BigDecimal.doubleValue()` / `floatValue()`
give.
# What changes are included in this PR?
- `single_decimal_to_float_lossy` keeps the existing arithmetic as a
fast path, but only where both
operands are exact: `-22 <= scale <= 22` and `|unscaled| < 2^53`. A
negative scale multiplies by
`10^-scale` rather than dividing by `10^scale`, since only the former is
an exact power of ten.
Everything outside that range goes through a correctly rounded
decimal-string parse.
- New `single_decimal_to_f32_lossy`, used by the `Float32` target,
narrowing in one step. Same shape,
smaller exact range: `10^k` is exact in an `f32` only up to `k = 10`
(`5^10` is the largest power
of five that fits the 24-bit significand), and the integer bound is
2^24.
- `Float16` is unchanged and now consumes the corrected `f64`. It is
still not correctly rounded —
`half::f16::from_str` is `f32::from_str(..).map(f16::from_f32)`, so
there is no correctly rounded
decimal → `f16` conversion to compare against. Noted as a remaining
limitation.
# Are these changes tested?
Yes, six tests in `arrow-cast/src/cast/mod.rs`:
- `Float64` and `Float32` against expected constants, each including
control cases that already
passed before this change.
- `Decimal256`, where the `i256 -> f64` step is least likely to leave
the value intact.
- A test that every value lands on the same float as parsing its own
text — that is the definition of
correctly rounded, and it needs no expected constants. It sweeps both
sides of each fast-path
boundary: the 2^53 / 2^24 significand limits and the scale ±22 / ±10
limits on exact powers of ten.
- A test for values carrying more digits than their declared precision,
which must not be truncated.
- A test for the narrow-precision path, which every other test here
misses by declaring precision 38
or 76.
Also checked out of tree against randomized sweeps of `(unscaled,
scale)` pairs, comparing every
result to an independently built decimal string, for `Decimal128` to
both `f32` and `f64`: 0
mismatches, where the same sweep failed on a large share of the wide
cases before the change.
# Are there any user-facing changes?
Yes. Casts from decimal to `Float32`/`Float64` can now return a
different — correctly rounded — value
for wide decimals. No public API is added or changed.
**Performance** (benchmark bot): `cast decimal128 to float64` ~12%
slower, `cast decimal256 to
float64` ~7% faster. The cost is the per-value exactness test; the gain
is the divisor being computed
once per array instead of per value.
# AI disclosure
This change was developed with AI assistance (Claude).
---------
Co-authored-by: Claude Opus 5 (1M context) <no...@anthropic.com>
---
arrow-cast/src/cast/mod.rs | 355 ++++++++++++++++++++++++++++++++++++++++++++-
1 file changed, 347 insertions(+), 8 deletions(-)
diff --git a/arrow-cast/src/cast/mod.rs b/arrow-cast/src/cast/mod.rs
index f008bbcae4..0d8400a8e1 100644
--- a/arrow-cast/src/cast/mod.rs
+++ b/arrow-cast/src/cast/mod.rs
@@ -82,8 +82,16 @@ pub use decimal::parse_string_to_decimal_native;
pub use decimal::{DecimalCast, rescale_decimal, single_float_to_decimal};
pub use string::cast_single_string_to_boolean_default;
+/// Integers below 2^53 convert to an `f64` exactly.
+const F64_EXACT_INT_LIMIT: f64 = 9_007_199_254_740_992.0;
+/// Integers below 2^24 convert to an `f32` exactly.
+const F32_EXACT_INT_LIMIT: f64 = 16_777_216.0;
+
/// Lossy conversion from decimal to float.
///
+/// Returns the `f64` nearest to the decimal's exact value, rounding once, for any
+/// scale that fits in an `i8` -- as every Arrow decimal's does.
+///
/// Conversion is lossy and follows standard floating point semantics. Values
/// that exceed the representable range become `INFINITY` or `-INFINITY` without
/// returning an error.
@@ -93,7 +101,127 @@ where
D: DecimalType,
F: Fn(D::Native) -> f64,
{
- f(x) / 10_f64.powi(scale)
+ let unscaled = f(x);
+ // Both operands are exact in this range, so this rounds once. A negative scale
+ // multiplies, since `10^-scale` is the exact power of ten there.
+ if (-22..=22).contains(&scale) && unscaled.abs() < F64_EXACT_INT_LIMIT {
+ return if scale >= 0 {
+ unscaled / 10_f64.powi(scale)
+ } else {
+ unscaled * 10_f64.powi(-scale)
+ };
+ }
+ match i8::try_from(scale) {
+ Ok(scale) => decimal_to_f64_rounded_once::<D>(x, scale, unscaled),
+ Err(_) => unscaled / 10_f64.powi(scale),
+ }
+}
+
+/// Rounds once for the values the arithmetic cannot convert exactly, by parsing
+/// the decimal's own text.
+#[cold]
+#[inline(never)]
+fn decimal_to_f64_rounded_once<D: DecimalType>(x: D::Native, scale: i8, unscaled: f64) -> f64 {
+ // `format_decimal` always writes a decimal literal, so the parse cannot fail.
+ // The fallback is there to keep that from being a panic, not because it runs.
+ D::format_decimal(x, u8::MAX, scale)
+ .parse::<f64>()
+ .unwrap_or_else(|_| unscaled / 10_f64.powi(scale.into()))
+}
+
+/// As [`decimal_to_f64_rounded_once`], but narrowing to `f32` in one step.
+/// Rounding to `f64` first and then to `f32` rounds twice: a decimal just above an
+/// `f32` midpoint can collapse onto that midpoint in `f64`, and round-half-even
+/// then sends it the wrong way.
+#[cold]
+#[inline(never)]
+fn decimal_to_f32_rounded_once<D: DecimalType>(x: D::Native, scale: i8, unscaled: f64) -> f32 {
+ // Unreachable, as in [`decimal_to_f64_rounded_once`].
+ D::format_decimal(x, u8::MAX, scale)
+ .parse::<f32>()
+ .unwrap_or_else(|_| unscaled as f32 / 10_f32.powi(scale.into()))
+}
+
+/// Casts a decimal array to `Float64`, rounding each value once.
+fn cast_decimal_to_f64<D, F>(
+ array: &dyn Array,
+ as_float: &F,
+ scale: i8,
+) -> Result<ArrayRef, ArrowError>
+where
+ D: DecimalType + ArrowPrimitiveType,
+ F: Fn(D::Native) -> f64,
+{
+ let array = array.as_primitive::<D>();
+ // No power of ten outside this range is exactly representable.
+ if !(-22..=22).contains(&scale) {
+ let values = array
+ .unary::<_, Float64Type>(|x| decimal_to_f64_rounded_once::<D>(x, scale, as_float(x)));
+ return Ok(Arc::new(values));
+ }
+ let pow = 10_f64.powi(scale.unsigned_abs().into());
+ let values = if scale >= 0 {
+ array.unary::<_, Float64Type>(|x| {
+ let unscaled = as_float(x);
+ if unscaled.abs() < F64_EXACT_INT_LIMIT {
+ unscaled / pow
+ } else {
+ decimal_to_f64_rounded_once::<D>(x, scale, unscaled)
+ }
+ })
+ } else {
+ array.unary::<_, Float64Type>(|x| {
+ let unscaled = as_float(x);
+ if unscaled.abs() < F64_EXACT_INT_LIMIT {
+ unscaled * pow
+ } else {
+ decimal_to_f64_rounded_once::<D>(x, scale, unscaled)
+ }
+ })
+ };
+ Ok(Arc::new(values))
+}
+
+/// Casts a decimal array to `Float32`, rounding each value once. Same shape as
+/// [`cast_decimal_to_f64`], with the smaller bounds an `f32` allows: integers are
+/// exact below 2^24, and `10^k` only up to `k = 10`.
+fn cast_decimal_to_f32<D, F>(
+ array: &dyn Array,
+ as_float: &F,
+ scale: i8,
+) -> Result<ArrayRef, ArrowError>
+where
+ D: DecimalType + ArrowPrimitiveType,
+ F: Fn(D::Native) -> f64,
+{
+ let array = array.as_primitive::<D>();
+ // No power of ten outside this range is exactly representable.
+ if !(-10..=10).contains(&scale) {
+ let values = array
+ .unary::<_, Float32Type>(|x| decimal_to_f32_rounded_once::<D>(x, scale, as_float(x)));
+ return Ok(Arc::new(values));
+ }
+ let pow = 10_f32.powi(scale.unsigned_abs().into());
+ let values = if scale >= 0 {
+ array.unary::<_, Float32Type>(|x| {
+ let unscaled = as_float(x);
+ if unscaled.abs() < F32_EXACT_INT_LIMIT {
+ unscaled as f32 / pow
+ } else {
+ decimal_to_f32_rounded_once::<D>(x, scale, unscaled)
+ }
+ })
+ } else {
+ array.unary::<_, Float32Type>(|x| {
+ let unscaled = as_float(x);
+ if unscaled.abs() < F32_EXACT_INT_LIMIT {
+ unscaled as f32 * pow
+ } else {
+ decimal_to_f32_rounded_once::<D>(x, scale, unscaled)
+ }
+ })
+ };
+ Ok(Arc::new(values))
}
/// CastOptions provides a way to override the default cast behaviors
@@ -2378,13 +2506,8 @@ where
<i32 as From<i8>>::from(*scale),
))
}),
- Float32 => cast_decimal_to_float::<D, Float32Type, _>(array, |x| {
- single_decimal_to_float_lossy::<D, F>(&as_float, x, <i32 as From<i8>>::from(*scale))
- as f32
- }),
- Float64 => cast_decimal_to_float::<D, Float64Type, _>(array, |x| {
- single_decimal_to_float_lossy::<D, F>(&as_float, x, <i32 as From<i8>>::from(*scale))
- }),
+ Float32 => cast_decimal_to_f32::<D, F>(array, &as_float, *scale),
+ Float64 => cast_decimal_to_f64::<D, F>(array, &as_float, *scale),
Utf8View => value_to_string_view(array, cast_options),
Utf8 => value_to_string::<i32>(array, cast_options),
LargeUtf8 => value_to_string::<i64>(array, cast_options),
@@ -15288,4 +15411,220 @@ mod tests {
assert_eq!(expected, actual);
}
+
+ fn decimal128(unscaled: i128, precision: u8, scale: i8) -> Decimal128Array {
+ Decimal128Array::from(vec![unscaled])
+ .with_precision_and_scale(precision, scale)
+ .unwrap()
+ }
+
+ #[test]
+ fn test_cast_decimal_to_float_is_correctly_rounded() {
+ // (unscaled, scale, nearest double). The last four already passed.
+ for (unscaled, scale, expected) in [
+ (12345678901234567890i128, 2i8, 1.2345678901234568e17f64),
+ (10i128.pow(37), 37, 1.0),
+ (123456789012345678901, 20, 1.2345678901234567),
+ (1, 37, 1e-37),
+ (1, 1, 0.1),
+ (15, 1, 1.5),
+ (123456, 3, 123.456),
+ (-123456, 3, -123.456),
+ ] {
+ let out = cast(&decimal128(unscaled, 38, scale), &DataType::Float64).unwrap();
+ assert_eq!(
+ out.as_primitive::<Float64Type>().value(0),
+ expected,
+ "Decimal128({unscaled}, scale={scale}) -> Float64"
+ );
+ }
+
+ // (unscaled, scale, nearest float). The first three differ from
+ // `(f32) (f64) value`, so fixing the f64 path alone leaves them wrong.
+ for (unscaled, scale, expected) in [
+ (13631072500000000514758830i128, 18i8, 13631073.0f32),
+ (72073620000000000000000582908005, 24, 72073624.0),
+ (-3273316900000000000957536840, 20, -32733170.0),
+ (123456, 3, 123.456),
+ (-12345678, 3, -12345.678),
+ (1, 10, 1e-10),
+ ] {
+ let out = cast(&decimal128(unscaled, 38, scale), &DataType::Float32).unwrap();
+ assert_eq!(
+ out.as_primitive::<Float32Type>().value(0),
+ expected,
+ "Decimal128({unscaled}, scale={scale}) -> Float32"
+ );
+ }
+ }
+
+ // Both sides of every boundary the conversion switches on.
+ #[test]
+ fn test_cast_decimal_to_float_matches_parsing_its_own_text() {
+ for (unscaled, precision, scale) in [
+ (9007199254740991i128, 38u8, 6i8), // last integer exact in an f64
+ (9007199254740992, 38, 6), // 2^53, the first that is not
+ (9007199254740993, 38, 6), // one past that
+ (16777215, 38, 3), // last integer exact in an f32
+ (16777216, 38, 3), // 2^24, the first that is not
+ (16777217, 38, 3), // one past that
+ (123456789, 38, 10), // last exact power of ten for an f32
+ (123456789, 38, 11), // one past it
+ (123456789, 38, 22), // last exact power of ten for an f64
+ (123456789, 38, 23), // one past it
+ (12345, 38, 0), // lowest scale the division handles
+ (12345, 38, -1), // one below it: negative scales
+ (12345, 38, -23), // are all rescaled as text
+ // only the upper bound of a scale is validated, so this is legal
+ (i128::MAX, 38, i8::MIN),
+ (999999999999999, 15, 3), // widest value precision 15 allows
+ (-999999999999999, 15, 15),
+ (999999999999999, 16, 3), // one precision past the skipped test
+ (9999999, 7, 3), // widest value precision 7 allows
+ (-9999999, 7, 7),
+ (9999999, 8, 3), // one precision past it, for `f32`
+ (-99999999999999999999999999999999999999, 38, 20),
+ (1, 38, 38),
+ ] {
+ let text = Decimal128Type::format_decimal(unscaled, u8::MAX, scale);
+ let array = decimal128(unscaled, precision, scale);
+ let label = format!("Decimal128({unscaled}, precision={precision}, scale={scale})");
+
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(
+ as_f64.as_primitive::<Float64Type>().value(0),
+ text.parse::<f64>().unwrap(),
+ "{label} -> Float64 disagrees with parsing {text:?}"
+ );
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(
+ as_f32.as_primitive::<Float32Type>().value(0),
+ text.parse::<f32>().unwrap(),
+ "{label} -> Float32 disagrees with parsing {text:?}"
+ );
+ }
+ }
+
+ #[test]
+ fn test_cast_decimal256_to_float_is_correctly_rounded() {
+ for (unscaled, scale) in [
+ ("12345678901234567890", 2i8),
+ ("10000000000000000000000000000000000000", 37),
+ ("13631072500000000514758830", 18),
+ ("123456", 3),
+ ] {
+ let value = i256::from_string(unscaled).unwrap();
+ let array = Decimal256Array::from(vec![value])
+ .with_precision_and_scale(76, scale)
+ .unwrap();
+ let text = Decimal256Type::format_decimal(value, u8::MAX, scale);
+
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(
+ as_f64.as_primitive::<Float64Type>().value(0),
+ text.parse::<f64>().unwrap(),
+ "Decimal256({unscaled}, scale={scale}) -> Float64"
+ );
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(
+ as_f32.as_primitive::<Float32Type>().value(0),
+ text.parse::<f32>().unwrap(),
+ "Decimal256({unscaled}, scale={scale}) -> Float32"
+ );
+ }
+ }
+
+ #[test]
+ fn test_cast_decimal_to_float_saturates_out_of_range() {
+ // 1.7e39: past f32::MAX, still far inside f64.
+ for (unscaled, want_f32) in [(i128::MAX, f32::INFINITY), (i128::MIN, f32::NEG_INFINITY)] {
+ let array = decimal128(unscaled, 38, -1);
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(as_f32.as_primitive::<Float32Type>().value(0), want_f32);
+
+ let text = Decimal128Type::format_decimal(unscaled, u8::MAX, -1);
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(
+ as_f64.as_primitive::<Float64Type>().value(0),
+ text.parse::<f64>().unwrap()
+ );
+ }
+
+ // 1e-76: below the smallest f32 subnormal, representable in an f64.
+ let array = Decimal256Array::from(vec![i256::from(1)])
+ .with_precision_and_scale(76, 76)
+ .unwrap();
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(as_f32.as_primitive::<Float32Type>().value(0), 0.0);
+
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(as_f64.as_primitive::<Float64Type>().value(0), 1e-76);
+ }
+
+ #[test]
+ fn test_single_decimal_to_float_lossy_scale_outside_i8() {
+ // `format_decimal` takes an `i8` scale; narrowing 128 into one wraps it to
+ // -128 and turns 1e-128 into 1e128. The edges of the `i8` range still round
+ // once, and anything past them falls back to the division.
+ let as_float = |x: i128| x as f64;
+ let lossy =
+ |scale: i32| single_decimal_to_float_lossy::<Decimal128Type, _>(&as_float, 1, scale);
+ for scale in [i8::MAX, i8::MIN] {
+ let text = Decimal128Type::format_decimal(1, u8::MAX, scale);
+ assert_eq!(
+ lossy(scale.into()),
+ text.parse::<f64>().unwrap(),
+ "scale={scale}"
+ );
+ }
+ for scale in [128, -129] {
+ assert_eq!(lossy(scale), 1.0 / 10_f64.powi(scale), "scale={scale}");
+ }
+ }
+
+ // Keeping only `MAX_PRECISION` digits would scale the result by ten.
+ #[test]
+ fn test_cast_decimal_to_float_keeps_digits_beyond_declared_precision() {
+ // 39 digits, one more than `Decimal128Type::MAX_PRECISION`.
+ let array = decimal128(i128::MAX, 38, 0);
+
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(
+ as_f64.as_primitive::<Float64Type>().value(0),
+ 1.7014118346046923e38 // 2^127, the double nearest i128::MAX
+ );
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(
+ as_f32.as_primitive::<Float32Type>().value(0),
+ 1.7014118e38 // 2^127, the float nearest i128::MAX
+ );
+
+ // A precision of 15 (7 for `f32`) promises an integer the float holds
+ // exactly, but values are not validated against the precision they declare.
+ for (unscaled, precision, scale) in
+ [(12345678901234567890i128, 15u8, 2i8), (16777217, 7, 1)]
+ {
+ let text = Decimal128Type::format_decimal(unscaled, u8::MAX, scale);
+ let array = decimal128(unscaled, precision, scale);
+
+ let as_f64 = cast(&array, &DataType::Float64).unwrap();
+ assert_eq!(
+ as_f64.as_primitive::<Float64Type>().value(0),
+ text.parse::<f64>().unwrap(),
+ "{text} at precision {precision} -> Float64"
+ );
+
+ let as_f32 = cast(&array, &DataType::Float32).unwrap();
+ assert_eq!(
+ as_f32.as_primitive::<Float32Type>().value(0),
+ text.parse::<f32>().unwrap(),
+ "{text} at precision {precision} -> Float32"
+ );
+ }
+ }
}