You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/21 15:22:04 UTC
[PR] Align native read schema evolution, ROW decoding, and merge semantics [paimon-rust]
JingsongLi opened a new pull request, #907:
URL: https://github.com/apache/paimon-rust/pull/907
## Summary
- Match Java/Python schema evolution for constructed values cast to STRING and truncating DECIMAL scale reductions.
- Decode ROW files using their physical write columns, including KV key/system fields and partial-column Data Evolution files.
- Honor descending user-sequence sort order while keeping null sequences first.
- Protect partition primary-key columns from default aggregation.
These fixes unblock enabling Rust native read in the Python Rust Plan CI job (apache/paimon#10067). The Python-side Variant overflow key-order correction is in that PR.
## Validation
-
running 3010 tests
....................................................................................... 87/3010
....................................................................................... 174/3010
....................................................................................... 261/3010
..............i........................................................................ 348/3010
....................................................................................... 435/3010
..................i.................................................................... 522/3010
....................................................................................... 609/3010
....................................................................................... 696/3010
....................................................................................... 783/3010
....................................................................................... 870/3010
....................................................................................... 957/3010
....................................................................................... 1044/3010
....................................................................................... 1131/3010
....................................................................................... 1218/3010
....................................................................................... 1305/3010
....................................................................................... 1392/3010
....................................................................................... 1479/3010
....................................................................................... 1566/3010
....................................................................................... 1653/3010
....................................................................................... 1740/3010
......................................................i................................ 1827/3010
....................................................................................... 1914/3010
........................i.............................................................. 2001/3010
....................................................................................... 2088/3010
....................................................................................... 2175/3010
....................................................................................... 2262/3010
...................................................................i................... 2349/3010
....................................................................................... 2436/3010
....................................................................................... 2523/3010
....................................................................................... 2610/3010
....................................................................................... 2697/3010
....................................................................................... 2784/3010
....................................................................................... 2871/3010
......................................................................................i 2958/3010
....................................................
test result: ok. 3004 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 10.86s
running 20 tests
....................
test result: ok. 20 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.07s
running 6 tests
......
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.08s
running 4 tests
....
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 38 tests
......................................
test result: ok. 38 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.14s
running 0 tests
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.10s
running 8 tests
........
test result: ok. 8 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.26s
running 10 tests
..........
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.20s
running 7 tests
.......
test result: ok. 7 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
running 1 test
.
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.01s
running 45 tests
.............................................
test result: ok. 45 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.11s
running 79 tests
...............................................................................
test result: ok. 79 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.54s
running 10 tests
..........
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
running 6 tests
......
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.02s
running 15 tests
...............
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
running 6 tests
iiiii.
test result: ok. 1 passed; 0 failed; 5 ignored; 0 measured; 0 filtered out; finished in 0.70s: all unit and integration tests passed (3004 unit tests passed, 6 ignored).
- : passed.
- : passed.
- Built the Python wheel and ran targeted native plan + native read regressions for schema evolution, aggregation, descending sequence, ROW, and Variant: 15 passed, 10 subtests passed; reran remaining ROW cases: 3 passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align native read schema evolution, ROW decoding, and merge semantics [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #907:
URL: https://github.com/apache/paimon-rust/pull/907#discussion_r4067960775
##########
crates/paimon/src/arrow/nested_evolution.rs:
##########
@@ -64,6 +66,62 @@ pub(crate) fn evolve_column(
return Ok(source.clone());
}
+ if let (
+ ArrowDataType::Decimal128(_, source_scale),
+ ArrowDataType::Decimal128(target_precision, target_scale),
+ ) = (source.data_type(), &target_arrow)
+ {
+ if source_scale > target_scale {
+ // Arrow rounds Decimal128 scale reductions. Paimon schema casts,
+ // like Python's unsafe Arrow cast, truncate toward zero instead.
+ let shift = u32::try_from(i16::from(*source_scale) - i16::from(*target_scale))
+ .map_err(|_| crate::Error::DataInvalid {
+ message: "invalid decimal scale reduction".to_string(),
+ source: None,
+ })?;
+ let divisor = 10_i128
+ .checked_pow(shift)
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: "decimal scale reduction exceeds Decimal128 range".to_string(),
+ source: None,
+ })?;
+ let decimals = source
+ .as_any()
+ .downcast_ref::<Decimal128Array>()
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: format!("expected Decimal128 array, got {:?}", source.data_type()),
+ source: None,
+ })?;
+ let reduced = Decimal128Array::from(
+ (0..decimals.len())
+ .map(|index| {
+ decimals
+ .is_valid(index)
+ .then(|| decimals.value(index) / divisor)
+ })
+ .collect::<Vec<_>>(),
+ )
+ .with_precision_and_scale(*target_precision, *target_scale)
Review Comment:
Fixed in aeba0cdf. Arrow safe cast now nulls values that exceed the target precision after rounding. The regression covers precision+scale reduction, rounding-induced overflow, negative overflow, and validate_decimal_precision(3); it failed before the fix and passes now. Full Rust lib suite: 3009 passed, 6 ignored.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align native read schema evolution, ROW decoding, and merge semantics [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #907:
URL: https://github.com/apache/paimon-rust/pull/907#discussion_r4067959566
##########
crates/paimon/src/arrow/nested_evolution.rs:
##########
@@ -64,6 +66,62 @@ pub(crate) fn evolve_column(
return Ok(source.clone());
}
+ if let (
+ ArrowDataType::Decimal128(_, source_scale),
+ ArrowDataType::Decimal128(target_precision, target_scale),
+ ) = (source.data_type(), &target_arrow)
+ {
+ if source_scale > target_scale {
+ // Arrow rounds Decimal128 scale reductions. Paimon schema casts,
+ // like Python's unsafe Arrow cast, truncate toward zero instead.
+ let shift = u32::try_from(i16::from(*source_scale) - i16::from(*target_scale))
+ .map_err(|_| crate::Error::DataInvalid {
+ message: "invalid decimal scale reduction".to_string(),
+ source: None,
+ })?;
+ let divisor = 10_i128
+ .checked_pow(shift)
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: "decimal scale reduction exceeds Decimal128 range".to_string(),
+ source: None,
+ })?;
+ let decimals = source
+ .as_any()
+ .downcast_ref::<Decimal128Array>()
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: format!("expected Decimal128 array, got {:?}", source.data_type()),
+ source: None,
+ })?;
+ let reduced = Decimal128Array::from(
+ (0..decimals.len())
+ .map(|index| {
+ decimals
+ .is_valid(index)
+ .then(|| decimals.value(index) / divisor)
Review Comment:
Fixed in aeba0cdf. Removed the truncating Decimal128 special case so scale reductions use Arrow safe cast, matching Java HALF_UP. Added positive/negative and exact-half regression cases; both failed before the fix and pass now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align native read schema evolution, ROW decoding, and merge semantics [paimon-rust]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #907:
URL: https://github.com/apache/paimon-rust/pull/907#discussion_r4067815358
##########
crates/paimon/src/arrow/nested_evolution.rs:
##########
@@ -64,6 +66,62 @@ pub(crate) fn evolve_column(
return Ok(source.clone());
}
+ if let (
+ ArrowDataType::Decimal128(_, source_scale),
+ ArrowDataType::Decimal128(target_precision, target_scale),
+ ) = (source.data_type(), &target_arrow)
+ {
+ if source_scale > target_scale {
+ // Arrow rounds Decimal128 scale reductions. Paimon schema casts,
+ // like Python's unsafe Arrow cast, truncate toward zero instead.
+ let shift = u32::try_from(i16::from(*source_scale) - i16::from(*target_scale))
+ .map_err(|_| crate::Error::DataInvalid {
+ message: "invalid decimal scale reduction".to_string(),
+ source: None,
+ })?;
+ let divisor = 10_i128
+ .checked_pow(shift)
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: "decimal scale reduction exceeds Decimal128 range".to_string(),
+ source: None,
+ })?;
+ let decimals = source
+ .as_any()
+ .downcast_ref::<Decimal128Array>()
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: format!("expected Decimal128 array, got {:?}", source.data_type()),
+ source: None,
+ })?;
+ let reduced = Decimal128Array::from(
+ (0..decimals.len())
+ .map(|index| {
+ decimals
+ .is_valid(index)
+ .then(|| decimals.value(index) / divisor)
+ })
+ .collect::<Vec<_>>(),
+ )
+ .with_precision_and_scale(*target_precision, *target_scale)
Review Comment:
[P2] Enforce the target precision on the reduced decimal values
`Decimal128Array::with_precision_and_scale` validates the type parameters, not whether the array's values fit the declared precision. Therefore this `map_err` does not catch value overflow, and the new branch can return an invalid decimal array.
Independent of the rounding policy, a valid `DECIMAL(6,3)` value `999.999` converted to `DECIMAL(3,2)` currently returns `999.99` (unscaled `99999`) in an array declared to have precision 3; the maximum representable value is `9.99`. Calling `validate_decimal_precision(3)` on the result fails. The previous Arrow-cast path returns NULL for this overflow, as does Java's decimal cast.
Please check the reduced values against the target precision and preserve overflow-to-NULL behavior, including when precision and scale both decrease. A regression test should validate the resulting array, not just its declared data type.
##########
crates/paimon/src/arrow/nested_evolution.rs:
##########
@@ -64,6 +66,62 @@ pub(crate) fn evolve_column(
return Ok(source.clone());
}
+ if let (
+ ArrowDataType::Decimal128(_, source_scale),
+ ArrowDataType::Decimal128(target_precision, target_scale),
+ ) = (source.data_type(), &target_arrow)
+ {
+ if source_scale > target_scale {
+ // Arrow rounds Decimal128 scale reductions. Paimon schema casts,
+ // like Python's unsafe Arrow cast, truncate toward zero instead.
+ let shift = u32::try_from(i16::from(*source_scale) - i16::from(*target_scale))
+ .map_err(|_| crate::Error::DataInvalid {
+ message: "invalid decimal scale reduction".to_string(),
+ source: None,
+ })?;
+ let divisor = 10_i128
+ .checked_pow(shift)
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: "decimal scale reduction exceeds Decimal128 range".to_string(),
+ source: None,
+ })?;
+ let decimals = source
+ .as_any()
+ .downcast_ref::<Decimal128Array>()
+ .ok_or_else(|| crate::Error::DataInvalid {
+ message: format!("expected Decimal128 array, got {:?}", source.data_type()),
+ source: None,
+ })?;
+ let reduced = Decimal128Array::from(
+ (0..decimals.len())
+ .map(|index| {
+ decimals
+ .is_valid(index)
+ .then(|| decimals.value(index) / divisor)
Review Comment:
[P2] Preserve Java's HALF_UP semantics for decimal scale reductions
The integer division changes a previously supported core read from rounding to truncation. Java's `DecimalToDecimalCastRule` calls `DecimalUtils.castToDecimal`, which calls `Decimal.fromBigDecimal`; that method uses `setScale(scale, RoundingMode.HALF_UP)`, not truncation.
I reproduced `DECIMAL(6,3) -> DECIMAL(6,2)` with `4.567` and `-4.567`: the existing Arrow-cast path returns `4.57` and `-4.57`, while this branch returns `4.56` and `-4.56`. This silently changes the values read from existing files and makes Rust disagree with Java.
Python's current unsafe Arrow cast does truncate, but matching that behavior in the shared Rust read path does not preserve Java semantics. Could we keep HALF_UP here and reconcile the Python expectation separately, with positive/negative rounding cases in the regression coverage?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Align native read schema evolution, ROW decoding, and merge semantics [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #907:
URL: https://github.com/apache/paimon-rust/pull/907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org