You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/17 05:57:29 UTC
[PR] [core] Normalize Arrow layouts for upserts and data evolution updates [paimon-rust]
JingsongLi opened a new pull request, #1144:
URL: https://github.com/apache/paimon-rust/pull/1144
### Purpose
Keep PyPaimon's existing large-string upserts on Native execution, including strings used as match keys, partition fields and nested values. PyPaimon currently falls back because core write normalization rejects `LargeUtf8` for the canonical `Utf8` STRING representation.
Testing this path also exposes a DE update failure for ROW/ARRAY/MAP types with required children: runtime read schemas relax nested nullability, while the partial-file writer uses the declared table schema. Interleaving against the runtime schema leaves incompatible child types in the output.
### Brief change log
- Normalize `LargeUtf8` to `Utf8` in the existing core write normalizer. This is an Arrow layout conversion for Java/Paimon's STRING values, with checked offset bounds; it does not introduce implicit numeric or binary casts.
- Reuse recursive normalization for nested strings and for the key encoding performed by core upserts.
- Normalize merged DE update values back to their declared write layout before constructing partial-column batches. Preserve NULL parents while validating visible required children.
- Add a core upsert round trip covering multiple batches, duplicate and NULL keys, updates to multiple target rows, absent columns on append, and stable row IDs.
- Cover sliced strings, Unicode, empty values, nested layouts and visible versus hidden NULL children.
### Tests
- `cargo +1.98.0 test --quiet --locked -p paimon --lib`: 4151 passed, 6 ignored.
- `cargo +1.98.0 clippy --quiet --locked -p paimon --lib --tests -- -D warnings`.
- `cargo +1.98.0 fmt --all --check`.
- Built Python bindings and verified the corresponding PyPaimon execution paths: 30 new Native/Python parity cases passed; the related DE/update/BLOB regression selection passed 568 tests with 2 skips.
- Independent review of input eligibility, nested constraints and failure paths.
### API and Format
No new public interfaces, options, dependencies or storage formats. Java STRING values and the existing declared write constraints remain authoritative.
### Documentation
No documentation changes are required.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Normalize Arrow layouts for upserts and data evolution updates [paimon-rust]
Posted by "JeonDaehong (via GitHub)" <gi...@apache.org>.
JeonDaehong commented on code in PR #1144:
URL: https://github.com/apache/paimon-rust/pull/1144#discussion_r4301841244
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
+ // not change the logical values, including in keys and nested fields.
+ // This specific cast checks offset overflow without enabling value casts.
+ (DataType::LargeUtf8, DataType::Utf8) => {
Review Comment:
`arrow_cast` checks the slice's offsets as absolute values, so a small slice taken past the 2 GiB mark of a LargeUtf8 array fails even though the slice itself is tiny. I built a 2-row array (2^31 + 16 bytes, then `"abc"`) and normalized `slice(1, 1)`:
```
LargeString array too large to cast to String array
```
The same `"abc"` as a fresh array goes through. pyarrow `to_batches()` hands over zero-copy slices, so a large_string chunk over 2 GiB split into small batches would hit this. Rebasing the offsets (subtract the first one, slice the values) before the cast should fix it, like `visible_children` already does for lists.
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
Review Comment:
Question: the same reasoning applies to `LargeBinary` -> `Binary` for BYTES columns, and that still fails on this head with `Arrow write type LargeBinary is incompatible with table type Binary`. Is that planned for a follow-up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Normalize Arrow layouts for upserts and data evolution updates [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #1144:
URL: https://github.com/apache/paimon-rust/pull/1144
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core] Normalize Arrow layouts for upserts and data evolution updates [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #1144:
URL: https://github.com/apache/paimon-rust/pull/1144#discussion_r4301943673
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
+ // not change the logical values, including in keys and nested fields.
+ // This specific cast checks offset overflow without enabling value casts.
+ (DataType::LargeUtf8, DataType::Utf8) => {
Review Comment:
Fixed in 1a0d6217. The conversion now subtracts the first visible offset and slices the shared value buffer before narrowing offsets to i32. This accepts a small slice starting beyond 2 GiB without copying its payload, while still rejecting a visible byte range that exceeds the 32-bit limit.
Added a sparse-mmap regression covering the high starting offset, Unicode, NULLs, empty slices, buffer sharing and actual range overflow. The 23 normalization tests, 2 core upsert tests, Clippy and rustfmt checks pass.
##########
crates/paimon/src/table/write_batch_normalize.rs:
##########
@@ -230,6 +230,12 @@ fn normalize_visible_array(
}
match (array.data_type(), expected) {
+ // Java has one STRING value representation. Arrow offset widths do
Review Comment:
Keeping LargeBinary/Binary conversions out of this PR. PyPaimon maps binary to BYTES and large_binary to BLOB; accepting them interchangeably would require coordinated handling across schema validation and the write/update paths, expanding the scope considerably.
The existing rules remain in place, with regression assertions that neither direction is implicitly accepted by write normalization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org