You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/09/21 13:43:15 UTC
[PR] Enable native read in Python Rust Plan CI [paimon]
JingsongLi opened a new pull request, #10067:
URL: https://github.com/apache/paimon/pull/10067
## Summary
- Force native reading alongside native planning in the Rust Plan pull-request CI job.
- Track successful native-read setup and fail a green test session if the lane never exercised native read.
- Keep the mixed Java/Python interop phase outside the forced-native fixture scope.
## Validation
- Local Data Evolution subset excluding Vortex: 22 passed, 3 deselected; 37 native plans and 34 native reads.
- Full Rust Plan suite will run in pull-request CI with its pinned Linux dependencies.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Enable native read in Python Rust Plan CI [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10067:
URL: https://github.com/apache/paimon/pull/10067#discussion_r4068235442
##########
paimon-python/pypaimon/data/variant_shredding.py:
##########
@@ -328,7 +332,14 @@ def _build_object_value(fields: List[Tuple[int, bytes]]) -> bytes:
buf.append(0) # offset[0] = 0 (sentinel)
return bytes(buf)
- fields = sorted(fields, key=lambda f: f[0])
+ if key_dict is None:
+ fields = sorted(fields, key=lambda f: f[0])
+ else:
+ id_to_name = {key_id: name for name, key_id in key_dict.items()}
+ fields = sorted(
+ fields,
+ key=lambda f: id_to_name[f[0]].encode('utf-16-be'),
Review Comment:
Verified fixed in acece3ccd57f5ea84a05d53145879a4a9acb7de8. The original regression now passes, the added test covers both overflow writing and reconstruction, and an independent end-to-end native-read test preserves both values and UTF-8 key order. Resolving this finding.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Enable native read in Python Rust Plan CI [paimon]
Posted by "leaves12138 (via GitHub)" <gi...@apache.org>.
leaves12138 commented on code in PR #10067:
URL: https://github.com/apache/paimon/pull/10067#discussion_r4068086717
##########
paimon-python/pypaimon/data/variant_shredding.py:
##########
@@ -328,7 +332,14 @@ def _build_object_value(fields: List[Tuple[int, bytes]]) -> bytes:
buf.append(0) # offset[0] = 0 (sentinel)
return bytes(buf)
- fields = sorted(fields, key=lambda f: f[0])
+ if key_dict is None:
+ fields = sorted(fields, key=lambda f: f[0])
+ else:
+ id_to_name = {key_id: name for name, key_id in key_dict.items()}
+ fields = sorted(
+ fields,
+ key=lambda f: id_to_name[f[0]].encode('utf-16-be'),
Review Comment:
[P2] Preserve UTF-8 byte ordering when rebuilding Variant objects
The Parquet Variant specification requires object field IDs/offsets to be ordered by field names using unsigned UTF-8 bytes, not UTF-16 code units. Current Java `GenericVariantBuilder.FieldEntry.compareTo` and Python `GenericVariant._finish_writing_object` already use UTF-8 ordering.
A concrete regression is `GenericVariant.from_python({'keep': 0, '\uff21': 1, '\U0001f600': 2})` with only `keep` shredded. The metadata IDs for the remaining keys are already in the required order, so the base writes `[U+FF21, U+1F600]`; this change writes `[U+1F600, U+FF21]`. I reproduced the same test passing against the base and failing against this head. Since `decompose_variant` and `_decompose_field_bytes` call this helper, this affects persisted overflow bytes, not just an in-memory presentation order. A conforming reader using UTF-8 binary search can consequently miss existing keys.
Please sort by `.encode('utf-8')` and add a regression mixing a supplementary-plane key with a BMP key above U+DFFF, covering both overflow writing and reconstruction. If this exposes a UTF-16-only assumption in the Rust reader, that compatibility issue should be fixed there rather than emitting non-conforming bytes here.
Spec: https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#object-field-id-order-and-uniqueness
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Enable native read in Python Rust Plan CI [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10067:
URL: https://github.com/apache/paimon/pull/10067
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org