You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/10/07 10:25:20 UTC
[PR] [python] Use native upsert for rows with different supplied fields [paimon]
JingsongLi opened a new pull request, #10414:
URL: https://github.com/apache/paimon/pull/10414
### Purpose
Use Native execution for PyPaimon's existing `upsert_by_key` inputs whose named `GenericRow` values have different field sets.
Depends on https://github.com/apache/paimon-rust/pull/1061. Keep this PR as a draft until the Rust change is merged; Native CI continues to build Rust `main`.
The adapter currently falls back to Python when rows have different supplied fields. Encode consecutive rows with the same field set as an Arrow RecordBatch and submit the ordered batches to Rust. This preserves missing fields, explicit NULL values and last-write-wins order without adding matching or update logic to the adapter.
Rust core owns key matching, deduplication and column validation. Matched rows use the update projection; appended rows retain their actual write type, following Java's underlying append/Data Evolution write semantics. The public Python API is unchanged.
### Tests
Added 20 cases exercising both Python and Native paths in batch and stream mode:
- Different field sets and source field orders.
- Last-write-wins across field sets, including superseded rows missing update columns.
- Fan-out to duplicate target rows and ignored extra matched fields.
- Explicit NULL versus absent fields, with append write-column metadata assertions.
- Different append field sets in different partitions; partition keys implicitly distinguish matches.
- Missing winning update columns and inconsistent surviving append shapes; errors leave snapshots and visible data unchanged.
Native cases explicitly reject Python matching and per-partition upsert fallback.
Local regression verification using the companion Rust binding:
```text
table_upsert_by_key_test.py
native_upsert_row_shapes_test.py
table_update_test.py
table_update_by_row_id_chunked_test.py
native_upsert_capabilities_test.py
native_nested_update_test.py
native_blob_update_test.py
native_update_test.py
```
With Native plan/read/write enabled: **991 passed, 2 skipped, 36 subtests passed**. Native execution counters: 1112 plans, 1345 reads, 760 writes.
The complete Rust Python binding suite passed 321 tests. Flake8 for both changed Python files and diff checks passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [python] Use native upsert for rows with different supplied fields [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #10414:
URL: https://github.com/apache/paimon/pull/10414
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org