You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/01 10:06:02 UTC

[PR] [python] Add data evolution merge_into [paimon]

JingsongLi opened a new pull request, #8416:
URL: https://github.com/apache/paimon/pull/8416

   ## Summary
   
   Adds single-process PyPaimon `TableUpdate.merge_into` support for data-evolution tables. The implementation follows the existing Ray merge clause helpers, supports batch and stream commit lifecycles, and documents the new API.
   
   ## Changes
   
   - Expose `merge_into` on `BatchTableUpdate` and `StreamTableUpdate`.
   - Add a local PyArrow-based merge implementation for PyPaimon table, Arrow table, and pandas DataFrame sources.
   - Support matched updates, not-matched inserts, renamed `ON` keys, conditions, and self-merge by `_ROW_ID`.
   - Treat target self-assignments like `t.col = t.col` as no-op updates in the shared merge normalization path.
   - Add merge tests covering inserts, updates, row-id stability, renamed keys, partition-column rejection, duplicate matches, self-merge, and stream commit identifiers.
   - Update the PyPaimon data-evolution docs and keep stream-mode guidance in a single section.
   
   ## Testing
   
   - [x] `python -m unittest pypaimon.tests.table_merge_into_test`
   - [x] `python -m unittest pypaimon.tests.table_update_test.TableUpdateBatchTest.test_update_existing_column`
   - [x] `python -m unittest pypaimon.tests.table_upsert_by_key_test.TableUpsertByKeyBatchTest.test_mixed_update_and_append`
   - [x] `python -m py_compile pypaimon/table/data_evolution_merge_into.py pypaimon/ray/data_evolution_merge_into.py pypaimon/write/table_update.py pypaimon/tests/table_merge_into_test.py`
   - [x] `git diff --check`
   
   ## Notes
   
   - One condition-clause test is skipped locally when `datafusion` is not installed.
   - Ray merge tests were not run locally because the current Python environment does not have `ray` installed; the shared helper behavior was covered by the PyPaimon merge tests and a direct normalize smoke check.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python] Support TableUpdate.merge_into in PyPaimon [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8416:
URL: https://github.com/apache/paimon/pull/8416


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org