[PR] [python] Support delete in data evolution APIs [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi opened a new pull request, #8426:
URL: https://github.com/apache/paimon/pull/8426

   ## Summary
   
   This PR adds DELETE support to PyPaimon data-evolution APIs and wires delete branches into table, multimodal, and Ray MERGE INTO flows. It also changes partition-only DELETE to emit file-level `deleted_files` commit messages, including matching index deletes, so the commit path uses overwrite-style conflict detection instead of a partition-filter shortcut.
   
   ## Changes
   
   - Add `delete_by_predicate` and `delete_by_row_id` support for data-evolution tables using deletion vectors.
   - Support `WhenMatched.delete()` alongside `WhenMatched.update()` in table MERGE INTO, multimodal MERGE, and Ray MERGE INTO.
   - Replace the internal partition-delete commit-message shortcut with `deleted_files` manifest deletes and matching `index_deletes`.
   - Trigger overwrite-style conflict detection for table DELETE entries and deletion-vector index changes, including stale partition detection for partition-only DELETE.
   - Split Ray merge update/delete branches so row-id actions are validated and committed separately.
   - Update PyPaimon docs for data evolution, multimodal APIs, and Ray data APIs.
   
   ## Testing
   
   - [x] `python -m pytest paimon-python/pypaimon/tests/table_update_test.py paimon-python/pypaimon/tests/data_evolution_deletion_vector_test.py paimon-python/pypaimon/tests/table_merge_into_test.py paimon-python/pypaimon/tests/multimodal_table_test.py paimon-python/pypaimon/tests/partition_predicate_test.py paimon-python/pypaimon/tests/overwrite_changes_cache_test.py -q`
   - [x] `python -m pytest paimon-python/pypaimon/tests/file_store_commit_test.py paimon-python/pypaimon/tests/table_commit_test.py paimon-python/pypaimon/tests/write/changelog_producer_test.py -q`
   - [x] `PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/write/conflict_detection_test.py -q`
   - [x] `python -m py_compile paimon-python/pypaimon/write/commit_message.py paimon-python/pypaimon/write/file_store_commit.py paimon-python/pypaimon/write/table_update.py paimon-python/pypaimon/write/commit/conflict_detection.py paimon-python/pypaimon/write/commit/overwrite_changes_provider.py paimon-python/pypaimon/ray/data_evolution_merge_into.py paimon-python/pypaimon/ray/data_evolution_merge_join.py paimon-python/pypaimon/ray/data_evolution_merge_transform.py`
   - [x] `git diff --check`
   
   ## Notes
   
   Ray e2e tests were not run locally because the environment does not have the `ray` Python package installed (`ModuleNotFoundError: No module named 'ray'`).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [python] Support delete in data evolution APIs [paimon]

["JingsongLi (via GitHub)" <gi...@apache.org>]

JingsongLi commented on PR #8426:
URL: https://github.com/apache/paimon/pull/8426#issuecomment-4863295092

   Addressed the DV delete concurrency review comment.\n\nChanges:\n- Pass index changes into commit conflict detection.\n- Detect stale deletion-vector index commits by requiring replaced DV index files to still exist in the latest index manifest and rejecting adds that would duplicate a newer DV entry for the same data file.\n- Added a regression test for two row-level deletes prepared from the same snapshot; the stale commit now fails instead of resurrecting the earlier delete.\n\nLocal verification:\n- `python -m pytest paimon-python/pypaimon/tests/table_update_test.py -q`\n- `PYTHONPATH=paimon-python python -m pytest paimon-python/pypaimon/tests/write/conflict_detection_test.py -q`\n- `PYTHONPATH=/Users/lijingsong/IdeaProjects/paimon3/paimon-python python -m pytest paimon-python/pypaimon/tests/table_update_test.py paimon-python/pypaimon/tests/table_merge_into_test.py paimon-python/pypaimon/tests/multimodal_table_test.py paimon-python/pypaimon/tests/overwrite_changes_cache_test.p
 y paimon-python/pypaimon/tests/write/conflict_detection_test.py -q`\n- targeted flake8 on changed Python files\n- `git diff --check`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org