You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/08/08 07:40:19 UTC

[PR] [python][ray] Add row-id range processing API [paimon]

JingsongLi opened a new pull request, #9112:
URL: https://github.com/apache/paimon/pull/9112

   ## What changed
   
   - add `process_row_id_ranges` to plan logical file-group ranges and invoke a user processor sequentially in target-sized batches
   - keep overlapping base, data-evolution, BLOB, and VECTOR files in one indivisible range group
   - fix row-id updates of columns added after existing data by retaining `_ROW_ID` as a row-count anchor
   - document a resumable Ray embedding backfill from BLOB payloads with one commit per approximately one million rows
   
   ## Why
   
   Long-running transforms such as embedding generation need bounded commits and application-owned distributed processing. The new API provides stable row-id/file boundaries while leaving Ray reads, model execution, commits, and retry policy to the processor.
   
   The row-count anchor fix is required for first-time backfills of a newly added nullable VECTOR column; without it, reading only a column absent from old files yielded a zero-row table.
   
   ## Validation
   
   - `python -m pytest -q pypaimon/tests/ray_process_row_id_ranges_test.py pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest::test_backfill_vector_column_added_after_existing_rows` — 6 passed, 7 subtests passed
   - `python -m pytest -q pypaimon/tests/table_update_test.py` — 96 passed, 14 subtests passed
   - `python -m pytest -q pypaimon/tests/vector_table_test.py::VectorTableWriteReadTest -k 'not vortex'` — 5 passed, 1 skipped, 1 deselected
   - changed Python files compile with `py_compile`
   - the embedding documentation example parses with `ast.parse`
   
   The full vector test class was not completed locally because the installed Vortex native extension segfaults under Python 3.13 in the existing Vortex-specific test.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][ray] Add row-id range processing API [paimon]

Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #9112:
URL: https://github.com/apache/paimon/pull/9112#issuecomment-5225267289

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][ray] Add row-id range processing API [paimon]

Posted by "XiaoHongbo-Hope (via GitHub)" <gi...@apache.org>.
XiaoHongbo-Hope commented on PR #9112:
URL: https://github.com/apache/paimon/pull/9112#issuecomment-5225277724

   +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [python][ray] Add row-id range processing API [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #9112:
URL: https://github.com/apache/paimon/pull/9112


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org