You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/01 05:41:34 UTC

[PR] [index] Reuse incremental global index planning [paimon]

JingsongLi opened a new pull request, #8407:
URL: https://github.com/apache/paimon/pull/8407

   ## Summary
   
   This PR makes global index build incremental by default across Java/Flink/Spark/Python and extracts shared planning logic for reusable row-range/shard handling.
   
   ## Changes
   
   - Add reusable core planning utilities to compute indexed/unindexed row ranges and shard indexed splits.
   - Reuse the planner in Flink and Spark generic index topologies, while sorted index builders use the shared incremental scan.
   - Add a Python `globalindex.build_plan` module and route Python index builders through the same planning model.
   - Extend core, Flink, Spark, and Python tests for incremental/unindexed range behavior.
   
   ## Testing
   
   - [x] `python -m pytest paimon-python/pypaimon/tests/global_index_build_test.py`
   - [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -Dtest=GlobalIndexBuilderUtilsTest,SortedGlobalIndexBuilderTest test`
   - [x] `mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -DfailIfNoTests=false -Dtest=SortedIndexTopoBuilderTest,GenericIndexTopoBuilderTest test`
   - [x] `mvn -pl paimon-spark/paimon-spark-common -am -Pfast-build -DfailIfNoTests=false -Dtest=CreateGlobalIndexProcedureTest test`
   
   ## Notes
   
   The default/procedure build path now skips already indexed data and only builds unindexed row ranges. Explicit max-indexed-row-id entry points keep their explicit semantics.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [index] Reuse incremental global index planning [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8407:
URL: https://github.com/apache/paimon/pull/8407#issuecomment-4850710758

   Addressed the Python lint review comments.
   
   Changes:
   - Removed the unused planner helper imports from `create_global_index.py`.
   - Updated planner tests to import those helpers directly from `build_plan.py`.
   
   Verification:
   - `python -m py_compile paimon-python/pypaimon/globalindex/build_plan.py paimon-python/pypaimon/globalindex/create_global_index.py paimon-python/pypaimon/tests/global_index_build_test.py`
   - `PYTHONPATH=$PWD/paimon-python python -m pytest paimon-python/pypaimon/tests/global_index_build_test.py -q`
   - `/tmp/paimon-flake8-venv/bin/python -m flake8 --config=paimon-python/dev/cfg.ini paimon-python/pypaimon/globalindex/create_global_index.py paimon-python/pypaimon/globalindex/build_plan.py paimon-python/pypaimon/tests/global_index_build_test.py`
   - `git diff --check`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [index] Reuse incremental global index planning [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8407:
URL: https://github.com/apache/paimon/pull/8407


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [index] Reuse incremental global index planning [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on PR #8407:
URL: https://github.com/apache/paimon/pull/8407#issuecomment-4850762775

   Fixed the CI failure caused by Spotless formatting in `GlobalIndexBuilderUtilsTest`.
   
   Root cause:
   - CI runs Java builds without `-Pfast-build`, so Spotless checked formatting and failed before Python lint/tests could run.
   
   Verification:
   - `mvn -pl paimon-core -am -DfailIfNoTests=false -Dtest=GlobalIndexBuilderUtilsTest,SortedGlobalIndexBuilderTest test`
   - `/tmp/paimon-flake8-venv/bin/python -m flake8 --config=paimon-python/dev/cfg.ini paimon-python/pypaimon/globalindex/create_global_index.py paimon-python/pypaimon/globalindex/build_plan.py paimon-python/pypaimon/tests/global_index_build_test.py`
   - `PYTHONPATH=$PWD/paimon-python python -m pytest paimon-python/pypaimon/tests/global_index_build_test.py -q`
   - `git diff --check`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org