You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/14 05:49:24 UTC
[PR] [core][connectors] Complete primary-key sorted-index support [paimon]
JingsongLi opened a new pull request, #8615:
URL: https://github.com/apache/paimon/pull/8615
## Summary
Complete source-backed primary-key BTree and Bitmap support by enabling postpone-bucket batch pruning, adding Flink and Spark end-to-end coverage, and documenting configuration and fallback behavior.
## Changes
- Allow `PrimaryKeyBatchScan` to apply scalar indexes to real-bucket splits of postpone-bucket tables.
- Add a regression test proving postpone-bucket scans return `IndexedSplit` when a snapshot-scoped sorted index is available.
- Exercise mixed BTree and Bitmap maintenance and ordinary scan pruning through Flink and Spark SQL, including updates, deletes, `AND`/`OR` predicates, partial coverage, and unindexed fallback.
- Document column-list configuration, field options, fixed/postpone bucket requirements, asynchronous coverage, residual filtering, schema evolution, and current limitations.
- Cross-link scalar indexes from the vector-index documentation.
## Testing
- [x] `mvn -pl paimon-codegen-loader -am -Pfast-build -DskipTests package`
- [x] `mvn -pl paimon-core -am -Pfast-build -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexBatchScanTest test`
- [x] `mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink1 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexITCase test`
- [x] `JAVA_HOME=/Users/lijingsong/Library/Java/JavaVirtualMachines/corretto-11.0.25/Contents/Home mvn -pl paimon-flink/paimon-flink-common -am -Pfast-build -Pflink2 -DfailIfNoTests=false -DwildcardSuites=none -Dtest=PrimaryKeySortedIndexITCase test`
- [x] `mvn -pl paimon-spark/paimon-spark-ut -am -Pfast-build -Pspark3 -DfailIfNoTests=false -DwildcardSuites=org.apache.paimon.spark.sql.PrimaryKeySortedIndexTest -Dtest=none test`
- [x] `mvn package -Pgenerate-docs -pl paimon-docs -nsu -DskipTests -am`
- [x] `mvn -pl paimon-core -am -DskipTests validate`
- [x] `mvn -pl paimon-flink/paimon-flink-common -Pflink1 -DskipTests validate`
- [x] `git diff --check`
## Notes
The targeted connector tests require `paimon-codegen-loader` resources produced during `prepare-package`, so the loader package step was run before the test commands.
This follows the batch-scan integration merged in #8603.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [core][connectors][docs] Complete primary-key sorted-index support [paimon]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8615:
URL: https://github.com/apache/paimon/pull/8615
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org