You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "JingsongLi (via GitHub)" <gi...@apache.org> on 2026/07/13 08:57:56 UTC

[PR] Support PK-vector indexes for postpone buckets [paimon]

JingsongLi opened a new pull request, #8594:
URL: https://github.com/apache/paimon/pull/8594

   ## What changed
   
   - Allow primary-key vector indexes in postpone-bucket mode (`bucket = -2`) while continuing to reject dynamic buckets.
   - Capture one snapshot for postpone planning and writer restore.
   - Make Flink and Spark background compaction process both bucket `-2` rows and real-bucket Level-0 files.
   - Build and publish ANN index changes together with compacted data in one strict atomic commit.
   - Exclude postponed data from vector search until compaction and document the visibility behavior.
   
   ## Why
   
   Foreground fixed-bucket writes for postpone tables remain `write-only = true`. In deletion-vector mode, both real-bucket Level-0 files and bucket `-2` files must be compacted before they become visible. The previous postpone compactor only consumed bucket `-2`, so marker-only Level-0 data could not be compacted and indexed by the background job.
   
   ## Validation
   
   - 38 focused Core tests, including strict compact snapshot behavior.
   - Flink pinned-snapshot source test and mixed postpone/Level-0 PK-vector integration test.
   - Spark `PrimaryKeyVectorSearchTest`: 9 tests.
   - Spark `PostponeBucketTableTest`: 7 tests.
   - Spotless and Core/Flink non-fast validation.
   - `git diff --check`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] [core][flink][spark] Support PK-vector indexes for postpone buckets [paimon]

Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #8594:
URL: https://github.com/apache/paimon/pull/8594


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org