You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "QuakeWang (via GitHub)" <gi...@apache.org> on 2026/09/12 07:25:32 UTC
[PR] feat(file_index): generate indexes for append writes [paimon-rust]
QuakeWang opened a new pull request, #819:
URL: https://github.com/apache/paimon-rust/pull/819
### Purpose
Closes #818.
Rust can read FileIndexes, but ordinary append writes leave index metadata empty. This change generates configured indexes during append writes so committed files can benefit from FileIndex pruning.
### Brief change log
- Validate top-level Bitmap/Bloom Filter configuration and build indexes per physical file, including rolling files.
- Store complete indexes embedded or as sidecars using the configured threshold (default 500 B), and clean up failed writes.
### Tests
- Cover commit/reload, actual pruning, Bloom false positives, NULLs, rolling, threshold boundaries, failure cleanup and commit abort.
- Formatting, workspace Clippy and core tests with `fulltext,vortex` passed (2968 passed, 2 ignored).
### API and Format
No public API or index format changes. Reuses existing FileIndex serialization and metadata; generation is limited to ordinary append writes.
### Documentation
Added append FileIndex configuration and scope to the getting-started guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file_index): generate indexes for append writes [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi merged PR #819:
URL: https://github.com/apache/paimon-rust/pull/819
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file_index): generate indexes for append writes [paimon-rust]
Posted by "JingsongLi (via GitHub)" <gi...@apache.org>.
JingsongLi commented on code in PR #819:
URL: https://github.com/apache/paimon-rust/pull/819#discussion_r3996373311
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -148,6 +149,8 @@ pub struct TableWrite {
has_dedicated_vector_fields: bool,
row_kind_generator: Option<RowKindGenerator>,
row_kind_filter: Option<RowKindFilter>,
+ file_index_options: Option<Arc<FileIndexOptions>>,
+ indexed_write_failed: bool,
Review Comment:
Why need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file_index): generate indexes for append writes [paimon-rust]
Posted by "QuakeWang (via GitHub)" <gi...@apache.org>.
QuakeWang commented on code in PR #819:
URL: https://github.com/apache/paimon-rust/pull/819#discussion_r3996791056
##########
crates/paimon/src/table/table_write.rs:
##########
@@ -148,6 +149,8 @@ pub struct TableWrite {
has_dedicated_vector_fields: bool,
row_kind_generator: Option<RowKindGenerator>,
row_kind_filter: Option<RowKindFilter>,
+ file_index_options: Option<Arc<FileIndexOptions>>,
+ indexed_write_failed: bool,
Review Comment:
This was an unnecessary defensive check to prevent writer reuse after a failure. Removed in 2fa63e3; error propagation and file cleanup are preserved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org