You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@paimon.apache.org by "zjw1111 (via GitHub)" <gi...@apache.org> on 2026/08/21 07:44:34 UTC
[PR] feat(file-index): support BSI and Bloom filter writers [paimon-cpp]
zjw1111 opened a new pull request, #234:
URL: https://github.com/apache/paimon-cpp/pull/234
### Purpose
Linked issue: N/A
Add file-index writer support for the BSI and Bloom filter plugins so they can be produced through `DataFileIndexWriter` and read back by the existing readers.
- Serialize BSI positive/negative roaring bitmaps with null-row handling.
- Serialize Java-compatible Bloom filter indexes and validate writer options.
- Simplify existing bitmap and range-bitmap writer creation by passing the selected Arrow field directly.
- Add plugin-level compatibility tests and an end-to-end embedded-index round trip.
### Tests
- `cmake --build build --target paimon-common-test -j 64`
- `cmake --build build --target paimon-core-test -j 64`
- 50 affected common-module tests covering bitmap, range bitmap, BSI, and Bloom filters.
- 7 `DataFileIndexWriterTest` cases.
- `pre-commit run --files <changed files>`
- `git diff --cached --check`
### API and Format
No public API under `include/paimon/` is changed. This adds BSI and Bloom filter writer output compatible with the existing readers and Java index formats.
### Documentation
No user-facing documentation change. The feature extends the existing file-index write interface.
### Generative AI tooling
Generated-by: OpenAI Codex (GPT-5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-index): support BSI and Bloom filter writers [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 commented on code in PR #234:
URL: https://github.com/apache/paimon-cpp/pull/234#discussion_r3842310329
##########
src/paimon/common/file_index/bloomfilter/bloom_filter_file_index.cpp:
##########
@@ -58,9 +66,85 @@ Result<std::shared_ptr<FileIndexReader>> BloomFilterFileIndex::CreateReader(
return BloomFilterFileIndexReader::Create(arrow_type, bytes);
}
+Result<std::shared_ptr<FileIndexWriter>> BloomFilterFileIndex::CreateWriter(
+ ::ArrowSchema* c_arrow_schema, const std::shared_ptr<MemoryPool>& pool) const {
+ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema> arrow_schema,
+ arrow::ImportSchema(c_arrow_schema));
+ if (arrow_schema->num_fields() != 1) {
+ return Status::Invalid(
+ "invalid schema for BloomFilterFileIndexWriter, supposed to have single field.");
+ }
+ return BloomFilterFileIndexWriter::Create(arrow_schema->field(0), options_, pool);
+}
+
+Result<std::shared_ptr<BloomFilterFileIndexWriter>> BloomFilterFileIndexWriter::Create(
+ const std::shared_ptr<arrow::Field>& field, const std::map<std::string, std::string>& options,
+ const std::shared_ptr<MemoryPool>& pool) {
+ PAIMON_ASSIGN_OR_RAISE(FastHash::HashFunction hash_function,
+ FastHash::GetHashFunction(field->type()));
+ PAIMON_ASSIGN_OR_RAISE(
+ int32_t items, OptionsUtils::GetValueFromMap<int32_t>(options, BloomFilterFileIndex::kItems,
+ BloomFilterFileIndex::kDefaultItems));
+ PAIMON_ASSIGN_OR_RAISE(
+ double fpp, OptionsUtils::GetValueFromMap<double>(options, BloomFilterFileIndex::kFpp,
+ BloomFilterFileIndex::kDefaultFpp));
+ std::shared_ptr<arrow::DataType> struct_type = arrow::struct_({field});
+ PAIMON_ASSIGN_OR_RAISE(BloomFilter64 filter, BloomFilter64::Create(items, fpp, pool));
+ return std::shared_ptr<BloomFilterFileIndexWriter>(
+ new BloomFilterFileIndexWriter(struct_type, hash_function, std::move(filter), pool));
+}
+
+BloomFilterFileIndexWriter::BloomFilterFileIndexWriter(
+ const std::shared_ptr<arrow::DataType>& struct_type,
+ const FastHash::HashFunction& hash_function, BloomFilter64&& filter,
+ const std::shared_ptr<MemoryPool>& pool)
+ : struct_type_(struct_type),
+ hash_function_(hash_function),
+ filter_(std::move(filter)),
+ pool_(pool) {}
+
+Status BloomFilterFileIndexWriter::AddBatch(::ArrowArray* batch) {
+ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Array> array,
+ arrow::ImportArray(batch, struct_type_));
+ if (!array || array->type_id() != arrow::Type::STRUCT) {
+ return Status::Invalid(
+ "invalid batch for BloomFilterFileIndexWriter, expected a struct array");
+ }
+ std::shared_ptr<arrow::StructArray> struct_array =
+ checked_pointer_cast<arrow::StructArray>(array);
+ if (struct_array->num_fields() != 1) {
+ return Status::Invalid(
+ "invalid batch for BloomFilterFileIndexWriter, expected a struct array with exactly "
+ "one field");
+ }
+ PAIMON_ASSIGN_OR_RAISE(
+ std::vector<Literal> values,
+ LiteralConverter::ConvertLiteralsFromArray(*struct_array->field(0), /*own_data=*/false));
+ for (const Literal& value : values) {
+ if (!value.IsNull()) {
+ filter_.AddHash(hash_function_(value));
+ }
+ }
+ return Status::OK();
+}
+
+Result<PAIMON_UNIQUE_PTR<Bytes>> BloomFilterFileIndexWriter::SerializedBytes() const {
+ constexpr int32_t kHeaderLength = sizeof(int32_t);
+ const int32_t bit_set_length = filter_.GetBitSet().ByteLength();
+ PAIMON_UNIQUE_PTR<Bytes> bytes =
+ Bytes::AllocateBytes(kHeaderLength + bit_set_length, pool_.get());
+ const auto num_hash_functions = static_cast<uint32_t>(filter_.GetNumHashFunctions());
+ bytes->data()[0] = static_cast<char>((num_hash_functions >> 24) & 0xff);
+ bytes->data()[1] = static_cast<char>((num_hash_functions >> 16) & 0xff);
+ bytes->data()[2] = static_cast<char>((num_hash_functions >> 8) & 0xff);
+ bytes->data()[3] = static_cast<char>(num_hash_functions & 0xff);
+ filter_.GetBitSet().ToByteArray(kHeaderLength, bit_set_length, bytes->data());
Review Comment:
Could we use the built-in utility function here instead?
##########
src/paimon/common/file_index/bloomfilter/bloom_filter_file_index.cpp:
##########
@@ -58,9 +66,85 @@ Result<std::shared_ptr<FileIndexReader>> BloomFilterFileIndex::CreateReader(
return BloomFilterFileIndexReader::Create(arrow_type, bytes);
}
+Result<std::shared_ptr<FileIndexWriter>> BloomFilterFileIndex::CreateWriter(
+ ::ArrowSchema* c_arrow_schema, const std::shared_ptr<MemoryPool>& pool) const {
+ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Schema> arrow_schema,
+ arrow::ImportSchema(c_arrow_schema));
+ if (arrow_schema->num_fields() != 1) {
+ return Status::Invalid(
+ "invalid schema for BloomFilterFileIndexWriter, supposed to have single field.");
+ }
+ return BloomFilterFileIndexWriter::Create(arrow_schema->field(0), options_, pool);
+}
+
+Result<std::shared_ptr<BloomFilterFileIndexWriter>> BloomFilterFileIndexWriter::Create(
+ const std::shared_ptr<arrow::Field>& field, const std::map<std::string, std::string>& options,
+ const std::shared_ptr<MemoryPool>& pool) {
+ PAIMON_ASSIGN_OR_RAISE(FastHash::HashFunction hash_function,
+ FastHash::GetHashFunction(field->type()));
+ PAIMON_ASSIGN_OR_RAISE(
+ int32_t items, OptionsUtils::GetValueFromMap<int32_t>(options, BloomFilterFileIndex::kItems,
+ BloomFilterFileIndex::kDefaultItems));
+ PAIMON_ASSIGN_OR_RAISE(
+ double fpp, OptionsUtils::GetValueFromMap<double>(options, BloomFilterFileIndex::kFpp,
+ BloomFilterFileIndex::kDefaultFpp));
+ std::shared_ptr<arrow::DataType> struct_type = arrow::struct_({field});
+ PAIMON_ASSIGN_OR_RAISE(BloomFilter64 filter, BloomFilter64::Create(items, fpp, pool));
+ return std::shared_ptr<BloomFilterFileIndexWriter>(
+ new BloomFilterFileIndexWriter(struct_type, hash_function, std::move(filter), pool));
+}
+
+BloomFilterFileIndexWriter::BloomFilterFileIndexWriter(
+ const std::shared_ptr<arrow::DataType>& struct_type,
+ const FastHash::HashFunction& hash_function, BloomFilter64&& filter,
+ const std::shared_ptr<MemoryPool>& pool)
+ : struct_type_(struct_type),
+ hash_function_(hash_function),
+ filter_(std::move(filter)),
+ pool_(pool) {}
+
+Status BloomFilterFileIndexWriter::AddBatch(::ArrowArray* batch) {
+ PAIMON_ASSIGN_OR_RAISE_FROM_ARROW(std::shared_ptr<arrow::Array> array,
+ arrow::ImportArray(batch, struct_type_));
+ if (!array || array->type_id() != arrow::Type::STRUCT) {
+ return Status::Invalid(
+ "invalid batch for BloomFilterFileIndexWriter, expected a struct array");
+ }
+ std::shared_ptr<arrow::StructArray> struct_array =
+ checked_pointer_cast<arrow::StructArray>(array);
+ if (struct_array->num_fields() != 1) {
+ return Status::Invalid(
+ "invalid batch for BloomFilterFileIndexWriter, expected a struct array with exactly "
+ "one field");
+ }
+ PAIMON_ASSIGN_OR_RAISE(
+ std::vector<Literal> values,
+ LiteralConverter::ConvertLiteralsFromArray(*struct_array->field(0), /*own_data=*/false));
+ for (const Literal& value : values) {
+ if (!value.IsNull()) {
+ filter_.AddHash(hash_function_(value));
Review Comment:
Could you please verify that the NaN handling is compatible with Java and add corresponding tests?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(file-index): support BSI and Bloom filter writers [paimon-cpp]
Posted by "lxy-9602 (via GitHub)" <gi...@apache.org>.
lxy-9602 merged PR #234:
URL: https://github.com/apache/paimon-cpp/pull/234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@paimon.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org