You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "laskoviymishka (via GitHub)" <gi...@apache.org> on 2026/04/03 12:35:59 UTC
[I] feat: bloom filter write support and scan pruning [iceberg-go]
laskoviymishka opened a new issue, #834:
URL: https://github.com/apache/iceberg-go/issues/834
### Feature Request / Improvement
Parent: #829 (v2 spec completion)
Bloom filter table properties are defined but never applied. The Arrow Parquet library fully supports bloom filters on both sides — we just need to wire them in.
### Write side (not wired)
Properties already defined in `table/properties.go` and `table/internal/parquet_files.go`:
- `write.parquet.bloom-filter-max-bytes` (default 1MB)
- `write.parquet.bloom-filter-enabled.column.<name>`
But `GetWriteProperties()` in `parquet_files.go` never reads them. The underlying Arrow Parquet writer supports:
- `parquet.WithBloomFilterEnabled(bool)` — global default
- `parquet.WithBloomFilterEnabledFor(path, bool)` — per-column
- `parquet.WithMaxBloomFilterBytes(n)` — max size
Just need to parse the properties and append to `writerProps` in `GetWriteProperties`.
### Read side (not implemented)
Row group filtering currently uses column statistics only (`newParquetRowGroupStatsEvaluator` in `table/evaluators.go`, wired in `table/arrow_scanner.go`). After stats say "possible", a bloom filter check could eliminate additional row groups for `EqualTo` and `In` predicates.
Arrow Parquet reader provides:
- `file.Reader.GetBloomFilterReader()` → `BloomFilterReader`
- `RowGroupBloomFilterReader.GetColumnBloomFilter(i)` → `BloomFilter` with `CheckHash(uint64) bool`
Integration point: `wrapPqArrowReader.GetRecords` in `table/internal/parquet_files.go` — after the existing `testRg` stats pass, add a bloom filter pass over surviving row groups.
### Impact
Bloom filters are most valuable for high-cardinality columns used in equality filters (e.g. `user_id = X`). For tables with many row groups and selective equality predicates, this can dramatically reduce I/O by skipping row groups that column stats alone can't eliminate (when min/max range is wide but the actual value is absent).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: bloom filter write support and scan pruning [iceberg-go]
Posted by "tanmayrauth (via GitHub)" <gi...@apache.org>.
tanmayrauth commented on issue #834:
URL: https://github.com/apache/iceberg-go/issues/834#issuecomment-4241698663
@laskoviymishka, since the write part is now wired ( #878 ), I’ve created a [PR](https://github.com/apache/iceberg-go/pull/891) for the read side. Should we track this in this same issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: bloom filter write support and scan pruning [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #834:
URL: https://github.com/apache/iceberg-go/issues/834#issuecomment-4360681047
closing as completed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] feat: bloom filter write support and scan pruning [iceberg-go]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade closed issue #834: feat: bloom filter write support and scan pruning
URL: https://github.com/apache/iceberg-go/issues/834
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org