You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "renaudb (via GitHub)" <gi...@apache.org> on 2026/04/21 21:05:43 UTC
[PR] feat: Add support for writing bloom filters [iceberg-python]
renaudb opened a new pull request, #3265:
URL: https://github.com/apache/iceberg-python/pull/3265
Closes #850
# Rationale for this change
Add support for writing bloom filters to parquet files. This changes leverages the new `bloom_filter_options` `write_parquet` argument in pyarrow 24.
## Are these changes tested?
Added tests for the metadata parsing. Added a very basic test for the writing path (there is currently no way to test for the existence of a bloomfilter in a parquet file using pyarrow).
## Are there any user-facing changes?
N/A
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-5062909087
Pyarrow 25 was released so the test for bloomfilter existence should now pass.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #3265: feat: Add support for writing bloom filters
URL: https://github.com/apache/iceberg-python/pull/3265
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on code in PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#discussion_r3521296026
##########
tests/integration/test_writes/test_writes.py:
##########
@@ -712,6 +718,27 @@ def test_write_parquet_unsupported_properties(
tbl.append(arrow_table_with_null)
+@pytest.mark.integration
+@skip_if_bloom_filter_not_supported
+@pytest.mark.parametrize("format_version", [1, 2])
+def test_write_parquet_bloom_filter_properties(
Review Comment:
Added a check that the filter offset and and length exist in the underlying files. However, this requires pyarrow 25 which won't be officially released for a few days.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-4828087539
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that's incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-5208274708
Reopened here: https://github.com/apache/iceberg-python/pull/3757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "Fokko (via GitHub)" <gi...@apache.org>.
Fokko commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-4417439383
Thanks for working on this @renaudb. I've noticed that in the `uv.lock` we're behind some versions of boto3/botocore, not sure why this. Maybe we can bump this with `uv sync --upgrade` manually (in a separate PR).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-4509322761
@Fokko this should be ready for review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-5062927375
@abnobdoss do you have the ability to reopen the PR? I don't see to have the permission to do it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on code in PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#discussion_r3321758372
##########
pyiceberg/io/pyarrow.py:
##########
@@ -2757,14 +2877,25 @@ def parquet_file_to_data_file(io: FileIO, table_metadata: TableMetadata, file_pa
PYARROW_UNCOMPRESSED_CODEC = "none"
-def _get_parquet_writer_kwargs(table_properties: Properties) -> dict[str, Any]:
+def _get_parquet_writer_kwargs(table_properties: Properties, file_schema: Schema) -> dict[str, Any]:
from pyiceberg.table import TableProperties
- for key_pattern in [
+ unsupported_key_patterns = [
TableProperties.PARQUET_ROW_GROUP_SIZE_BYTES,
TableProperties.PARQUET_BLOOM_FILTER_MAX_BYTES,
- f"{TableProperties.PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX}.*",
- ]:
+ ]
+
+ from packaging import version
+
+ MIN_PYARROW_VERSION_SUPPORTING_BLOOM_FILTER_WRITES = "24.0.0"
+ if version.parse(pyarrow.__version__) < version.parse(MIN_PYARROW_VERSION_SUPPORTING_BLOOM_FILTER_WRITES):
Review Comment:
Would it be better to make this explicitly a different error since it is implemented but is gate on the dependency version?
##########
pyiceberg/io/pyarrow.py:
##########
@@ -2613,6 +2731,8 @@ def write_parquet(task: WriteTask) -> DataFile:
else:
file_schema = table_schema
+ parquet_writer_kwargs = _get_parquet_writer_kwargs(table_metadata.properties, file_schema)
Review Comment:
If I understand correctly, neither input varies per file: table_metadata.properties is table-level, and file_schema is derived solely from table_metadata.schema() (just sanitized), so it's identical for every task.
Would it make sense to lift the file_schema derivation and this _get_parquet_writer_kwargs call back out of write_parquet, computing them once per write_file call instead of once per write_parquet call?
##########
tests/integration/test_writes/test_writes.py:
##########
@@ -712,6 +718,27 @@ def test_write_parquet_unsupported_properties(
tbl.append(arrow_table_with_null)
+@pytest.mark.integration
+@skip_if_bloom_filter_not_supported
+@pytest.mark.parametrize("format_version", [1, 2])
+def test_write_parquet_bloom_filter_properties(
Review Comment:
Would it make sense to assert pq.ParquetWriter is called with bloom_filter_options?
##########
pyiceberg/io/pyarrow.py:
##########
@@ -2474,6 +2479,120 @@ def parquet_path_to_id_mapping(
return result
+def id_to_parquet_path_mapping(schema: Schema) -> dict[int, str]:
+ """
+ Compute the mapping of Iceberg column ID to parquet column path.
+
+ Args:
+ schema (pyiceberg.schema.Schema): The current table schema.
+ """
+ result: dict[int, str] = {}
+ for pair in pre_order_visit(schema, ID2ParquetPathVisitor()):
+ result[pair.field_id] = pair.parquet_path
+ return result
+
+
+@dataclass(frozen=True)
+class BloomFilterOptions:
+ parquet_path: str
+ ndv: int | None
+ fpp: float | None
+
+
+class BloomFilterOptionsCollector(PreOrderSchemaVisitor[list[BloomFilterOptions]]):
+ _field_id: int = 0
+ _schema: Schema
+ _properties: dict[str, str]
+
+ def __init__(self, schema: Schema, properties: dict[str, str], id_to_parquet_path_mapping: dict[int, str]):
+ self._schema = schema
+ self._properties = properties
+ self._id_to_parquet_path_mapping = id_to_parquet_path_mapping
+
+ def schema(
+ self, schema: Schema, struct_result: Callable[[], builtins.list[BloomFilterOptions]]
+ ) -> builtins.list[BloomFilterOptions]:
+ return struct_result()
+
+ def struct(
+ self, struct: StructType, field_results: builtins.list[Callable[[], builtins.list[BloomFilterOptions]]]
+ ) -> builtins.list[BloomFilterOptions]:
+ return list(itertools.chain(*[result() for result in field_results]))
+
+ def field(
+ self, field: NestedField, field_result: Callable[[], builtins.list[BloomFilterOptions]]
+ ) -> builtins.list[BloomFilterOptions]:
+ self._field_id = field.field_id
+ return field_result()
+
+ def list(
+ self, list_type: ListType, element_result: Callable[[], builtins.list[BloomFilterOptions]]
+ ) -> builtins.list[BloomFilterOptions]:
+ self._field_id = list_type.element_id
+ return element_result()
+
+ def map(
+ self,
+ map_type: MapType,
+ key_result: Callable[[], builtins.list[BloomFilterOptions]],
+ value_result: Callable[[], builtins.list[BloomFilterOptions]],
+ ) -> builtins.list[BloomFilterOptions]:
+ self._field_id = map_type.key_id
+ k = key_result()
+ self._field_id = map_type.value_id
+ v = value_result()
+ return k + v
+
+ def primitive(self, primitive: PrimitiveType) -> builtins.list[BloomFilterOptions]:
+ from pyiceberg.table import TableProperties
+
+ column_name = self._schema.find_column_name(self._field_id)
+ if column_name is None:
+ return []
+
+ parquet_path = self._id_to_parquet_path_mapping.get(self._field_id)
+ if parquet_path is None:
+ return []
+
+ bloom_filter_enabled = property_as_bool(
+ self._properties, f"{TableProperties.PARQUET_BLOOM_FILTER_COLUMN_ENABLED_PREFIX}.{column_name}", False
+ )
+ if not bloom_filter_enabled:
+ return []
+
+ bloom_filter_fpp = property_as_float(
+ self._properties, f"{TableProperties.PARQUET_BLOOM_FILTER_COLUMN_FPP_PREFIX}.{column_name}", None
+ )
+ bloom_filter_ndv = property_as_int(
+ self._properties, f"{TableProperties.PARQUET_BLOOM_FILTER_COLUMN_NDV_PREFIX}.{column_name}", None
+ )
+
+ return [BloomFilterOptions(parquet_path=parquet_path, ndv=bloom_filter_ndv, fpp=bloom_filter_fpp)]
+
+
+def get_bloom_filter_options(
+ schema: Schema,
+ table_properties: dict[str, str],
+) -> dict[str, dict[str, Any]]:
+ """
+ Get the bloom filter options from the table properties.
+
+ Args:
+ schema (pyiceberg.schema.Schema): The current table schema.
+ table_properties (dict[str, str]): The table properties.
+ """
+ bloom_filter_options = pre_order_visit(
Review Comment:
I think parquet_path_to_id_mapping(file_schema) already walks this schema with ID2ParquetPathVisitor for stats. If I'm reading it right, the bloom path adds a couple more passes over the same schema. Would it make sense to do a single ID2ParquetPathVisitor pass and deriving both the bloom options and the stats mapping from it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "renaudb (via GitHub)" <gi...@apache.org>.
renaudb commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-4422157395
@Fokko the issue is with Bodo, not Boto. Bodo forces `pyarrow>=23.0,<23.1`. I saw a bunch of issues filed on their hand about how this is too restrictive, but it looks like they are limited in changing it.
https://github.com/bodo-ai/Bodo/blob/main/pyproject.toml#L9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "abnobdoss (via GitHub)" <gi...@apache.org>.
abnobdoss commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-5086568536
Hi @renaudb, I don't believe I can either.
@KevinJiao / @Fokko / @geruh would you be able to re-open this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-5204065550
@renaudb otherwise you could open a new PR ( using the same branch ). Thanks again for all this works
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [PR] feat: Add support for writing bloom filters [iceberg-python]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3265:
URL: https://github.com/apache/iceberg-python/pull/3265#issuecomment-4940780487
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org