[jira] [Created] (IMPALA-15436) [DOCS] Document the Iceberg write.parquet.bloom-filter-* table properties

["Zoltán Borók-Nagy (Jira)" <ji...@apache.org>]

Zoltán Borók-Nagy created IMPALA-15436:
------------------------------------------

             Summary: [DOCS] Document the Iceberg write.parquet.bloom-filter-* table properties
                 Key: IMPALA-15436
                 URL: https://issues.apache.org/jira/browse/IMPALA-15436
             Project: IMPALA
          Issue Type: Documentation
            Reporter: Zoltán Borók-Nagy
            Assignee: Zoltán Borók-Nagy


IMPALA-12700 makes Impala honor Iceberg's own Parquet Bloom filter properties when writing Parquet files for Iceberg tables (INSERT, CTAS, UPDATE, MERGE, OPTIMIZE). The "Iceberg table properties" list in docs/topics/impala_iceberg.xml (concept iceberg_table_properties) ends at write.parquet.dict-size-bytes. Add:

- write.parquet.bloom-filter-enabled.column.<col> — 'true' writes a Parquet Bloom filter for that column (Parquet tables only)
- write.parquet.bloom-filter-max-bytes — positive 32-bit integer, Iceberg default 1048576; an upper bound on the bitset, which Impala clamps to the 64 B .. 128 MiB range
- write.parquet.bloom-filter-ndv.column.<col> — positive integer, the expected number of distinct values used to size the filter. Without it Impala uses the column's NDV statistic; with neither, the filter gets max-bytes.
- write.parquet.bloom-filter-fpp.column.<col> — false positive probability, 0 < fpp < 1, default 0.01

Then explain:
- Impala validates the values in CREATE TABLE and ALTER TABLE SET TBLPROPERTIES. CREATE TABLE also rejects these properties when write.format.default is not PARQUET ("Parquet Bloom filter properties should be set only for parquet file format"); ALTER checks only the values.
- If another engine stored invalid values, writes still succeed and warn: "Ignoring invalid Iceberg Parquet Bloom filter properties: ...".
- The Impala-specific parquet.bloom.filter.columns property wins for a column set in both.
- Writing is still gated by the PARQUET_BLOOM_FILTER_WRITE query option: NEVER disables it, and the default IF_NO_DICT writes a filter only for row groups that aren't fully dictionary-encoded.

A short example helps: CREATE TABLE ... STORED AS ICEBERG TBLPROPERTIES('write.parquet.bloom-filter-enabled.column.id'='true','write.parquet.bloom-filter-ndv.column.id'='100000').

Pre-existing gap worth folding in, not caused by IMPALA-12700: Parquet Bloom filters have no documentation at all. There is no page for PARQUET_BLOOM_FILTER_WRITE or PARQUET_BLOOM_FILTERING, and parquet.bloom.filter.columns ("col1:1024,col2", sizes rounded up to a power of two within 64 B .. 128 MiB) is undocumented. Adding those pages,
with ditamap and keydef entries, gives the Iceberg properties something to link to.

The behavior is additive, so no incompatible-changes entry.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org