You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@hudi.apache.org by "rangareddy (via GitHub)" <gi...@apache.org> on 2026/08/12 06:45:06 UTC

[PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

rangareddy opened a new pull request, #19590:
URL: https://github.com/apache/hudi/pull/19590

   ### Describe the issue this Pull Request addresses
   
   Closes #16063 (HUDI-6456), "Add parquet blooms documentation".
   
   Hudi forwards Parquet's per-column bloom filter settings through to the Parquet writer, but the only trace of
   bloom filters in the docs was one auto-generated row in `configurations.md` — and that row is for a **different
   feature** with a near-identical name. So nothing told a user that these keys exist, that they are per column,
   or that they are read from the Hadoop configuration:
   
   ```
   parquet.bloom.filter.enabled#<column>
   parquet.bloom.filter.expected.ndv#<column>
   ```
   
   ### Summary and Changelog
   
   Adds a **Parquet Bloom Filters** subsection to `performance.md`, under `Read Path` next to `Data Skipping`,
   because that is precisely the gap it fills: column stats prune on ranges and so help least for an equality
   predicate on a high-cardinality column whose min-max spans nearly every file — which is what a bloom filter
   covers.
   
   The section states the two keys, that they are per column, how to set them from Spark (`spark.hadoop.` prefix),
   how to size `expected.ndv`, and that it is a write-time decision so an existing table only picks them up as it
   is rewritten.
   
   Two caveats are called out, because both are easy to get wrong and neither was written down anywhere:
   
   1. **`hoodie.parquet.bloom.filter.enabled` is not this feature.** It controls whether Hudi writes a bloom
      filter **of record keys** into the footer for the [bloom index](https://hudi.apache.org/docs/indexes) during
      upserts. It defaults to `true`, applies only when meta fields are populated, and is implied anyway when
      `hoodie.index.type` names a `BLOOM` index. Setting it does **not** enable per-column Parquet filters. Given
      the names differ by one prefix, a user reading `configurations.md` could reasonably conclude the opposite.
   2. **The settings are applied reflectively and failure is swallowed.** On a Parquet without
      `withBloomFilterEnabled` / `withBloomFilterNDV`, the keys are silently ignored rather than rejected.
   
   ### Verification
   
   Docs change, so no test to add — said plainly rather than implied. Every claim was read off master
   (`3ba31dd37fff`):
   
   | claim | source |
   | --- | --- |
   | keys are read from the **Hadoop** config, per column | `HoodieBaseParquetWriter.handleParquetBloomFilters`, `:98-119` — iterates the `Configuration` and splits the column off each key |
   | exact key spelling | `HoodieBaseParquetWriter:52-53`, and `TestHoodieParquetBloom.scala:36-37` sets `parquet.bloom.filter.enabled#bloom_col` / `expected.ndv#bloom_col` on `jsc.hadoopConfiguration` |
   | forwarded via `withBloomFilterEnabled` / `withBloomFilterNDV` | `:105` and `:114`, invoked by reflection |
   | silently ignored on an older Parquet | `:107-109` and `:116-118` — `NoSuchMethodException` is caught and skipped |
   | `hoodie.parquet.bloom.filter.enabled` is about record-key blooms for the bloom index | `HoodieStorageConfig:252-258` (default `true`, since 0.15.0) and `HoodieFileWriterFactory.enableBloomFilter:141-146` (`populateMetaFields && (thisConfig ‖ index.type contains BLOOM)`) |
   | Flink sets them the same way | `TestHoodieRowDataParquetConfigInjector:100-101` |
   
   Markdown checked by running `markdownlint` over both files **before and after** and comparing rule *classes*
   rather than counts, since these files already carry findings:
   
   ```
   before: MD007 MD009 MD012 MD013 MD031 MD033 MD040
   after : MD007 MD009 MD012 MD013 MD031 MD033 MD040   ->  no new class of finding
   ```
   
   Also checked: every `<column>` occurrence sits inside inline code, so nothing new is exposed to the MDX
   parser; the `indexes.md` link target exists in **both** doc trees so the relative link resolves in the
   versioned copy too; and the inserted region is byte-identical between the two files.
   
   **Not done:** the Docusaurus build (`website/node_modules` absent, full install heavy). The section adds a
   table, a fenced block and two admonitions, all constructs already used on this page.
   
   ### Impact
   
   Documentation only — no code, config, API or format change. Two features whose names differ by one prefix are
   now distinguishable, and a write-side knob that previously required reading `HoodieBaseParquetWriter` to
   discover is now documented.
   
   ### Risk Level
   
   none
   
   ### Documentation Update
   
   This is the documentation update. Targets `asf-site`; applied to `docs/` and
   `versioned_docs/version-1.2.0/`, matching the file set of the recently merged #19572. Older versioned copies
   are frozen snapshots and are left alone.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable — n/a for docs; verification table above instead
   - [ ] CI passes on my PR — `asf-site` PRs do not run the `master` gates; `markdownlint` parity checked locally
         as described
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.

rangareddy commented on PR #19590:
URL: https://github.com/apache/hudi/pull/19590#issuecomment-5263499949

   Updating this PR rather than closing it, which retracts the "please close this in favour of #9056" line in my previous comment — that recommendation no longer stands, and I would rather correct it here than leave a reviewer acting on it.
   
   **Extended to every supported release.** The section now lands in the current docs plus all five supported versioned copies — 1.2.0, 1.1.1, 1.0.2, 1.0.1 and 1.0.0 — because the feature exists in all of them. Previously it was only `docs/` + 1.2.0, which was too narrow: I had followed #19572's file set, but that PR documented a table service whose page is 1.2-specific, whereas this is a feature that has shipped since 0.14.0.
   
   Checked rather than assumed, per version:
   
   * `handleParquetBloomFilters` is **absent** at `release-0.13.1` and present from `release-0.14.0` onward, so every versioned copy on the site (all ≥ 0.14.0) has the feature.
   * the `hoodie.parquet.bloom.filter.enabled` config this section contrasts against reads `defaultValue(true)` / `sinceVersion("0.15.0")` identically at `release-1.0.0`, `1.0.2`, `1.1.1` and `1.2.0` — so the admonition is accurate in each version it lands in, not just on master.
   * the five-version fan-out matches how other cross-version docs fixes have been applied here (#19555, #19459).
   
   One structural difference worth flagging for review: **`version-1.0.0/performance.md` has no `## Related Resources` heading** — that page simply ends after Data Skipping — so there the section is appended at the end of the file, while in the other five it is inserted before that anchor. The inserted text itself is byte-identical in all six.
   
   I left 0.15.x and 0.14.x alone deliberately: they have had no, or nearly no, doc commits in the last six months and are effectively frozen. Happy to extend to them if the project would rather have full coverage.
   
   Re-validated after the change, across all six files: `markdownlint` introduces no new class of finding (compared by rule class, since these pages already carry ~7 classes including 237 `MD013`s), every `<column>` sits inside inline code so nothing new reaches the MDX parser, and `indexes.md` exists in each of the five versioned trees so the relative link resolves everywhere.
   
   @parisni — #9056 is still the original and my review comments there stand on their own; if a committer prefers your dedicated page, I am glad to fold this content into it and close this.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.

hudi-agent commented on code in PR #19590:
URL: https://github.com/apache/hudi/pull/19590#discussion_r3764295331


##########
website/docs/performance.md:
##########
@@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to
   - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default)
   - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path)
 
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a
+high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the **Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for `<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct values, which sizes the filter |
+
+Set them on the Hadoop configuration your writer uses. From Spark, the `spark.hadoop.` prefix forwards them:

Review Comment:
   🤖 In the example, `--conf spark.hadoop.parquet.bloom.filter.enabled#uuid=true` uses `uuid` as the column, which reads as though it's a Hudi record-key column. Since the preceding paragraph is about high-cardinality equality predicates on arbitrary data columns, it might help to use a neutral column name (or add one sentence clarifying that `<column>` is any data column you filter on by equality), so readers don't conflate this with the record-key bloom index you carefully distinguish in the note just below.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



##########
website/docs/performance.md:
##########
@@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to
   - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default)
   - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path)
 
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a
+high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that

Review Comment:
   🤖 This section is placed under **Read Path** and states that "a reader consults them to skip row groups," but everything documented below is write-side (the `parquet.bloom.filter.enabled#<column>` / `expected.ndv` writer keys). It might help to spell out what the *reader* needs for these filters to take effect — Parquet bloom-filter row-group filtering has to be enabled on the read side (e.g. `parquet.filter.bloom.enabled=true`) and depends on predicate pushdown, and Spark's vectorized reader hasn't always honored bloom filters across versions. As written, a user could set the two write keys, see no query speedup, and reasonably conclude the docs are wrong. @yihua could you confirm whether Hudi's Spark read path actually consults Parquet bloom filters out of the box, and what (if any) reader-side config is required? That would let us either document the read step or reframe the section as write-side only.
   
   <sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.

rangareddy commented on PR #19590:
URL: https://github.com/apache/hudi/pull/19590#issuecomment-5263412148

Superseded — this duplicates **#9056**, "[HUDI-6456] [DOC] Add parquet blooms documentation" by @parisni, which has been open since 2023-06-26 and is the original work for #16063. parisni wrote the parquet bloom support itself (#8716), so that PR should land, not this one.

I missed it when I opened this: I looked for cross-referenced PRs on the issue timeline, and because #9056 predates the JIRA-migrated issue it never linked there — even though hudi-bot had posted a "Linked PR(s)" comment naming it on #16063. The lesson for me is that the timeline is not a sufficient check for migrated issues.

I have moved the material worth keeping onto #9056 as review comments rather than leaving it here: the `hoodie.parquet.bloom.filter.enabled` name collision, the silently-swallowed reflection failure, a correction to the COW-only limitation (MOR base files go through the same writer; it is the log blocks that miss out), a version typo, and the fact that a feature shipped since 0.14.0 wants the versioned copies too.

Leaving this open only so a committer can compare shapes — #9056 adds a dedicated `parquet_bloom.md` page, this one adds a section to `performance.md` next to Data Skipping. If the page is preferred, which I would expect, please close this in favour of #9056.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.

rangareddy commented on code in PR #19590:
URL: https://github.com/apache/hudi/pull/19590#discussion_r3764335190


##########
website/docs/performance.md:
##########
@@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to
   - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default)
   - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path)
 
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a
+high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that
+case. They are written into the Parquet file itself, and a reader consults them to skip row groups that
+cannot contain the value being searched for.
+
+Hudi passes these through to the Parquet writer, per column, from the **Hadoop** configuration:
+
+| key | meaning |
+| --- | --- |
+| `parquet.bloom.filter.enabled#<column>` | write a bloom filter for `<column>` |
+| `parquet.bloom.filter.expected.ndv#<column>` | expected number of distinct values, which sizes the filter |
+
+Set them on the Hadoop configuration your writer uses. From Spark, the `spark.hadoop.` prefix forwards them:

Review Comment:
   Applied in `41a3a0d`. You are right that `uuid` was a poor choice directly above a note distinguishing this from the record-key bloom index — it invited exactly the conflation the note exists to prevent. The example now uses `session_id`, and I took the second half of your suggestion too rather than relying on the reader to infer it:
   
   > `<column>` is any data column you filter on by equality — not the record key, and unrelated to the record-key bloom index discussed in the note below.
   
   Both changes are in all six copies of the page.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.

rangareddy commented on code in PR #19590:
URL: https://github.com/apache/hudi/pull/19590#discussion_r3764333086


##########
website/docs/performance.md:
##########
@@ -132,6 +132,48 @@ To enable Data Skipping in your queries make sure to set following properties to
   - `hoodie.metadata.enable` (to enable metadata table use on the read path, enabled by default)
   - `hoodie.metadata.index.column.stats.enable` (to enable column stats index use on the read path)
 
+#### Parquet Bloom Filters
+
+Column stats prune on ranges, so they help least where they are needed most: an equality predicate on a
+high-cardinality column whose min-max range covers almost every file. Parquet's own bloom filters cover that

Review Comment:
   Good catch, and it needed answering rather than hand-waving — but the underlying premise turns out to be testable, and the answer is better than "reframe as write-side only".
   
   Hudi has an end-to-end test for exactly this, `TestHoodieParquetBloomFilter` in `hudi-spark`. It sets the two write keys on `jsc.hadoopConfiguration`, then deliberately disables the other two filter levels so nothing else can cause a skip:
   
   ```scala
   sparkSession.sql("set parquet.filter.columnindex.enabled=false")
   sparkSession.sql("set parquet.filter.stats.enabled=false")
   ```
   
   writes through `format("hudi")`, and reads back through `sparkSession.read.format("hudi")`:
   
   ```scala
   .filter("bloom_col = '3'")   // value absent  -> asserts 0 row groups scanned (Spark >= 3)
   .filter("bloom_col = '2'")   // value present -> asserts 1
   ```
   
   parameterised over BULK_INSERT, INSERT, UPSERT and INSERT_OVERWRITE on COW. So the Hudi Spark read path does consult Parquet bloom filters out of the box on Spark 3.x, and the row-group skip is asserted, not assumed. Note also what the test implies about your `parquet.filter.bloom.enabled` question: it switches off *stats* and *columnindex* but never touches the bloom switch, so bloom filtering is on at its parquet-mr default — and `grep` confirms Hudi never sets that key anywhere.
   
   You were still right that the section said "a reader consults them" and then documented only write-side keys, leaving a reader to guess. Added in `1d0…` → now `41a3a0d`:
   
   > On the read side nothing extra needs configuring for Spark 3.x. Reading the table back through the Hudi datasource consults the filters, provided the query carries an equality predicate that can be pushed down to the Parquet reader — a query filtering on a value no row group contains skips those row groups entirely. Parquet's own read-side switch, `parquet.filter.bloom.enabled`, is left at its default and Hudi never overrides it, so there is no reader-side flag to turn on.
   
   One deliberate omission: I do **not** name `TestHoodieParquetBloomFilter` in the docs page. That is the same objection you raised on #19486 — a user-facing reference to a test class rots silently on a rename — so the evidence lives in the commit message and this thread instead. I had it in the page for one revision and took it out for that reason.
   
   I have left the Spark-4 / vectorized-reader question alone rather than guessing: the test gates its assertion on `spark.version >= 3`, so Spark 3.x is what is actually pinned, and that is all the text claims.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "voonhous (via GitHub)" <gi...@apache.org>.

voonhous merged PR #19590:
URL: https://github.com/apache/hudi/pull/19590


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] docs(performance): document Parquet bloom filters and separate them from the bloom index [hudi]

Posted by "uros-b (via GitHub)" <gi...@apache.org>.

uros-b commented on PR #19590:
URL: https://github.com/apache/hudi/pull/19590#issuecomment-5392624866

   Thank you @rangareddy! +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org