You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/09/02 22:12:19 UTC

[PR] feat(reader): merge-aware pushdown gate, row-group pruning, and read-volume counters [hudi-rs]

linliu-code opened a new pull request, #708:
URL: https://github.com/apache/hudi-rs/pull/708

   ## Description
   
   Three related changes to base-file predicate pushdown in `reader_v2`, smallest first. Each commit stands alone.
   
   ### 1. A base-only slice may push, whatever the table type
   
   A file slice with no log files has no merge: no log record can supersede a base row and no delete block can remove one, so nothing can flip a pushed predicate's outcome. The gate on `ReaderContext` answered only the table-level question — CoW, or MOR whose predicate is confined to immutable primary keys — so on a MOR table with a data-column predicate, **every base-only slice lost pushdown**: the base read returned whole columns for a post-merge filter to discard, having read every byte of them.
   
   The declarative harness had been standing in for the missing rule by declaring a base-only slice `COPY_ON_WRITE` regardless of the fixture's real table type. It now passes `MERGE_ON_READ`, which is what those fixtures are, so the eight base-only filter cases exercise the split-level gate rather than an `is_cow()` branch that was never true in production for them. Those cases assert exact rows and poison if the filter is not applied — reverting the gate fails all eight.
   
   ### 2. The gate is about whether the read merges, not about the table type
   
   Keeping the table-type disjunct hides what the rule is: a CoW slice is safe because it carries no log files, not because of what `hoodie.table.type` says. Java agrees — `SparkFileFormatInternalRowReaderContext.getSchemaAndFiltersForRead` branches on `getHasLogFiles()` and never on the table type. So the gate becomes
   
   ```
   base_read_pushdown_is_safe() = no log files on the split || mor_pk_safe
   ```
   
   with the same outcomes, and `ReaderContext::can_push_row_filter` / `is_cow` go away. The doc comment records why the split's own file list is used rather than `ReaderContext::has_log_files` (different fact, different source — a safety gate should not rest on a caller-supplied boolean), and why Java's bootstrap tier has no counterpart here.
   
   The parquet log-block gate now reads `mor_pk_safe` directly. A log block only exists on a slice that has log files, so the base gate's first disjunct is false there by construction and the condition already reduced to exactly this. No behaviour change: the disjunct it drops could only be true for a CoW table with log blocks.
   
   ### 3. Row groups can be pruned, and reads can be measured
   
   A `RowFilter` never skips IO — it decides per row, after the predicate columns have been fetched and decoded. Nothing on this path could skip a row group.
   
   - `RowGroupSelector`: a caller-supplied closure over the file's parsed footer returning the row groups to keep, `None` meaning "no opinion". It runs against metadata the reader has already fetched, so consulting it costs nothing, and before the row filter is installed, so the filter only ever sees groups that survived. It reaches a read through `BaseFileReadOptions::with_row_group_selector` or `HoodieFileGroupReaderBuilder::with_row_group_selector`.
   - Both mechanisms share **one** safety gate, bound to a local so the sharing is structural rather than two call sites that happen to agree. Pruning is the one that must not drift: a row filter still sees every row, while a pruned row group is gone before the merge could have updated one of its rows into a match.
   - `ReadVolume` on `Storage` (one per file-group read) counts bytes fetched, round trips, row groups scanned vs. row groups the file has, rows the file has, and rows the stream yielded — plus `row_group_selector_calls` and `row_group_selector_suppressed`. Bytes and calls are counted in a `CountingReader` at the `AsyncFileReader` boundary, which makes them exact and cache-independent: a warm re-read reports the same bytes as a cold one, which wall-clock does not.
   - A selector the gate refuses is counted, because a suppressed selector and an absent one both read as zero calls and only the counter separates them.
   
   `BaseFileReader::read_schema` is new (with a default implementation, so no other format changes) for the base read's schema-evolution probe, which was opening a whole stream to take its schema and dropping it. The parquet override answers from the footer instead. This is correctness for the counters, not tidiness: the probe went through the same read path it was measuring, so a three-row-group file reported six row groups scanned before a single data byte moved.
   
   ### API changes
   
   | Change | Note |
   |---|---|
   | removed `ReaderContext::can_push_row_filter`, `ReaderContext::is_cow` | superseded by `base_read_pushdown_is_safe()`; the table type no longer decides anything about a read |
   | added `RowGroupSelector`, `ReadVolume`, `Storage::read_volume()` | new, in `storage` |
   | added `BaseFileReadOptions::with_row_group_selector`, `HoodieFileGroupReaderBuilder::with_row_group_selector` | mirror the existing row-filter setters |
   | added `BaseFileReader::read_schema` | defaulted; parquet overrides it |
   
   No behaviour change for `Table` or DataFusion reads: `resolve_reader_context` installs neither a filter nor a selector, so those paths push nothing, exactly as before.
   
   ## How are the changes test-covered
   
   - [ ] N/A
   - [x] Automated tests (unit and/or integration tests)
   - [ ] Manual tests
   
   Each change was falsified by reverting it and watching the corresponding tests go red, not only by watching them pass:
   
   | Change | Discriminating test | With the change reverted |
   |---|---|---|
   | split-level gate | 8 base-only harness cases, exact-row asserts on MOR fixtures | 8 failed, 3 passed |
   | row-group pruning | 3-row base file written one row per row group, selector keeping group 0 | 2 failed — 3 rows returned instead of 1 |
   | suppression counter | selector installed on a merging slice with a non-PK-safe predicate | n/a — asserts `calls == 0 && suppressed == 1` |
   | read volume | a `RowFilter` rejecting every row | `rows_out == 0` while `bytes_read > 0`, which is the case the counters exist for |
   
   `cargo test --no-fail-fast --all-targets --all-features --workspace` and `cargo clippy --all-targets --all-features --workspace --no-deps -- -D warnings` are clean.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(reader): merge-aware pushdown gate, row-group pruning, and read-volume counters [hudi-rs]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #708:
URL: https://github.com/apache/hudi-rs/pull/708


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(reader): merge-aware pushdown gate, row-group pruning, and read-volume counters [hudi-rs]

Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #708:
URL: https://github.com/apache/hudi-rs/pull/708#discussion_r3919813301


##########
crates/core/src/file_group/base_file/parquet.rs:
##########
@@ -229,22 +328,39 @@ impl BaseFileReader for ParquetBaseFileReader {
             let builder = self
                 .open_builder(relative_path, options.row_index_column.as_deref())
                 .await?;
-            let builder = Self::apply_options(builder, &options)?;
+            let builder = self.apply_options(builder, &options)?;
             let full_schema = builder.schema().clone();
             let stream = builder.build()?;
             let schema = Self::schema_with_row_index(
                 stream.schema(),
                 &full_schema,
                 options.row_index_column.as_deref(),
             )?;
+            // Rows the stream actually yields — after any row filter. Against
+            // `file_rows` this is the read's selectivity; against `bytes_read`,
+            // what that selectivity cost.
+            let volume = self.storage.read_volume.clone();
             let mapped_stream = stream
-                .map(|result| result.map_err(StorageError::from))
+                .map(move |result| {
+                    let batch = result.map_err(StorageError::from)?;
+                    volume.add_rows_out(batch.num_rows() as u64);
+                    Ok(batch)
+                })
                 .boxed();
 
             Ok(BaseFileStream::new(schema, mapped_stream))
         })
     }
 
+    /// Answered from the footer alone: no stream is built, and no read-volume
+    /// counter moves for a call that reads no data.
+    fn read_schema<'a>(

Review Comment:
   non-blocking: get_schema converts with `None` key-value metadata, so a file carrying ARROW:schema footer metadata reports different types than the stream the default impl consults (the current caller only uses field names, so nothing breaks today). Deriving the answer from ArrowReaderMetadata::try_new on the already-fetched footer would make the override exactly the stream schema at the same IO cost.



##########
crates/core/src/file_group/reader_v2/engine.rs:
##########
@@ -1629,18 +1761,235 @@ mod tests {
         let reader = HoodieFileGroupReader::builder()
             .with_reader_context(dummy_reader_context("MERGE_ON_READ"))
             .with_storage(storage)
-            .with_input_split(dummy_input_split())
+            .with_input_split(merging_input_split())
             .with_row_filter_builder(make_row_filter_builder())
             // mor_pk_safe defaults to false
             .build()
             .unwrap();
         assert!(!reader.reader_context.mor_pk_safe);
         assert!(
-            !reader.reader_context.can_push_row_filter(),
+            !reader.base_read_pushdown_is_safe(),
             "MOR without PK-safety must NOT push (mirrors Java's morFilters gate)"
         );
     }
 
+    /// A MOR slice with no log files does not merge, so the predicate is safe to
+    /// push whatever `mor_pk_safe` says. Parameterized over both values so the
+    /// "does it merge" rule is shown to be independent of PK safety.
+    #[test]
+    fn base_only_mor_slice_allows_pushdown_regardless_of_pk_safety() {
+        for mor_pk_safe in [false, true] {
+            let storage = Storage::new_with_base_url(parse_uri("file:///tmp").unwrap()).unwrap();
+            let reader = HoodieFileGroupReader::builder()
+                .with_reader_context(dummy_reader_context("MERGE_ON_READ"))
+                .with_storage(storage)
+                .with_input_split(InputSplit::new(
+                    Some("base.parquet".to_string()),
+                    None,
+                    // No log files => no merge => nothing can flip the predicate.
+                    vec![],
+                    "p1".to_string(),
+                ))
+                .with_row_filter_builder(make_row_filter_builder())
+                .with_mor_pk_safe(mor_pk_safe)
+                .build()
+                .unwrap();
+            assert!(
+                reader.base_read_pushdown_is_safe(),
+                "base-only slice must push regardless of mor_pk_safe ({mor_pk_safe})"
+            );
+        }
+    }
+
+    /// The split rule must not weaken the real MOR case: with log files present
+    /// the merge can supersede or delete a base row, so a non-PK-safe predicate
+    /// still may not be pushed.
+    #[test]
+    fn mor_slice_with_log_files_still_blocks_pushdown_when_not_pk_safe() {
+        let storage = Storage::new_with_base_url(parse_uri("file:///tmp").unwrap()).unwrap();
+        let reader = HoodieFileGroupReader::builder()
+            .with_reader_context(dummy_reader_context("MERGE_ON_READ"))
+            .with_storage(storage)
+            .with_input_split(InputSplit::new(
+                Some("base.parquet".to_string()),
+                None,
+                vec![".log.1".to_string()],
+                "p1".to_string(),
+            ))
+            .with_row_filter_builder(make_row_filter_builder())
+            // mor_pk_safe defaults to false
+            .build()
+            .unwrap();
+        assert!(reader.input_split.has_log_files());
+        assert!(
+            !reader.base_read_pushdown_is_safe(),
+            "MOR with log files and no PK safety must NOT push"
+        );
+    }
+
+    #[test]
+    fn builder_routes_row_group_selector_into_reader_context() {
+        let storage = Storage::new_with_base_url(parse_uri("file:///tmp").unwrap()).unwrap();
+        let reader = HoodieFileGroupReader::builder()
+            .with_reader_context(dummy_reader_context("MERGE_ON_READ"))
+            .with_storage(storage)
+            .with_input_split(dummy_input_split())
+            .with_row_group_selector(std::sync::Arc::new(|_| None))
+            .build()
+            .unwrap();
+        assert!(
+            reader.reader_context.row_group_selector.is_some(),
+            "with_row_group_selector should land on reader_context"
+        );
+        assert!(
+            reader.reader_context.row_filter_builder.is_none(),
+            "the two mechanisms are independent: one may be set without the other"
+        );
+    }
+
+    /// Three rows, one per row group. A selector keeping only the first must
+    /// leave the read with that row group's row and no other -- and the volume
+    /// counters must show that the other two were never scanned, which is the
+    /// difference between pruning and filtering.
+    #[tokio::test]
+    async fn a_selector_prunes_row_groups_when_the_read_does_not_merge() {
+        use std::sync::atomic::Ordering::Relaxed;
+
+        let (tmp, base_name, schema) = three_row_groups();
+        let mut reader = test_file_group_reader_for_base_file(tmp.path(), &base_name, schema).await;
+        let volume = reader.storage.read_volume();
+        install_selector(&mut reader, |_| Some(vec![0]), false);
+
+        let out = drain_base_source(reader.base_file_source().await.unwrap()).await;
+
+        assert_eq!(out.num_rows(), 1, "only the kept row group was read");
+        assert_eq!(volume.row_group_selector_calls.load(Relaxed), 1);
+        assert_eq!(volume.row_group_selector_suppressed.load(Relaxed), 0);
+        assert_eq!(volume.file_row_groups.load(Relaxed), 3);
+        assert_eq!(
+            volume.row_groups_read.load(Relaxed),
+            1,
+            "the other two row groups were never fetched"
+        );
+    }
+
+    /// The same selector on a slice that merges, with a predicate that is not
+    /// primary-key-safe. Pruning would drop base rows before the merge could
+    /// update them into a match, so the gate refuses it -- and counts the
+    /// refusal, because a suppressed selector otherwise reads as "no caller ever
+    /// installed one": both are zero calls.
+    #[tokio::test]
+    async fn a_selector_the_gate_refuses_is_counted_not_silently_dropped() {
+        use std::sync::atomic::Ordering::Relaxed;
+
+        let (tmp, base_name, schema) = three_row_groups();
+        let mut reader = test_file_group_reader_for_base_file(tmp.path(), &base_name, schema).await;
+        let volume = reader.storage.read_volume();
+        reader.input_split = InputSplit::new(
+            Some(base_name.clone()),
+            Some("20240101120000000".to_string()),
+            vec![".f1-0_20240101130000000.log.1_0-1-1".to_string()],
+            String::new(),
+        );
+        install_selector(&mut reader, |_| Some(vec![0]), false);
+
+        let out = drain_base_source(reader.base_file_source().await.unwrap()).await;
+
+        assert_eq!(out.num_rows(), 3, "every base row still reaches the merge");
+        assert_eq!(
+            volume.row_group_selector_calls.load(Relaxed),
+            0,
+            "the selector never ran"
+        );
+        assert_eq!(
+            volume.row_group_selector_suppressed.load(Relaxed),
+            1,
+            "and the reason it never ran is on the record"
+        );
+        assert_eq!(volume.row_groups_read.load(Relaxed), 3);
+    }
+
+    /// The same merging slice with a primary-key-safe predicate: the gate opens,
+    /// so the selector runs. Pairs with the case above -- same file, same
+    /// selector, opposite outcome from `mor_pk_safe` alone.
+    #[tokio::test]
+    async fn a_pk_safe_predicate_lets_the_selector_run_on_a_merging_slice() {

Review Comment:
   non-blocking: these gate tests stop at base_file_source, and the one position-merge harness case has no selector, so the claim that RowNumber stays absolute under with_row_groups (which the pruning-under-merge safety rests on) is only guaranteed by parquet-rs internals today. An end-to-end case with a selector, log files, and use_record_position asserting exact merged rows would pin that invariant against future parquet upgrades.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(reader): merge-aware pushdown gate, row-group pruning, and read-volume counters [hudi-rs]

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #708:
URL: https://github.com/apache/hudi-rs/pull/708#issuecomment-5518415841

   ## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/708?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   :x: Patch coverage is `68.36735%` with `31 lines` in your changes missing coverage. Please review.
   :white_check_mark: Project coverage is 79.25%. Comparing base ([`71538f2`](https://app.codecov.io/gh/apache/hudi-rs/commit/71538f24b39bdd2a5cadd87c4b5f2527d9971e18?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b94b7b8`](https://app.codecov.io/gh/apache/hudi-rs/commit/b94b7b800362135ee7095b615abdc4b58dd5b4f2?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
   
   | [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/708?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
   |---|---|---|
   | [crates/core/src/file\_group/base\_file/parquet.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Fbase_file%2Fparquet.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvYmFzZV9maWxlL3BhcnF1ZXQucnM=) | 62.16% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [crates/core/src/file\_group/reader\_v2/engine.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fengine.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2VuZ2luZS5ycw==) | 51.72% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [crates/core/src/file\_group/base\_file/reader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Fbase_file%2Freader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvYmFzZV9maWxlL3JlYWRlci5ycw==) | 80.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   | [...es/core/src/file\_group/reader\_v2/reader\_context.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Freader_context.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL3JlYWRlcl9jb250ZXh0LnJz) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/708?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
   
   <details><summary>Additional details and impacted files</summary>
   
   
   
   ```diff
   @@            Coverage Diff             @@
   ##             main     #708      +/-   ##
   ==========================================
   - Coverage   79.34%   79.25%   -0.10%     
   ==========================================
     Files         122      122              
     Lines       11988    12069      +81     
   ==========================================
   + Hits         9512     9565      +53     
   - Misses       2476     2504      +28     
   ```
   </details>
   
   [:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/708?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).   
   :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   <details><summary> :rocket: New features to boost your workflow: </summary>
   
   - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org