You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "linliu-code (via GitHub)" <gi...@apache.org> on 2026/08/03 18:05:03 UTC
[PR] [REVIEW ONLY, DO NOT MERGE] feat(core): port the merge-on-read file group reader (#639-#659 squashed) [hudi-rs]
linliu-code opened a new pull request, #660:
URL: https://github.com/apache/hudi-rs/pull/660
> **Review only — please do not merge this.** It is #639–#659 squashed into a single diff, opened at a reviewer's request for a whole-picture read. The reviewable increments are those 21 PRs; merging should happen there. I'll close this once the review is done.
## What this is
The merge-on-read file group reader ported from `onehouseinc/hudi-rs-internal` into `hudi-core`, plus the switch that reaches it. ~31.8k lines across 59 files, 32 new modules.
**Nothing changes for existing users.** The ported reader is `pub(crate)` and reachable only via `hoodie.read.merge.engine = v2`, which defaults to `legacy` — the reader that has always served reads.
## How to read it
The one thing worth understanding first is the seam, because it explains why the other 20 PRs are shaped the way they are. `FileGroupReader` keeps its exact public API and becomes a dispatcher, with one decision point inside `read_file_slice_from_paths`:
```
read_file_slice_from_paths(base, logs, options)
├─ is_metadata_table() → existing reader (always, setting ignored)
├─ no log files / read-opt. → base-file read (nothing to merge)
├─ engine == legacy → existing reader (the default)
└─ engine == v2 → adapter → ported reader
│
all four ──┴──→ apply_eager_options(filters, projection)
```
`Table`, DataFusion, the Python binding and the cxx bridge are **untouched** — the ported reader was given a matching operation set precisely so the seam could sit here. Filters and projection stay out of the adapter so both engines run through the same `apply_eager_options` and cannot drift on what a filter means.
Three rules keep it safe:
1. **A metadata table always uses the existing reader**, whatever the setting says — its base files and log blocks are HFile, which the ported reader cannot read. Permanent, not transitional.
2. **Refusals, not fallbacks.** No resolvable schema → error. Unparseable engine name → error. `get_or_default` would have silently swallowed a typo and read with the *other* engine, leaving a caller convinced they'd tested v2.
3. **Base-file-only slices stay on the existing path** — not a capability limit, but the ported engine takes its schema from `hoodie.table.create.schema` where that path takes it from the base file, and the Avro conversion models a map as `Dictionary(Utf8, V)` (invalid Arrow). Every parquet fixture here has a map column.
## Corresponding PRs
| PR | New lines | What |
|---|---|---|
| #639 | +83/-12 | reader context, resolver, entry point |
| #640 | +457/-5 | stats, profiling, input-split leaves |
| #641 | +798 | delete context, output converter |
| #642 | +819 | Avro schema comparison, Parquet list normalization |
| #643 | +1767 | schema evolution for merged batches |
| #644 | +465 | row serialization, record positions |
| #645 | +1095 | buffered record |
| #646 | +1038 | record merger, update processor, converter |
| #647 | +3247 | record context, row extraction |
| #648 | +1377 | schema handler |
| #649 | +2399 | spillable merge map — adds `rocksdb` |
| #650 | +1948/-5 | record buffer, merge iterator |
| #651 | +427/-51 | replace reader context with the ported one |
| #652 | +9791/-1 | log record readers, key-based buffer |
| #653 | +1728/-17 | position-based merging, buffer loader |
| #654 | +1866 | the file group reader itself |
| #655 | +173/-31 | streaming reads |
| #656 | +52 | pins an existing-reader defect (independent, off `main`) |
| #657 | +1129/-25 | 19 fixtures vs Spark snapshots |
| #658 | +306/-193 | split metadata-table reading out |
| #659 | +498/-2 | the switch described above |
The order isn't arbitrary: the module import graph was computed, two dependency cycles removed upstream first, and the remainder topologically sorted into 9 layers so each PR compiles against only what already landed.
## Decisions that need a maintainer's call
**1. `hoodie.record.merge.mode` is never read.** The crate reads `hoodie.table.record.merge.strategy`, which Hudi never writes — zero occurrences across every fixture here. What tables actually write is `hoodie.record.merge.mode`, read nowhere. So the strategy is always *inferred* from whether an ordering field is set: correct for event-time tables, wrong for commit-time-ordered ones, which legitimately have none and so derive `append_only`.
**2. `append_only` silently drops deletes** (#656). It concatenates data batches and never consults `delete_batches`. Combined with (1): **a commit-time-ordered MOR table returns duplicate rows and ignores deletes today.** #656 pins this with a test rather than changing it, because the fix is a design choice.
**3. `rocksdb` as a required dependency** (#649) — the spill backend for the merge map. Bundles RocksDB 8.10, needs `libclang`. Licenses compatible.
**4. `base64` + `roaring`** (#644) — Hudi stores record positions as a base64 roaring bitmap, so this is the storage format's encoding rather than a design choice.
## Fixes to the existing read path
Three, all in shared `avro_to_arrow` / schema handling, so they fix the **current** reader too:
- **Decimals had no Arrow conversion.** A test *asserted* that failure as expected behavior; it now asserts the read.
- **Avro `timestamp-*` logical types lost their UTC zone** (the naive variants are `local-timestamp-*`), so a log batch could not be concatenated with the parquet base batch it was merged against.
- **`hoodie.table.create.schema` is stored properties-escaped** (`\:`) and one of its two consumers wasn't unescaping it.
## Known gaps — v2 is not at parity
Opt-in until it serves everything the existing reader serves.
**Blocks enablement:** schema supply for the standalone entry (`new_with_options` from a bare URI — what the cxx bridge uses); `append_only`, which has no v2 equivalent and is *derived* rather than opt-in, so ordinary tables reach it.
**Decide, don't necessarily bridge:** metadata table / HFile (recommendation: keep permanently separate — #658 makes that structural); the merge-mode semantics above, which also gate a differential harness, since until both engines derive semantics identically they legitimately disagree and the harness would report v2 being *correct* as a mismatch.
**Ported but unreachable:** position-based merge (the engine drops the row-number column the buffer needs) and predicate pushdown (`RowFilterBuilder` exists, nothing installs one). Neither blocks parity — the existing reader has neither — but they're advertised capabilities of the ported code.
**Fixture coverage: 6 of 15 match the Spark gold snapshots.** The other 9 fail on the **existing reader too**, verified by reading each through `Table::read` — shared gaps in Avro and log-format coverage, not port regressions: Avro maps as `Dictionary(Utf8, V)` ×3, the two delete-record ordering shapes ×6, no corrupt-block detection ×1, and empty `hoodie.table.partition.fields` failing validation above the reader. The gap list lives in an `#[ignore]`d test that **fails if a gap starts passing**, so fixes get promoted rather than going unnoticed.
## Status
Local: 1115 lib + 79 table-read + 39 datafusion + 21 + 12, all green; `cargo fmt --check` clean; build warning-free.
**CI has never run on any of these PRs** — `PR` reports `action_required`, `CI` and `Code` report `startup_failure`, identically on all of them. The same workflows succeed on fork PRs from committers, `main` hasn't moved, and every PR is `MERGEABLE`, so this looks like a workflow-approval gate rather than anything in the diffs. **A committer approving the runs would unblock it.** Local verification ran on rustc 1.97.1 while the repo pins 1.94, so `make check` (`-D warnings`) on the pinned toolchain is unverified.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788329720
##########
crates/core/src/file_group/reader_v2/resolver.rs:
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Derives a [`ReaderContext`] from a table's resolved configs.
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::error::ConfigError;
+use crate::config::read::HudiReadConfig;
+use crate::config::table::{BaseFileFormatValue, HudiTableConfig};
+use crate::error::CoreError;
+use crate::file_group::reader_v2::buffer::spillable_map;
+use crate::file_group::reader_v2::reader_context::{CONFIG_MERGE_TYPE, MergeMode, ReaderContext};
+use crate::file_group::reader_v2::record_context::RecordContext;
+use crate::file_group::reader_v2::schema_handler::FileGroupReaderSchemaHandler;
+use crate::timeline::selector::InstantRange;
+use std::collections::HashMap;
+
+/// Resolve the MOR reader context from `hudi_configs`, which the caller has
+/// already merged read options into.
+#[allow(dead_code)]
+pub(crate) fn resolve_reader_context(
+ hudi_configs: &HudiConfigs,
+ has_log_files: bool,
+) -> Result<ReaderContext> {
+ let table_path: String = hudi_configs.get(HudiTableConfig::BasePath)?.into();
+
+ // Resolved by `Table::prepare_reader_options` for table-level reads. A
+ // `FileGroupReader` built straight from a base URI never loads the timeline,
+ // so the caller must supply it; defaulting here would silently widen the read.
+ let latest_commit_time: String = hudi_configs
+ .try_get(HudiReadConfig::EndTimestamp)?
+ .ok_or_else(|| ConfigError::NotFound(HudiReadConfig::EndTimestamp.as_ref().to_string()))?
+ .into();
+
+ let merge_mode = resolve_merge_mode(hudi_configs)?;
Review Comment:
**Blocking:** this resolves the merge mode unconditionally, so a base-file-only read (COW or read-optimized) of a table declaring `CUSTOM` errors here even though `version_two_unsupported_reason` deliberately serves it — reproduced on a COW fixture with `hoodie.record.merge.mode=CUSTOM`, which fails every eager read under the default while the streaming path (returning early before the gate) serves it fine. Could this skip `resolve_merge_mode` when `has_log_files` is false, with an end-to-end test for the no-merge CUSTOM case? Also, the payload-class-inferred CUSTOM error doesn't name the `version=1` way back the way the gate's error does.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/src/file_group/log_file/content.rs:
##########
@@ -17,35 +17,128 @@
* under the License.
*/
use crate::Result;
-use crate::avro_to_arrow::arrow_array_reader::AvroArrowArrayReader;
use crate::config::HudiConfigs;
use crate::error::CoreError;
-use crate::file_group::log_file::avro::AvroDataBlockContentReader;
+use crate::file_group::log_file::avro::AvroBlockDecoder;
use crate::file_group::log_file::log_block::{
BlockMetadataKey, BlockType, LogBlockContent, LogBlockVersion,
};
use crate::file_group::log_file::log_format::LogFormatVersion;
use crate::file_group::record_batches::RecordBatches;
use crate::hfile::{HFileReader, HFileRecord};
-use crate::schema::delete::{avro_schema_for_delete_record, avro_schema_for_delete_record_list};
-use apache_avro::{Schema as AvroSchema, from_avro_datum};
+use crate::schema::delete::delete_record_list_schema_json;
+use crate::schema::extended_promotion::record_needs_rewrite_for_extended_promotion;
+use crate::schema::parquet_list_norm::normalize_parquet_metadata;
+use crate::schema::resolver::avro_json_to_arrow_schema;
+use crate::storage::RowFilterBuilder;
+use arrow_array::{Array, ArrayRef, ListArray, RecordBatch, StructArray, UnionArray};
+use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
-use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::file::metadata::ParquetMetaDataReader;
use std::collections::HashMap;
use std::io::{Read, Seek};
use std::sync::Arc;
+/// Turn the wrapped ordering values into a plain column.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records, so a decode
+/// against that schema yields a union of one-field structs. The merge wants the
+/// value itself.
+///
+/// A block writes one ordering type, so exactly one branch is populated; that
+/// branch's `value` child is the column. A block mixing branches is rejected
+/// rather than silently reduced to one of them.
+fn unwrap_ordering_values(ordering: &ArrayRef) -> Result<ArrayRef> {
+ let union = ordering
+ .as_any()
+ .downcast_ref::<UnionArray>()
+ .ok_or_else(|| {
+ CoreError::LogBlockError(format!(
+ "Expected orderingVal to be a union, got {}",
+ ordering.data_type()
+ ))
+ })?;
+
+ let mut active: Option<i8> = None;
+ for i in 0..union.len() {
+ let type_id = union.type_id(i);
+ match active {
+ None => active = Some(type_id),
+ Some(seen) if seen == type_id => {}
+ Some(seen) => {
+ return Err(CoreError::LogBlockError(format!(
+ "Delete block mixes ordering types (union branches {seen} and {type_id})"
+ )));
+ }
+ }
+ }
+ let Some(active) = active else {
+ return Ok(ordering.clone());
+ };
+
+ let child = union.child(active);
+ // Null is a branch like any other; there is nothing to unwrap out of it.
Review Comment:
**Non-blocking (follow-up):** collapsing the union to one column makes two Java-legal delete blocks fail the read — an all-null `orderingVal` decodes to a `DataType::Null` column that `delete_batch_to_keys_with_ordering` rejects loudly, and a block mixing wrapper types (e.g. one null among longs) errors above as "mixes ordering types". Java decodes each DeleteRecord independently and merges both fine, and a delete written without a precombine value is the most default-shaped delete there is. Worth a follow-up plus fixtures for both.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/src/file_group/reader_v2/resolver.rs:
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Derives a [`ReaderContext`] from a table's resolved configs.
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::error::ConfigError;
+use crate::config::read::HudiReadConfig;
+use crate::config::table::{BaseFileFormatValue, HudiTableConfig};
+use crate::error::CoreError;
+use crate::file_group::reader_v2::buffer::spillable_map;
+use crate::file_group::reader_v2::reader_context::{CONFIG_MERGE_TYPE, MergeMode, ReaderContext};
+use crate::file_group::reader_v2::record_context::RecordContext;
+use crate::file_group::reader_v2::schema_handler::FileGroupReaderSchemaHandler;
+use crate::timeline::selector::InstantRange;
+use std::collections::HashMap;
+
+/// Resolve the MOR reader context from `hudi_configs`, which the caller has
+/// already merged read options into.
+#[allow(dead_code)]
+pub(crate) fn resolve_reader_context(
+ hudi_configs: &HudiConfigs,
+ has_log_files: bool,
+) -> Result<ReaderContext> {
+ let table_path: String = hudi_configs.get(HudiTableConfig::BasePath)?.into();
+
+ // Resolved by `Table::prepare_reader_options` for table-level reads. A
+ // `FileGroupReader` built straight from a base URI never loads the timeline,
+ // so the caller must supply it; defaulting here would silently widen the read.
+ let latest_commit_time: String = hudi_configs
+ .try_get(HudiReadConfig::EndTimestamp)?
+ .ok_or_else(|| ConfigError::NotFound(HudiReadConfig::EndTimestamp.as_ref().to_string()))?
+ .into();
+
+ let merge_mode = resolve_merge_mode(hudi_configs)?;
+ let base_file_format = BaseFileFormatValue::resolve_from_configs(hudi_configs, None)?;
+ let instant_range = resolve_instant_range(hudi_configs)?;
+ let (table_config, hoodie_reader_config) = partition_configs(hudi_configs);
+
+ Ok(ReaderContext {
+ table_path,
+ latest_commit_time,
+ merge_mode: merge_mode.as_ref().to_string(),
+ base_file_format: base_file_format.as_ref().to_string(),
+ has_log_files,
+ instant_range: Some(instant_range),
+ table_config,
+ hoodie_reader_config,
+ should_merge_use_record_position: resolve_use_record_position(hudi_configs)?,
+ // Only one iterator mode is implemented.
+ iterator_mode: "ENGINE_RECORD".to_string(),
+ // Dispatch is on `merge_mode`; the strategy id is carried, not consulted.
+ merge_strategy_id: String::new(),
+ // No bootstrap support in this crate.
+ has_bootstrap_base_file: false,
+ needs_bootstrap_merge: false,
+ enable_logical_timestamp_field_repair: false,
+ // Predicate pushdown into the merge path has no caller here, so no
+ // filter is installed and the primary-key-safety gate is irrelevant.
+ row_filter_builder: None,
+ mor_pk_safe: false,
+ // The table-version < 8 completion gate needs a timeline the caller
+ // has not loaded; leaving it unset keeps the gate a no-op.
+ completion_gate_inputs: None,
Review Comment:
**Non-blocking (question):** `completion_gate_inputs` is `None` at every wired construction site, so the inflight/completion gate never runs and a multi-writer inflight append to an already-sliced log file reads as committed. Version 1 has the same gap so this isn't a regression, but the Pass-1 docs read as if the gate were wired — is populating this from `Table`'s timeline a planned follow-up?
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/src/file_group/reader_v2/log_record_reader.rs:
##########
@@ -0,0 +1,1805 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+
+//! Mirrors `org.apache.hudi.common.table.log.BaseHoodieLogRecordReader`.
+//!
+//! The 2-pass log scanning engine that reads log blocks, filters them through
+//! 5 gates, resolves compaction, and dispatches to the record buffer.
+//!
+//! ## 3-pass algorithm (matching Java's `scanInternal`):
+//!
+//! **Pass 1 — Forward scan with 5 gates:**
+//! - Gate 1: Corrupt → skip
+//! - Gate 2: Future (instant > latestInstantTime) → skip
+//! - Gate 3: Completed/inflight → skip uncommitted/inflight data/delete blocks (table version < 8;
+//! applied only when the completion sets are supplied)
+//! - Gate 4: Instant range filter → skip
+//! - Rollback command → remove target instant
+//! - Data/Delete → add to `instant_to_blocks_map`
+//!
+//! **Pass 2 — Reverse iteration with compaction resolution:**
+//! Iterate instants newest→oldest, `Collections.reverse(logBlocks)` then `addLast`.
+//! Result: deque has latest-first at head, earliest-last at tail.
+//!
+//! **Pass 3 — `processQueuedBlocksForInstant`:**
+//! Drain deque via `pollLast` (tail-first) = oldest instant processed first.
+
+use crate::Result;
+use crate::file_group::log_file::log_block::{BlockMetadataKey, BlockType, LogBlock};
+use crate::file_group::log_file::reader::LogFileReader;
+use crate::file_group::reader_v2::buffer::HoodieFileGroupRecordBuffer;
+use crate::file_group::reader_v2::profiling::profile_once;
+use crate::file_group::reader_v2::reader_context::{CompletionGateInputs, ReaderContext};
+use crate::storage::Storage;
+use crate::timeline::selector::InstantRange;
+use std::collections::{HashMap, HashSet, VecDeque};
+use std::sync::Arc;
+
+// =========================================================================
+// Pass 1 result
+// =========================================================================
+
+/// Output of Pass 1 (forward scan with 5 gates).
+#[derive(Debug)]
+pub struct Pass1Result {
+ /// Instants in first-seen order (no duplicates).
+ pub ordered_instants_list: Vec<String>,
+ /// instant time → blocks for that instant (in file-read order).
+ pub instant_to_blocks_map: HashMap<String, Vec<LogBlock>>,
+ /// Stats
+ pub total_log_blocks: u64,
+ pub total_corrupt_blocks: u64,
+ pub total_rollbacks: u64,
+}
+
+/// Borrowed view of [`CompletionGateInputs`] used inside a single scan pass.
+pub struct CompletionGate<'a> {
+ completed_instants: &'a HashSet<String>,
+ inflight_instants: &'a HashSet<String>,
+ archived_boundary: Option<&'a str>,
+}
+
+impl<'a> CompletionGate<'a> {
+ /// Build a borrowing gate from owned inputs.
+ pub fn new(inputs: &'a CompletionGateInputs) -> Self {
+ Self {
+ completed_instants: &inputs.completed_instants,
+ inflight_instants: &inputs.inflight_instants,
+ archived_boundary: inputs.archived_boundary.as_deref(),
+ }
+ }
+
+ /// Whether a data/delete block at `instant_time` is admitted (committed and not inflight).
+ ///
+ /// Mirrors `TimelineView::is_committed` (completed-set membership OR older than the archived
+ /// boundary) plus the explicit inflight exclusion, matching Java's scanInternalV1 filter.
+ ///
+ /// # No-positive-information fallback
+ ///
+ /// When the gate carries NO positive completion information — an empty
+ /// `completed_instants` set AND no `archived_boundary` — it cannot establish
+ /// committedness for any instant, so it admits everything (a no-op) rather
+ /// than excluding all blocks. Java always builds the gate from a non-empty
+ /// active timeline (`filterCompletedInstants()`), so an empty set here means
+ /// the inputs were not populated by the caller (e.g. the FFI bridge forwarded
+ /// an empty list). Excluding on that basis would silently drop EVERY log delta
+ /// — including committed ones — and return base-file-only data (the
+ /// C-INFLIGHT silent-wrong: a committed later delta wrongly dropped). Deferring
+ /// to the other gates preserves the pre-gate behavior for a mis-wired gate; a
+ /// correctly-populated gate is unaffected because its completed set is non-empty.
+ fn admits(&self, instant_time: &str) -> bool {
+ if self.is_unpopulated() {
+ return true;
+ }
+ let is_committed = self.completed_instants.contains(instant_time)
+ || self
+ .archived_boundary
+ .is_some_and(|boundary| instant_time < boundary);
+ is_committed && !self.inflight_instants.contains(instant_time)
+ }
+
+ /// Whether the gate carries NO positive completion information — an empty
+ /// `completed_instants` set AND no `archived_boundary`. Such a gate cannot establish
+ /// committedness for any instant, so [`admits`](Self::admits) degrades to admit-all.
+ ///
+ /// When the gate was supplied deliberately (the caller passes `Some` only for a table
+ /// version < 8 snapshot), being unpopulated is a mis-wire signal, not a genuine empty
+ /// timeline: Java always builds the gate from a non-empty active timeline
+ /// (`filterCompletedInstants()`), and a v1 snapshot always has >= 1 completed instant.
+ /// `forward_scan_pass1` warns once when this holds so a gate whose inputs were dropped
+ /// across the FFI boundary is diagnosable rather than a silent no-op.
+ fn is_unpopulated(&self) -> bool {
+ self.completed_instants.is_empty() && self.archived_boundary.is_none()
+ }
+}
+
+/// Run Pass 1: forward scan with 5 gates.
+///
+/// Processes a flat list of blocks (from all log files, in file-read order)
+/// and classifies them into `instant_to_blocks_map` + `ordered_instants_list`.
+///
+/// ## 5 Gates (per-block):
+/// 1. Corrupt → skip
+/// 2. Future (instant > latest_instant_time) → skip
+/// 3. Completed/inflight → skip data/delete blocks whose instant is uncommitted or inflight
+/// (C-INFLIGHT-DELTA). Applied only when `completion_gate` is `Some` — callers pass it for
+/// table version < 8 and leave it `None` for v8+ (and when no timeline is available), which
+/// preserves the prior behavior of admitting such blocks.
+/// 4. Instant range → skip if not in range
+/// 5. Rollback command → remove target instant from maps
+pub fn forward_scan_pass1(
+ all_blocks: Vec<LogBlock>,
+ latest_instant_time: &str,
+ instant_range: &Option<InstantRange>,
+ timezone: &str,
+ completion_gate: Option<&CompletionGate>,
+) -> Result<Pass1Result> {
+ let mut target_rollback_instants: HashSet<String> = HashSet::new();
+ let mut instant_to_blocks_map: HashMap<String, Vec<LogBlock>> = HashMap::new();
+ let mut ordered_instants_list: Vec<String> = Vec::new();
+ let mut total_log_blocks: u64 = 0;
+ let mut total_corrupt_blocks: u64 = 0;
+ let mut total_rollbacks: u64 = 0;
+
+ log::debug!(
+ "[Pass1] forward_scan: {} total blocks, latest_instant_time={}, has_instant_range={}",
+ all_blocks.len(),
+ latest_instant_time,
+ instant_range.is_some(),
+ );
+
+ // The completion gate was supplied (table version < 8 snapshot) but carries no positive
+ // completion info — its inputs were not populated across the FFI boundary. Rather than
+ // silently degrade to admit-all (Gate 3 becomes a no-op, so a straddling uncommitted delta
+ // could be merged), warn once so the mis-wire is diagnosable. Behavior is unchanged (still
+ // fail-open); this only makes the condition visible.
+ if completion_gate.is_some_and(CompletionGate::is_unpopulated) {
+ log::warn!(
+ "[Pass1] completion gate supplied but unpopulated (empty completed set, no archived \
+ boundary): Gate 3 (C-INFLIGHT) admits all delta blocks. A table version < 8 snapshot \
+ always has >= 1 completed instant, so this indicates the gate inputs were dropped \
+ across the FFI boundary. latest_instant_time={latest_instant_time}"
+ );
+ }
+
+ for block in all_blocks {
+ total_log_blocks += 1;
+
+ // Gate 1: Corrupt blocks → skip
+ if block.block_type == BlockType::Corrupted {
+ log::debug!("[Pass1] Gate1: corrupt block #{total_log_blocks} skipped");
+ total_corrupt_blocks += 1;
+ continue;
+ }
+
+ let instant_time = match block.instant_time() {
+ Ok(t) => t.to_string(),
+ Err(_) => {
+ log::debug!("[Pass1] block #{total_log_blocks} has no instant time, skipping");
+ continue;
+ }
+ };
+
+ // Gate 2: Future blocks → skip (instant > latestInstantTime)
+ if block.block_type != BlockType::Command && instant_time.as_str() > latest_instant_time {
+ log::debug!(
+ "[Pass1] Gate2: future block #{total_log_blocks} instant={instant_time} > {latest_instant_time}, skipped"
+ );
+ continue;
+ }
+
+ // Gate 3: Completed/inflight check (Java scanInternalV1, tableVersion < 8).
+ // Applied only when the caller supplies the timeline sets (None for v8+ or
+ // when no timeline is available). A data/delete block whose instant is not
+ // committed, or is inflight, is skipped -- without this an uncommitted
+ // "straddling" instant (time < latest completed, no in-log rollback block)
+ // is wrongly merged (C-INFLIGHT-DELTA). Command (rollback) blocks are never
+ // gated here so they can still remove their target instant below.
+ if block.block_type != BlockType::Command
+ && let Some(gate) = completion_gate
+ && !gate.admits(&instant_time)
+ {
+ log::debug!(
+ "[Pass1] Gate3: block #{total_log_blocks} instant={instant_time} uncommitted/inflight, skipped"
+ );
+ continue;
+ }
+
+ // Gate 4: Instant range filter (not for command blocks)
+ if block.block_type != BlockType::Command
+ && let Some(range) = instant_range
+ && range.not_in_range(&instant_time, timezone)?
+ {
+ log::debug!(
+ "[Pass1] Gate4: block #{total_log_blocks} instant={instant_time} out of range, skipped"
+ );
+ continue;
+ }
+
+ log::debug!(
+ "[Pass1] block #{total_log_blocks} passed all gates: type={:?} instant={instant_time}",
+ block.block_type,
+ );
+
+ // Classify the block
+ match block.block_type {
+ BlockType::AvroData | BlockType::ParquetData | BlockType::Delete => {
+ let blocks_list = instant_to_blocks_map
+ .entry(instant_time.clone())
+ .or_default();
+ if blocks_list.is_empty() {
+ ordered_instants_list.push(instant_time.clone());
+ }
+ blocks_list.push(block);
+ }
+ BlockType::Command if block.is_rollback_block() => {
+ total_rollbacks += 1;
+ if let Ok(target) = block.target_instant_time() {
+ let target = target.to_string();
+ log::debug!("[Pass1] ROLLBACK: removing instant={target}");
+ target_rollback_instants.insert(target.clone());
+ ordered_instants_list.retain(|t| t != &target);
+ instant_to_blocks_map.remove(&target);
+ }
+ }
+ _ => {}
+ }
+ }
+
+ log::debug!(
+ "[Pass1] complete: ordered_instants={ordered_instants_list:?} \
+ total_blocks={total_log_blocks} corrupt={total_corrupt_blocks} \
+ rollbacks={total_rollbacks}",
+ );
+ for (instant, blocks) in &instant_to_blocks_map {
+ log::debug!(
+ "[Pass1] instant={instant}: {} block(s) [{:?}]",
+ blocks.len(),
+ blocks
+ .iter()
+ .map(|b| format!("{:?}", b.block_type))
+ .collect::<Vec<_>>(),
+ );
+ }
+
+ Ok(Pass1Result {
+ ordered_instants_list,
+ instant_to_blocks_map,
+ total_log_blocks,
+ total_corrupt_blocks,
+ total_rollbacks,
+ })
+}
+
+// =========================================================================
+// Pass 2 result
+// =========================================================================
+
+/// Output of Pass 2 (reverse iteration with compaction resolution).
+#[derive(Debug)]
+pub struct Pass2Result {
+ /// The final ordered deque of blocks to be processed.
+ /// Latest instant at head (front), earliest at tail (back).
+ /// Drain via `pop_back` for oldest-first processing.
+ pub current_instant_log_blocks: VecDeque<LogBlock>,
+ /// Dedup guard — instants whose blocks are enqueued.
+ /// Recorded for parity with Java's scan result; nothing reads it back here.
+ #[allow(dead_code)]
+ pub instant_times_included: HashSet<String>,
+ /// Which instant times actually contributed blocks (ordered output).
+ pub valid_block_instants: Vec<String>,
+}
+
+/// Run Pass 2: reverse iteration with compaction resolution.
+///
+/// Iterates `ordered_instants_list` from newest→oldest, applying:
+/// - `Collections.reverse(logBlocks)` then `addLast` (Rust: `reverse()` + `push_back`)
+/// - Compaction: if block has `COMPACTED_BLOCK_TIMES` header, register mapping
+/// - Deduplication: `instant_times_included` prevents double-enqueue
+///
+/// ## Invariants (from FS_logFileReadInputs_fileOrderRequestTime.md):
+///
+/// **Invariant 1**: `instant_times_included` and `valid_block_instants` contain
+/// exactly the same instant strings (Set vs List).
+///
+/// **Invariant 2**: Every instant in `instant_times_included`/`valid_block_instants`
+/// has its blocks in `current_instant_log_blocks`, and vice versa.
+///
+/// **Invariant 3**: Each instant appears in `instant_times_included` at most once (dedup).
+///
+/// **Invariant 4**: `current_instant_log_blocks` is ordered latest-first (reverse chronological).
+/// Drain via `pop_back` produces oldest-first processing order.
+pub fn reverse_scan_pass2(pass1: &mut Pass1Result) -> Pass2Result {
+ log::debug!(
+ "[Pass2] reverse_scan: {} instants to process (newest→oldest)",
+ pass1.ordered_instants_list.len(),
+ );
+
+ let mut current_instant_log_blocks: VecDeque<LogBlock> = VecDeque::new();
+ let mut instant_times_included: HashSet<String> = HashSet::new();
+ let mut valid_block_instants: Vec<String> = Vec::new();
+ let mut block_time_to_compaction_block_time_map: HashMap<String, String> = HashMap::new();
+
+ for i in (0..pass1.ordered_instants_list.len()).rev() {
+ let instant_time = &pass1.ordered_instants_list[i];
+ let instants_blocks = match pass1.instant_to_blocks_map.get(instant_time) {
+ Some(blocks) if !blocks.is_empty() => blocks,
+ _ => continue,
+ };
+
+ let first_block = &instants_blocks[0];
+
+ // Check for compacted blocks (has COMPACTED_BLOCK_TIMES header)
+ if let Some(compacted_times) = first_block
+ .header
+ .get(&BlockMetadataKey::CompactedBlockTimes)
+ {
+ for original_instant in compacted_times.split(',') {
+ let original_instant = original_instant.trim().to_string();
+ let final_instant = block_time_to_compaction_block_time_map
+ .get(instant_time)
+ .cloned()
+ .unwrap_or_else(|| instant_time.clone());
+ block_time_to_compaction_block_time_map.insert(original_instant, final_instant);
+ }
+ } else {
+ // Normal data block — check if it's been compacted
+ let compacted_final = block_time_to_compaction_block_time_map.get(instant_time);
+
+ if let Some(final_instant) = compacted_final {
+ let final_instant = final_instant.clone();
+ if instant_times_included.contains(&final_instant) {
+ continue;
+ }
+ if let Some(blocks) = pass1.instant_to_blocks_map.get(&final_instant) {
+ let mut reversed = blocks.clone();
+ reversed.reverse();
+ for block in reversed {
+ current_instant_log_blocks.push_back(block);
+ }
+ }
+ instant_times_included.insert(final_instant.clone());
+ valid_block_instants.push(final_instant);
+ } else {
+ // Not compacted — add blocks directly
+ // Java: Collections.reverse(logBlocks) then forEach(addLast)
+ let blocks = pass1
+ .instant_to_blocks_map
+ .get(instant_time)
+ .cloned()
+ .unwrap_or_default();
+ let mut reversed = blocks;
+ reversed.reverse();
+ for block in reversed {
+ current_instant_log_blocks.push_back(block);
+ }
+ instant_times_included.insert(instant_time.clone());
+ valid_block_instants.push(instant_time.clone());
+ }
+ }
+ }
+
+ log::debug!(
+ "[Pass2] complete: {} blocks in deque, valid_instants={:?}",
+ current_instant_log_blocks.len(),
+ valid_block_instants,
+ );
+
+ Pass2Result {
+ current_instant_log_blocks,
+ instant_times_included,
+ valid_block_instants,
+ }
+}
+
+// =========================================================================
+// BaseHoodieLogRecordReader
+// =========================================================================
+
+/// The base log record reader implementing the 3-pass scanning algorithm.
+///
+/// Mirrors Java's `BaseHoodieLogRecordReader<T>` (abstract base class).
+///
+/// ## Java hierarchy:
+/// ```text
+/// BaseHoodieLogRecordReader<T> (abstract — this struct)
+/// └─ HoodieMergedLogRecordReader<T> (concrete — wraps this via composition)
+/// ```
+///
+/// ## Fields matching Java:
+/// - `reader_context` ↔ `readerContext`
+/// - `storage` ↔ `storage`
+/// - `log_file_paths` ↔ `logFiles`
+/// - `latest_instant_time` ↔ `latestInstantTime`
+/// - `instant_range` ↔ `instantRange`
+/// - `force_full_scan` ↔ `forceFullScan`
+/// - `record_buffer` ↔ `recordBuffer`
+/// - `allow_inflight_instants` ↔ `allowInflightInstants`
+pub struct BaseHoodieLogRecordReader {
+ pub reader_context: Arc<ReaderContext>,
+ pub storage: Arc<Storage>,
+ pub log_file_paths: Vec<String>,
+ pub latest_instant_time: String,
+ /// Mirrors Java's `private final Option<InstantRange> instantRange`.
+ pub instant_range: Option<InstantRange>,
+ /// Mirrors Java's `protected final boolean forceFullScan`.
+ /// When true, scanning happens eagerly in the constructor.
+ pub force_full_scan: bool,
+ pub record_buffer: Box<dyn HoodieFileGroupRecordBuffer>,
+ /// Carried for parity. Nothing reads it, so it is inert — the completion gate
+ /// below is what actually decides whether an inflight instant is admitted,
+ /// and `ReaderParameters` never sets this true.
+ #[allow(dead_code)]
+ pub allow_inflight_instants: bool,
+ /// Inputs for the Gate-3 completed/inflight check (C-INFLIGHT-DELTA). `Some` only for
+ /// table version < 8 (v1 timeline layout); `None` for v8+ and when no timeline is available,
+ /// in which case Gate 3 is a no-op. Populated by the builder / FFI wiring from the active
+ /// timeline (completed + inflight instant sets + the first active instant).
+ pub completion_gate_inputs: Option<CompletionGateInputs>,
+
+ // ── Stats / state (mirrors Java's AtomicLong counters + progress) ──
+ pub valid_block_instants: Vec<String>,
+ pub total_log_files: u64,
+ pub total_log_blocks: u64,
+ pub total_log_records: u64,
+ pub total_corrupt_blocks: u64,
+ pub total_rollbacks: u64,
+ /// Mirrors Java's `private float progress` (0.0 → 1.0).
+ pub progress: f32,
+
+ // ── Stage timings (perf harness) ──────────────────────
+ /// Wall ms spent reading log-block metadata + bytes off storage (Pass 1).
+ pub log_block_read_ms: u64,
+ /// Wall ms spent in Pass-3 block dispatch (inflate/decode + merge insert).
+ pub merge_insert_ms: u64,
+}
+
+impl BaseHoodieLogRecordReader {
+ /// Mirrors Java's `scanInternal(Option<KeySpec> keySpecOpt, boolean skipProcessingBlocks)`.
+ ///
+ /// # Arguments
+ /// - `skip_processing_blocks` — when `true`, Pass 3 (block processing) is skipped.
+ /// Java's `HoodieMergedLogRecordReader.scan(true)` uses this to collect block
+ /// metadata without actually merging records.
+ pub async fn scan_internal(&mut self, skip_processing_blocks: bool) -> Result<()> {
+ // Reset state (mirrors Java: currentInstantLogBlocks = new ArrayDeque<>(), progress = 0.0f, ...)
+ self.valid_block_instants.clear();
+ self.progress = 0.0;
+ self.total_log_files = 0;
+ self.total_log_blocks = 0;
+ self.total_log_records = 0;
+ self.total_corrupt_blocks = 0;
+ self.total_rollbacks = 0;
+
+ let timezone = self.reader_context.timezone();
+
+ // Read all blocks as metadata-only (no content decoding).
+ // Mirrors Java's Pass 1 where content bytes are skipped via seek().
+ // Content is loaded lazily via inflate() during Pass 3.
+ let mut all_blocks: Vec<LogBlock> = Vec::new();
+ let hudi_configs = Arc::new(crate::config::HudiConfigs::new(
+ self.reader_context.table_config.clone(),
+ ));
+ let unbounded_range =
+ InstantRange::new(self.reader_context.timezone(), None, None, false, true);
+ profile_once!(self.log_block_read_ms, {
+ for path in &self.log_file_paths.clone() {
+ // A5 / R5: stream the log file in bounded windows for the Pass-1
+ // header sweep instead of buffering the whole file. Each block
+ // gets a ranged fetcher so Pass-3 inflate reads only its content.
+ // Upstream sweeps block headers in bounded windows and inflates
+ // each block's content later. This crate's log file reader is
+ // whole-file, so the blocks come back with their content already
Review Comment:
**Non-blocking (follow-up):** the wired scan decodes every block's content before the gates run, so a decode failure inside a block the gates would discard (e.g. a rolled-back instant's block) fails the read, and a whole slice's decoded batches sit in memory before merging (the merge map spills; this vec doesn't). The streaming sweep that fixes both exists but is only called from `memory_bench.rs` — is wiring it a planned follow-up?
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/src/file_group/reader_v2/buffer/key_based.rs:
##########
@@ -0,0 +1,7459 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+
+//! Mirrors `org.apache.hudi.common.table.read.buffer.KeyBasedFileGroupRecordBuffer`.
+//!
+//! The default record buffer. Deduplicates by record key using a HashMap,
+//! resolves conflicts via ordering value comparison (delta merge).
+//!
+//! ## How it's selected (in `DefaultFileGroupRecordBufferLoader`):
+//! ```text
+//! !is_skip_merge && !sort_outputs && !(use_record_position && base_file_present)
+//! → KeyBasedFileGroupRecordBuffer (DEFAULT)
+//! ```
+//!
+//! ## Method mapping (Java → Rust):
+//! - `processDataBlock` → `process_data_block`
+//! - `processNextDataRecord` → `process_next_data_record`
+//! - `processDeleteBlock` → `process_delete_block`
+//! - `processNextDeletedRecord` → `process_next_deleted_record`
+//! - `doHasNext` → `do_has_next`
+//! - `hasNextBaseRecord` → `has_next_base_record`
+
+use crate::Result;
+use crate::file_group::log_file::log_block::{LogBlock, LogBlockContent};
+use crate::file_group::reader_v2::buffer::record_buffer::FileGroupRecordBuffer;
+use crate::file_group::reader_v2::buffer::record_positions::base_row_position_array;
+use crate::file_group::reader_v2::buffer::row_extraction::{
+ reconcile_batch_to_schema, records_to_batch,
+};
+use crate::file_group::reader_v2::buffer::spillable_map::{
+ CONFIG_MERGE_MAX_SIZE, MergeMap, SpillConfig, SpillableRecordMap,
+};
+use crate::file_group::reader_v2::buffer::{BufferType, HoodieFileGroupRecordBuffer};
+use crate::file_group::reader_v2::buffered_record::{
+ BufferedRecord, BufferedRecords, DeleteRecord, OrderingValue,
+};
+use crate::file_group::reader_v2::merge_iterator::DEFAULT_BATCH_SIZE;
+// `CoreError` / `RecordPayload` are used only by the test-only compaction
+// primitive and the unit tests (A6e moved production compaction onto the map).
+#[cfg(test)]
+use crate::error::CoreError;
+#[cfg(test)]
+use crate::file_group::reader_v2::buffered_record::RecordPayload;
+use crate::file_group::reader_v2::reader_context::ReaderContext;
+use crate::file_group::reader_v2::record_context::RecordContext;
+use crate::file_group::reader_v2::record_merger::{
+ BufferedRecordMergerFactory, should_keep_newer_record,
+};
+use crate::file_group::reader_v2::update_processor::{UpdateStats, create_update_processor};
+use arrow_array::{
+ Array, ArrayRef, BinaryArray, BooleanArray, Int32Array, Int64Array, LargeBinaryArray,
+ LargeStringArray, RecordBatch, StringArray,
+};
+use arrow_buffer::BooleanBufferBuilder;
+use arrow_schema::DataType;
+use arrow_schema::{Schema, SchemaRef};
+use std::collections::HashMap;
+use std::sync::Arc;
+
+// The compaction live-row threshold is a single source of truth in
+// `spillable_map` (A6e shares it between over-budget eviction and the
+// end-of-scan safety valve). Re-imported here for the free-function compaction
+// primitive's tests and the trait delegate.
+use crate::file_group::reader_v2::buffer::spillable_map::COMPACTION_LIVE_RATIO;
+
+#[cfg(test)]
+use crate::config::table::HudiTableConfig;
+
+/// Key-based file group record buffer.
+///
+/// Mirrors Java's `KeyBasedFileGroupRecordBuffer<T>`.
+///
+/// ## Per-record merge during log scanning:
+/// ```text
+/// processDataBlock(batch):
+/// for each row → processNextDataRecord(record, key)
+/// → records.get(key) → deltaMerge(new, existing) → records.put(key, merged)
+///
+/// processDeleteBlock(batch):
+/// for each row → processNextDeletedRecord(deleteRecord, key)
+/// → records.get(key) → deltaMergeDelete(delete, existing) → records.put(key, merged)
+/// ```
+///
+/// ## Base-vs-log merge at read time:
+/// ```text
+/// doHasNext():
+/// while baseIterator.hasNext():
+/// if hasNextBaseRecord(base): return true
+/// return hasNextLogRecord()
+///
+/// hasNextBaseRecord(base):
+/// key = extractKey(base)
+/// logRecord = records.remove(key)
+/// return super.hasNextBaseRecord(base, logRecord)
+/// ```
+/// How a base-file row is matched to a buffered log record during the merge.
+///
+/// The merge kernels are identical except for the per-row lookup key: key-based
+/// merge looks up the row's record key; position-based merge looks up the row's
+/// physical base-file position (read from the synthetic row-index column). This
+/// enum lets [`KeyBasedFileGroupRecordBuffer`]'s kernels serve both, so the
+/// position buffer reuses them rather than duplicating the vectorized/scalar
+/// merge logic.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub(super) enum BaseMatch {
+ /// Match by record key (key-based merge).
+ RecordKey,
+ /// Match by base-file row position, read from the row-index column
+ /// ([`ROW_INDEX_TEMPORARY_COLUMN_NAME`](super::record_positions::ROW_INDEX_TEMPORARY_COLUMN_NAME)).
+ Position,
+}
+
+#[derive(Debug)]
+pub struct KeyBasedFileGroupRecordBuffer {
+ /// Common buffer state.
+ pub base: FileGroupRecordBuffer,
+
+ /// Reader context (mirrors Java's readerContext).
+ pub reader_context: Arc<ReaderContext>,
+
+ /// Record context for record-level operations (key extraction, ordering, etc.).
+ /// Mirrors Java's `readerContext.getRecordContext()`.
+ pub record_context: RecordContext,
+
+ /// Partial-update (`FILL_UNAVAILABLE`) sentinel. `Some(v)` when the table's
+ /// payload uses a Debezium-style "unavailable value" marker (e.g. Postgres
+ /// TOAST: `__debezium_unavailable_value`): a winning log column equal to `v`
+ /// is replaced with the prior base value during merge. `None` disables the
+ /// rule (the common case) so non-partial-update reads pay nothing. Read from
+ /// `hoodie.record.merge.property.partial.update.unavailable.value`.
+ unavailable_value: Option<String>,
+
+ /// Partial-update `IGNORE_DEFAULTS` flag. `true` when the table's
+ /// `hoodie.table.partial.update.mode` is `IGNORE_DEFAULTS` (e.g. a table
+ /// written with `PartialUpdateAvroPayload`): a winning full-schema log
+ /// column whose value equals the field's declared default keeps the prior
+ /// (base/older) value during merge instead of overwriting with the default.
+ /// `false` (the common case) disables the rule so non-partial-update reads
+ /// pay nothing. Mirrors Java `PartialUpdateHandler.reconcileDefaultValues`.
+ ignore_defaults: bool,
+
+ /// Per-field IGNORE_DEFAULTS retention rule, keyed by field name, derived
+ /// from the table's declared Avro field defaults (see [`DefaultRetain`]).
+ /// Populated only when [`Self::ignore_defaults`] is set and the Avro schema
+ /// JSON is available; empty otherwise. A field absent from the map falls
+ /// back to [`DefaultRetain::OnNull`] (the behavior for a field with no
+ /// declared default).
+ default_retains: HashMap<String, DefaultRetain>,
+}
+
+/// Per-field IGNORE_DEFAULTS retention rule, mirroring Java
+/// `PartialUpdateHandler.reconcileDefaultValues`, whose
+/// `toJavaDefaultValue(field) == newValue` is a *reference* equality (`==`) on
+/// the boxed values `RecordContext.getValue` returns:
+/// - no default, or a null default → retain the prior value when the incoming
+/// value is NULL (`null == null`);
+/// - an integer-family default within the JVM Integer/Long cache
+/// (`-128..=127`) → retain when the incoming value equals it (autoboxing
+/// reuses the cached instance only inside that range, so `==` holds);
+/// - a boolean default → retain when the incoming value equals it (`Boolean`
+/// boxes are always cached);
+/// - anything else — string/bytes/float/double/enum/record/array/map, or an
+/// out-of-cache integer → never retain, because Java's `==` compares two
+/// distinct boxed objects and is always false.
+#[derive(Clone, Debug, PartialEq)]
+enum DefaultRetain {
+ /// Retain the prior value when the incoming value is null.
+ OnNull,
+ /// Retain when the incoming integer value equals this (cache-range) default.
+ OnInt(i64),
+ /// Retain when the incoming boolean value equals this default.
+ OnBool(bool),
+ /// Never retain — the default can't be matched by Java's reference `==`.
+ Never,
+}
+
+/// Reader-config key carrying the Debezium-style partial-update "unavailable
+/// value" sentinel (set by Hudi's `HoodieTableConfig` for payloads like
+/// `PostgresDebeziumAvroPayload`). Mirrors Java's
+/// `RECORD_MERGE_PROPERTY_PREFIX + "partial.update.unavailable.value"`.
+const PARTIAL_UPDATE_UNAVAILABLE_VALUE: &str =
+ "hoodie.record.merge.property.partial.update.unavailable.value";
+
+/// Table-config key carrying the partial-update mode
+/// (`hoodie.table.partial.update.mode`; set by Hudi's `HoodieTableConfig`, e.g.
+/// for `PartialUpdateAvroPayload`). When its value is
+/// [`PARTIAL_UPDATE_MODE_IGNORE_DEFAULTS`], a winning full-schema log column
+/// whose value is the field default keeps the prior value during merge. Mirrors
+/// Java `HoodieTableConfig.PARTIAL_UPDATE_MODE`.
+const PARTIAL_UPDATE_MODE: &str = "hoodie.table.partial.update.mode";
+
+/// `PARTIAL_UPDATE_MODE` value selecting the IGNORE_DEFAULTS blend (a default
+/// column in the incoming record retains the prior value). Mirrors Java
+/// `PartialUpdateMode.IGNORE_DEFAULTS`.
+const PARTIAL_UPDATE_MODE_IGNORE_DEFAULTS: &str = "IGNORE_DEFAULTS";
+
+/// Name prefix of Hudi's writer-managed meta columns (`_hoodie_record_key`,
+/// `_hoodie_commit_time`, ...). Used to exempt meta columns from the
+/// partial-update blends.
+const HOODIE_META_COLUMN_PREFIX: &str = "_hoodie_";
+
+/// True if `name` is a record-key column (member of `key_field_names`) or a
+/// `_hoodie_`-prefixed meta column — the columns the partial-update blends
+/// ([`fill_unavailable_from_base`] / [`reconcile_defaults_from_prior`]) must
+/// never substitute from the prior row.
+fn is_key_or_meta_column(name: &str, key_field_names: &[String]) -> bool {
+ name.starts_with(HOODIE_META_COLUMN_PREFIX) || key_field_names.iter().any(|k| k == name)
+}
+
+impl KeyBasedFileGroupRecordBuffer {
+ /// Fill a partial-update merge WINNER's default/unavailable columns from a
+ /// `source` row (the loser or the prior record), dispatching on the table's
+ /// partial-update mode: IGNORE_DEFAULTS → [`reconcile_defaults_from_prior`],
+ /// FILL_UNAVAILABLE → [`fill_unavailable_from_base`]. Mirrors Java
+ /// `PartialUpdateHandler.partialMerge`. A no-op (`Ok((target, false))`) when
+ /// neither mode is configured. `source_idx` selects the row in `source`
+ /// (log-vs-log passes a single-row batch at index 0).
+ ///
+ /// Returns the (possibly rewritten) record and a `bool` that is `true` only
+ /// when a default/sentinel column was actually substituted — callers on the
+ /// base-vs-log kept-row fast path use it to avoid materialising an unchanged
+ /// replacement.
+ fn fill_partial(
+ &self,
+ target: BufferedRecord,
+ source: &RecordBatch,
+ source_idx: usize,
+ ) -> Result<(BufferedRecord, bool)> {
+ if self.ignore_defaults {
+ reconcile_defaults_from_prior(
+ target,
+ source,
+ source_idx,
+ &self.default_retains,
+ &self.record_context.record_key_fields,
+ )
+ } else if let Some(sentinel) = self.unavailable_value.as_deref() {
+ fill_unavailable_from_base(
+ target,
+ source,
+ source_idx,
+ sentinel,
+ &self.record_context.record_key_fields,
+ )
+ } else {
+ Ok((target, false))
+ }
+ }
+
+ pub fn new(
+ reader_context: Arc<ReaderContext>,
+ merge_mode: String,
+ emit_delete: bool,
+ ) -> crate::Result<Self> {
+ let merger = BufferedRecordMergerFactory::create(&merge_mode)?;
+ let update_processor = create_update_processor(emit_delete);
+ // Get the shared RecordContext from ReaderContext (mirrors Java's
+ // readerContext.getRecordContext() returning the same instance).
+ let record_context = reader_context.get_record_context().clone();
+
+ // Extract reader_schema from schema_handler.required_schema at
+ // construction time. Mirrors Java's FileGroupRecordBuffer constructor:
+ // this.readerSchema = readerContext.getSchemaHandler().getRequiredSchema()
+ let reader_schema = reader_context.schema_handler.required_schema.clone();
+
+ // Get the canonical DeleteContext from the schema handler (single source
+ // of truth). Mirrors Java's `readerContext.getSchemaHandler().getDeleteContext()`.
+ //
+ // The schema handler creates the DeleteContext during
+ // `prepare_required_schema()` and stores it. HoodieFileGroupReader
+ // propagates the prepared schema_handler onto reader_context before
+ // the buffer is created, so `reader_context.schema_handler.delete_context()`
+ // always returns the canonical instance.
+ //
+ // Ordering invariant: the only construction path runs
+ // `prepare_required_schema` first, so a `None` here means an upstream
+ // wiring bug. This constructor already returns `Result`, so surface it
+ // as a typed error rather than panicking in library code.
+ let delete_context = reader_context
+ .schema_handler
+ .delete_context()
+ .cloned()
+ .ok_or_else(|| {
+ crate::error::CoreError::ReadFileSliceError(
+ "DeleteContext must be set on schema_handler by prepare_required_schema() \
+ before constructing the record buffer"
+ .to_string(),
+ )
+ })?;
+
+ // Enrich DeleteContext with reader schema at construction time.
+ // Mirrors Java's: deleteContext = readerContext.getSchemaHandler()
+ // .getDeleteContext().withReaderSchema(this.readerSchema)
+ let delete_context = if let Some(ref schema) = reader_schema {
+ delete_context.with_reader_schema(schema.clone())
+ } else {
+ delete_context
+ };
+
+ // Build the size-tracked, RocksDB-spillable merge map from the
+ // reader config. Parses hoodie.memory.merge.max.size / spillable.map.path
+ // / spillable.diskmap.type (BITCASK and ROCKS_DB both → RocksDB backend).
+ let spill_config = SpillConfig::from_config(&reader_context.hoodie_reader_config)?;
+ // Diagnostic: the resolved in-memory spill threshold and whether the
+ // merge-memory budget key reached this map. gluten now forwards
+ // `hoodie.memory.merge.max.size` in hoodie_reader_config, so `budget_key_present=true`
+ // and the threshold tracks the operator/computed budget; a `false` here means the budget
+ // was dropped and the threshold fell back to the 0.8×1 GiB default (OOM risk).
+ log::debug!(
+ "[hudi-rs-spill] SpillConfig resolved: max_in_memory_size={} bytes, \
+ budget_key_present={} (key '{}')",
+ spill_config.max_in_memory_size,
+ reader_context
+ .hoodie_reader_config
+ .contains_key(CONFIG_MERGE_MAX_SIZE),
+ CONFIG_MERGE_MAX_SIZE,
+ );
+ let records = SpillableRecordMap::with_config(spill_config);
+
+ let mut base = FileGroupRecordBuffer::new(
+ merge_mode,
+ merger,
+ update_processor,
+ reader_schema,
+ records,
+ );
+ base.delete_context = Some(delete_context);
+
+ // Partial-update (FILL_UNAVAILABLE) sentinel, if the table's payload
+ // configures one (Debezium toasted/unavailable values). Checked in the
+ // reader config first, then the table config; empty → disabled.
+ let unavailable_value = reader_context
+ .hoodie_reader_config
+ .get(PARTIAL_UPDATE_UNAVAILABLE_VALUE)
+ .or_else(|| {
+ reader_context
+ .table_config
+ .get(PARTIAL_UPDATE_UNAVAILABLE_VALUE)
+ })
+ .filter(|v| !v.is_empty())
+ .cloned();
+
+ // Partial-update IGNORE_DEFAULTS flag, if the table configures it (e.g.
+ // PartialUpdateAvroPayload). Reader config first, then table config —
+ // same precedence as the unavailable-value sentinel above.
+ let ignore_defaults = reader_context
+ .hoodie_reader_config
+ .get(PARTIAL_UPDATE_MODE)
+ .or_else(|| reader_context.table_config.get(PARTIAL_UPDATE_MODE))
+ .is_some_and(|v| v.eq_ignore_ascii_case(PARTIAL_UPDATE_MODE_IGNORE_DEFAULTS));
+
+ // Per-field IGNORE_DEFAULTS retention rules from the table's declared Avro
+ // field defaults. Prefer the data (writer/table) schema JSON, which carries
+ // the declared defaults; fall back to the required-schema JSON. Only parsed
+ // when the mode is active; a parse failure / absent JSON yields an empty map
+ // (every field then falls back to OnNull, the no-declared-default behavior).
+ let default_retains = if ignore_defaults {
+ reader_context
+ .schema_handler
+ .data_schema_json
+ .as_deref()
+ .or(reader_context.schema_handler.reader_schema_json.as_deref())
+ .map(parse_default_retains)
+ .unwrap_or_default()
+ } else {
+ HashMap::new()
+ };
+
+ Ok(Self {
+ base,
+ reader_context,
+ record_context,
+ unavailable_value,
+ ignore_defaults,
+ default_retains,
+ })
+ }
+
+ /// Mirrors Java's `KeyBasedFileGroupRecordBuffer.hasNextBaseRecord(T baseRecord)`.
+ ///
+ /// Looks up the record key in the records map, removes it if found,
+ /// then delegates to `FileGroupRecordBuffer.has_next_base_record(base, log)`.
+ pub(super) fn has_next_base_record_at(
+ &mut self,
+ base_batch: &Arc<RecordBatch>,
+ row_idx: usize,
+ base_match: BaseMatch,
+ ) -> Result<bool> {
+ // Key extraction needs a single-row view; slicing here is cheap and the
+ // slice is discarded after key extraction (the stored base record is a
+ // zero-copy BatchRef into the shared batch, not this slice).
+ let row = base_batch.slice(row_idx, 1);
+ let keys = self.record_context.get_record_keys(&row)?;
+ // The base record always carries its record key (needed for update
+ // stats and any key-based fallback). The map LOOKUP key is the record
+ // key (key-based) or the row's physical position (position-based).
+ let record_key = keys[0].clone();
+ let lookup_key = match base_match {
+ BaseMatch::RecordKey => record_key.clone(),
+ BaseMatch::Position => base_row_position_array(base_batch)?
+ .value(row_idx)
+ .to_string(),
+ };
+ let log_record = self.base.records.remove(&lookup_key)?;
+
+ // Base-record ordering value: EVENT_TIME final_merge compares
+ // base vs log by ordering value, so the base record must carry it —
+ // extracted from the same single-row slice used for the key, the same way
+ // log records get theirs (record_context.get_ordering_values, used by
+ // batch_to_buffered_records). COMMIT_TIME's merger ignores it, so this is
+ // safe (and cheap: single-row) for both modes.
+ let base_ordering = self
+ .record_context
+ .get_ordering_values(&row)?
+ .and_then(|mut v| v.drain(..).next().flatten());
+
+ // A2: base record is a zero-copy BatchRef into the shared base batch,
+ // removing the per-row `base_row.clone()` allocation.
+ let base_record =
+ BufferedRecord::new_batch_ref(record_key, base_batch.clone(), row_idx, base_ordering);
+
+ self.base
+ .has_next_base_record(&base_record, log_record.as_ref())
+ }
+
+ /// Mirrors Java's `KeyBasedFileGroupRecordBuffer.doHasNext()`, parameterized
+ /// by how base rows are matched to log records ([`BaseMatch`]).
+ ///
+ /// First iterates base file records, merging with log records.
+ /// Then iterates remaining log-only records.
+ pub(super) fn do_has_next_matched(&mut self, base_match: BaseMatch) -> Result<bool> {
+ // Handle merging: iterate base file rows. `?` propagates a base source
+ // error instead of ending iteration early (which would drop rows).
+ while let Some((base_batch, row_idx)) = self.base.next_base_row()? {
+ if self.has_next_base_record_at(&base_batch, row_idx, base_match)? {
+ return Ok(true);
+ }
+ }
+
+ // Handle records solely from log files
+ self.base.has_next_log_record()
+ }
+
+ /// Vectorized base-batch pull + merge, parameterized by [`BaseMatch`]. The
+ /// key-based trait entry point calls this with [`BaseMatch::RecordKey`]; the
+ /// position buffer calls it with [`BaseMatch::Position`].
+ pub(super) fn pull_and_merge_next_base_batch(
+ &mut self,
+ target_schema: &SchemaRef,
+ base_match: BaseMatch,
+ ) -> Result<Option<RecordBatch>> {
+ loop {
+ let source = match self.base.base_file_source.as_mut() {
+ Some(s) => s,
+ None => return Ok(None),
+ };
+ let base_batch = match source.next() {
+ None => {
+ self.base.base_file_source = None;
+ return Ok(None);
+ }
+ Some(Ok(b)) => b,
+ Some(Err(e)) => {
+ self.base.base_file_source = None;
+ log::error!(
+ "[KeyBasedBuffer] base file source error in \
+ pull_and_merge_next_base_batch: {e}"
+ );
+ return Err(crate::error::CoreError::ReadFileSliceError(format!(
+ "base file source error: {e}"
+ )));
+ }
+ };
+ if base_batch.num_rows() == 0 {
+ continue; // skip empty source batches
+ }
+ return self.merge_one_base_batch_kernel(&base_batch, target_schema, base_match);
+ }
+ }
+
+ /// Vectorized merge of one base RecordBatch against the in-buffer log map.
+ ///
+ /// Walks the keys once, builds a `BooleanArray` keep-mask + a sparse
+ /// `Vec<BufferedRecord>` of log-winning replacements. Returns the kept
+ /// base rows (filtered in one Arrow kernel call) concatenated with the
+ /// replacement rows (decoded via `records_to_batch`).
+ ///
+ /// Per-row allocation is paid only on the rows that actually have a log
+ /// entry — the no-conflict path is one bit and no allocation. This is
+ /// the hot path that replaces the legacy
+ /// `next_base_row` + per-row `has_next_base_record` + 4096-batch
+ /// `concat_batches` chain in `next_get_next_us`.
+ ///
+ /// Duplicate-key semantics match the legacy path: `HashMap::remove`
+ /// consumes the log entry on the first base row whose key matches;
+ /// subsequent duplicates within the same base batch find `None` and
+ /// pass through. (Duplicate base keys within a single file group are
+ /// already a writer invariant violation.)
+ pub(super) fn merge_one_base_batch_kernel(
+ &mut self,
+ base: &RecordBatch,
+ target_schema: &SchemaRef,
+ base_match: BaseMatch,
+ ) -> Result<Option<RecordBatch>> {
+ let num_rows = base.num_rows();
+ if num_rows == 0 {
+ return Ok(None);
+ }
+
+ // Per-row lookup key source. Key-based: borrow keys as `&str` straight
+ // from the contiguous key column (no per-row `String` alloc).
+ // Position-based: read the row's physical position from the row-index
+ // column and use its decimal string as the map key. The ordering column
+ // is resolved once; ordering values are built lazily, only for the rare
+ // conflict rows.
+ let key_array = match base_match {
+ BaseMatch::RecordKey => Some(self.record_context.record_key_array(base)?),
+ BaseMatch::Position => None,
+ };
+ let position_array = match base_match {
+ BaseMatch::Position => Some(base_row_position_array(base)?),
+ BaseMatch::RecordKey => None,
+ };
+ let orderings = self.record_context.ordering_accessor(base)?;
+
+ // Bit-packed keep-mask (1 bit/row) built directly, vs a `Vec<bool>`
+ // (1 byte/row) that arrow would then re-pack into a bitmap.
+ let mut keep = BooleanBufferBuilder::new(num_rows);
+ let mut replacements: Vec<BufferedRecord> = Vec::new();
+ // Tracks whether any base row is actually dropped; drives the all-keep
+ // short-circuit below.
+ let mut any_dropped = false;
+ let merge_mode = self.base.record_merge_mode.as_str();
+ // Partial-update sentinel (cloned once per batch; `None` on the common
+ // path so the hot loop skips the toasted-value blend entirely).
+ let unavailable_value = self.unavailable_value.clone();
+ let ignore_defaults = self.ignore_defaults;
+ let default_retains = self.default_retains.clone();
+ // Record-key field name(s) — the columns the partial-update blends skip.
+ let key_field_names = self.record_context.record_key_fields.clone();
+
+ for idx in 0..num_rows {
+ // Position-based lookup keys are owned decimal strings; key-based
+ // keys are borrowed from the key column. `position_key_buf` holds the
+ // owned string alive for the borrow within this iteration.
+ let position_key_buf;
+ let key: &str = match base_match {
+ BaseMatch::RecordKey => key_array.as_ref().unwrap().value(idx),
+ BaseMatch::Position => {
+ position_key_buf = position_array.as_ref().unwrap().value(idx).to_string();
+ position_key_buf.as_str()
+ }
+ };
+ let log_rec_opt = self.base.records.remove(key)?;
+ match log_rec_opt {
+ None => keep.append(true),
+ Some(log_rec) => {
+ // Build the base ordering value only now — on a conflict row.
+ let base_ordering = orderings.value_at(idx);
+ let winner = pick_winner(merge_mode, &log_rec, &base_ordering);
+ // Mirror the per-record counting the legacy row path performs
+ // in `RecordBuffer::has_next_base_record`: a key present in
+ // both base and log is an update unless the merged result is a
+ // delete (`Winner::LogDelete`). Base-only rows (the `None`
+ // arm) are intentionally not counted, matching gold.
+ self.base.update_processor.process_update(
+ key,
+ Some(&log_rec),
+ &log_rec,
+ matches!(winner, Winner::LogDelete),
+ )?;
+ match winner {
+ Winner::Base => {
+ // Base wins the ordering. For a partial-update table
+ // the winning base row must still be back-filled from
+ // the LOSING log record: Java's
+ // `EventTimePartialRecordMerger.finalMerge` fills the
+ // winner from the loser in BOTH directions (when the
+ // OLDER/base record wins, `partialMerge(base, log)`).
+ // A default/sentinel column on the base row therefore
+ // takes the log record's real value. Non-partial
+ // tables — and a losing delete tombstone, which has no
+ // columns to fill from — emit the base row unchanged
+ // (the common all-keep fast path).
+ let filled = match log_rec.get_record() {
+ Some(loser) if ignore_defaults => {
+ let base_rec = BufferedRecord::new_data(
+ key.to_string(),
+ base.slice(idx, 1),
+ base_ordering.clone(),
+ );
+ let (rec, changed) = reconcile_defaults_from_prior(
+ base_rec,
+ &loser,
+ 0,
+ &default_retains,
+ &key_field_names,
+ )?;
+ changed.then_some(rec)
+ }
+ Some(loser) => match unavailable_value.as_deref() {
+ Some(sentinel) => {
+ let base_rec = BufferedRecord::new_data(
+ key.to_string(),
+ base.slice(idx, 1),
+ base_ordering.clone(),
+ );
+ let (rec, changed) = fill_unavailable_from_base(
+ base_rec,
+ &loser,
+ 0,
+ sentinel,
+ &key_field_names,
+ )?;
+ changed.then_some(rec)
+ }
+ None => None,
+ },
+ // Losing log record is a delete tombstone (no
+ // columns) → nothing to fill from.
+ None => None,
+ };
+ match filled {
+ Some(rec) => {
+ keep.append(false);
+ any_dropped = true;
+ replacements.push(rec);
+ }
+ None => keep.append(true),
+ }
+ }
+ Winner::LogDelete => {
+ keep.append(false);
+ any_dropped = true;
+ }
+ Winner::LogData => {
+ keep.append(false);
+ any_dropped = true;
+ // Partial-update (IS_PARTIAL / KEEP_VALUES): a log record
+ // that carries only a subset of columns overlays its present
+ // columns onto this base row, keeping the base value for every
+ // column it omits. Detected by the record's narrow schema.
+ // Falls through to FILL_UNAVAILABLE (below) for full-schema
+ // records carrying the toasted/unavailable sentinel.
+ let log_batch = log_rec.get_record();
+ let rec = match log_batch {
+ Some(lb) if schema_is_partial(&lb.schema(), target_schema) => {
+ let merged = overlay_partial_over_prior(
+ &lb,
+ &base.slice(idx, 1),
+ target_schema,
+ )?;
+ BufferedRecord::new_data(
+ log_rec.record_key,
+ merged,
+ log_rec.ordering_value,
+ )
+ }
+ // Full-schema winning log record — apply the table's
+ // partial-update blend against the base row, if any. A
+ // table configures at most one mode, so these are mutually
+ // exclusive; IGNORE_DEFAULTS is checked first.
+ //
+ // IGNORE_DEFAULTS: a column whose value is the field default
+ // (null for the common nullable columns) keeps the prior base
+ // value (PartialUpdateAvroPayload).
+ _ if ignore_defaults => {
+ reconcile_defaults_from_prior(
+ log_rec,
+ base,
+ idx,
+ &default_retains,
+ &key_field_names,
+ )?
+ .0
+ }
+ // FILL_UNAVAILABLE: a column equal to the toasted/unavailable
+ // sentinel keeps the prior base value (Postgres-Debezium).
+ _ => match unavailable_value.as_deref() {
+ Some(sentinel) => {
+ fill_unavailable_from_base(
+ log_rec,
+ base,
+ idx,
+ sentinel,
+ &key_field_names,
+ )?
+ .0
+ }
+ None => log_rec,
+ },
+ };
+ replacements.push(rec);
+ }
+ }
+ }
+ }
+ }
+
+ // All-keep fast path: no base row dropped and no log replacements means
+ // the output equals the base batch. Return it directly (an `Arc` clone,
+ // zero row copies) instead of running `filter_record_batch`, which deep-
+ // copies every column even for an all-true mask. This is the common MOR
+ // case where a base batch has no overlapping log keys. The map mutation
+ // and update counting above have already happened, so this is sound.
+ if !any_dropped && replacements.is_empty() {
+ return Ok(Some(if base.schema() == *target_schema {
+ base.clone()
+ } else {
+ reconcile_batch_to_schema(base, target_schema)?
+ }));
+ }
+
+ // Apply the keep-mask in ONE Arrow kernel call (vs N per-row slices).
+ let mask = BooleanArray::new(keep.finish(), None);
+ let kept = arrow::compute::filter_record_batch(base, &mask).map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "filter_record_batch failed in merge_one_base_batch_kernel: {e}"
+ ))
+ })?;
+
+ // Reconcile kept rows to target_schema if the source's schema disagrees
+ // on nested-type child field names (e.g. Avro "element" vs Parquet
+ // "array"). Common case for projected reads: schema matches exactly,
+ // this is a no-op.
+ let kept = if kept.schema() == *target_schema {
+ kept
+ } else {
+ reconcile_batch_to_schema(&kept, target_schema)?
+ };
+
+ if replacements.is_empty() {
+ return Ok(Some(kept));
+ }
+
+ // Build the replacement batch from the sparse log-winning records.
+ let repl_batch = records_to_batch(replacements, target_schema.clone())?;
+ if repl_batch.num_rows() == 0 {
+ return Ok(Some(kept));
+ }
+
+ // ONE concat of two batches — vs the legacy path's concat of 4096
+ // single-row batches per chunk.
+ let refs: Vec<&RecordBatch> = vec![&kept, &repl_batch];
+ arrow::compute::concat_batches(target_schema, refs)
+ .map(Some)
+ .map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "concat_batches failed in merge_one_base_batch_kernel: {e}"
+ ))
+ })
+ }
+}
+
+/// Outcome of a base-vs-log row conflict in [`KeyBasedFileGroupRecordBuffer::merge_one_base_batch_kernel`].
+#[derive(Debug)]
+enum Winner {
+ /// Base row wins; emit base as-is (drop the log entry).
+ Base,
+ /// Log data wins; drop the base row, emit the log record.
+ LogData,
+ /// Log delete wins; drop both.
+ LogDelete,
+}
+
+/// Decide the winner of a base-vs-log conflict without constructing a
+/// `BufferedRecord` for the base row.
+///
+/// Mirrors the semantics of `BufferedRecordMerger::final_merge` for SAME-class
+/// ordering values (the only case the writers produce — a file group carries a
+/// single ordering type):
+/// - `COMMIT_TIME_ORDERING` → log always wins (`CommitTimeRecordMerger`).
+/// - Otherwise (e.g. `EVENT_TIME_ORDERING`) → higher ordering value wins,
+/// ties go to the log side (`EventTimeRecordMerger`'s `new >= old`).
+/// Missing ordering on either side → log wins (latest-write-wins fallback).
+///
+/// A CROSS-class comparison (e.g. `Long` base vs `Double` log, as can arise if a
+/// file group's ordering-value type changes across commits) applies the SAME
+/// `is_same_class` guard as `final_merge`/`should_keep_newer_record` and keeps the
+/// log (newer) record, instead of comparing by `OrderingValue`'s arbitrary
+/// cross-variant `Ord` rank. The vectorized kernel and the row path therefore
+/// agree on cross-class handling.
+///
+/// Keeping this inline (rather than calling the merger) avoids the per-row
+/// `BufferedRecord::new_data` + `RecordBatch::slice` cost on the dominant
+/// no-conflict path. For the conflict path the cost is identical — the
+/// `BufferedRecord` only materialises if there *is* a log entry to match.
+fn pick_winner(
+ merge_mode: &str,
+ log: &BufferedRecord,
+ base_ordering: &Option<OrderingValue>,
+) -> Winner {
+ // Deletes follow distinct semantics from data records. Mirrors Java
+ // `BufferedRecordMergerFactory.deltaMergeDeleteRecord` /
+ // `CommitTimeRecordMerger.deltaMerge(DeleteRecord, ...)`:
+ // - COMMIT_TIME → the delete always applies.
+ // - EVENT_TIME → a delete carrying the DEFAULT (0 / natural-order) ordering
+ // value always applies (`isCommitTimeOrderingDelete`); a delete with a
+ // NON-default ordering is obsolete (base kept) only if the base carries a
+ // same-class ordering value that strictly exceeds the delete's.
+ // The plain `log >= base` data-record test must NOT be used for deletes: a
+ // relocate/CDC delete legitimately carries ordering 0, which would otherwise
+ // lose to any positive base ordering and silently retain a deleted row.
+ if log.is_delete() {
+ let delete_obsolete = merge_mode != "COMMIT_TIME_ORDERING"
+ && match (&log.ordering_value, base_ordering) {
+ (Some(d), Some(b)) => !d.is_default() && d.is_same_class(b) && b > d,
+ _ => false,
+ };
+ return if delete_obsolete {
+ Winner::Base
+ } else {
+ Winner::LogDelete
+ };
+ }
+
+ let log_wins = if merge_mode == "COMMIT_TIME_ORDERING" {
+ true
+ } else {
+ match (&log.ordering_value, base_ordering) {
+ // Same-class compare by value; cross-class (e.g. a Double base vs a
+ // Long log after a precombine-field type change) keeps the newer
+ // record rather than comparing by the arbitrary cross-variant `Ord`
+ // rank -- mirrors `should_keep_newer_record` so the vectorized kernel
+ // and the row path agree.
+ (Some(l), Some(b)) => !l.is_same_class(b) || l >= b,
+ _ => true,
+ }
+ };
+ if log_wins {
+ Winner::LogData
+ } else {
+ Winner::Base
+ }
+}
+
+/// True if `arr`'s single (row-0) value is a non-null STRING/BYTES value equal
+/// to the Debezium `sentinel`. Mirrors `PostgresDebeziumAvroPayload`'s
+/// `containsStringToastedValues` / `containsBytesToastedValues` (STRING compares
+/// the text; BYTES compares the UTF-8 bytes). Non-string/bytes types never match.
+fn array_value_equals_sentinel(arr: &dyn Array, sentinel: &str) -> bool {
+ if arr.is_null(0) {
+ return false;
+ }
+ match arr.data_type() {
+ DataType::Utf8 => arr
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .is_some_and(|a| a.value(0) == sentinel),
+ DataType::LargeUtf8 => arr
+ .as_any()
+ .downcast_ref::<LargeStringArray>()
+ .is_some_and(|a| a.value(0) == sentinel),
+ DataType::Binary => arr
+ .as_any()
+ .downcast_ref::<BinaryArray>()
+ .is_some_and(|a| a.value(0) == sentinel.as_bytes()),
+ DataType::LargeBinary => arr
+ .as_any()
+ .downcast_ref::<LargeBinaryArray>()
+ .is_some_and(|a| a.value(0) == sentinel.as_bytes()),
+ _ => false,
+ }
+}
+
+/// Whole-column variant of [`array_value_equals_sentinel`] (which inspects row 0
+/// only): returns true if ANY row of `arr` equals the toasted `sentinel`. Same
+/// STRING/BYTES type coverage; short-circuits on the first match.
+fn column_contains_sentinel(arr: &dyn Array, sentinel: &str) -> bool {
+ match arr.data_type() {
+ DataType::Utf8 => arr
+ .as_any()
+ .downcast_ref::<StringArray>()
+ .is_some_and(|a| a.iter().any(|v| v == Some(sentinel))),
+ DataType::LargeUtf8 => arr
+ .as_any()
+ .downcast_ref::<LargeStringArray>()
+ .is_some_and(|a| a.iter().any(|v| v == Some(sentinel))),
+ DataType::Binary => arr
+ .as_any()
+ .downcast_ref::<BinaryArray>()
+ .is_some_and(|a| a.iter().any(|v| v == Some(sentinel.as_bytes()))),
+ DataType::LargeBinary => arr
+ .as_any()
+ .downcast_ref::<LargeBinaryArray>()
+ .is_some_and(|a| a.iter().any(|v| v == Some(sentinel.as_bytes()))),
+ _ => false,
+ }
+}
+
+/// Fail-closed guard against harm H7 (CDC unavailable-placeholder leakage): a
+/// `FILL_UNAVAILABLE` sentinel must never survive into a log-only-insert output
+/// batch. Such a record had no base row and no prior log record to fill from
+/// (the base-vs-log kernel and the log-vs-log blend in `process_next_data_record`
+/// both fill only against a *present* prior), so the toasted placeholder would
+/// otherwise be emitted as a real column value.
+///
+/// On the snapshot read path this is unreachable — a Debezium `op=u` (the only op
+/// that carries the sentinel) always has a reachable prior in the same file slice
+/// (base or an earlier log block). This guard turns a silent leak into a loud
+/// error if a future reader (incremental / skip_merge over a partial log range)
+/// makes the no-prior path reachable without adding explicit toasted-insert
+/// handling. Callers invoke it only when a sentinel is configured, so
+/// non-Debezium tables pay nothing.
+fn guard_no_unavailable_sentinel(
+ batch: &RecordBatch,
+ sentinel: &str,
+ key_field_names: &[String],
+) -> Result<()> {
+ for (i, field) in batch.schema().fields().iter().enumerate() {
+ if is_key_or_meta_column(field.name(), key_field_names) {
+ continue;
+ }
+ if column_contains_sentinel(batch.column(i).as_ref(), sentinel) {
+ return Err(crate::error::CoreError::ReadFileSliceError(format!(
+ "FILL_UNAVAILABLE sentinel leaked into a log-only insert (column '{}'): a toasted \
+ record reached output with no prior value to fill from (harm H7). Unreachable on \
+ the snapshot read path; if hit, a reader made the no-prior path reachable and needs \
+ explicit toasted-insert handling.",
+ field.name()
+ )));
+ }
+ }
+ Ok(())
+}
+
+/// Partial-update (`FILL_UNAVAILABLE`) blend for the merge WINNER: each
+/// STRING/BYTES column of `log_rec` whose value equals the toasted `sentinel` (a
+/// column Debezium couldn't capture) is replaced with the loser row's value for
+/// that same-named column. Mirrors
+/// `PostgresDebeziumAvroPayload.mergeToastedValuesIfPresent`.
+///
+/// Direction-agnostic: `log_rec` is the WINNER (kept) record and `base` is the
+/// LOSER row to fill from. The base-vs-log kernel uses it both ways — when the
+/// log wins (`log_rec` = log, `base` = base row) and when the base wins
+/// (`log_rec` = base row, `base` = losing log record). Java's
+/// `EventTimePartialRecordMerger.finalMerge` fills the winner from the loser in
+/// BOTH directions.
+///
+/// Returns `(record, changed)`; the fast path (no toasted column present) returns
+/// `(log_rec, false)` untouched, so a partial-update table pays the rebuild cost
+/// only on the rows that actually carry a sentinel.
+fn fill_unavailable_from_base(
+ log_rec: BufferedRecord,
+ base: &RecordBatch,
+ base_row_idx: usize,
+ sentinel: &str,
+ key_field_names: &[String],
+) -> Result<(BufferedRecord, bool)> {
+ let log_batch = match log_rec.get_record() {
+ Some(b) => b,
+ None => return Ok((log_rec, false)),
+ };
+ let base_row = base.slice(base_row_idx, 1);
+ let base_schema = base_row.schema();
+ let log_schema = log_batch.schema();
+
+ let mut changed = false;
+ let mut cols: Vec<ArrayRef> = Vec::with_capacity(log_batch.num_columns());
+ for (i, field) in log_schema.fields().iter().enumerate() {
+ let log_col = log_batch.column(i);
+ // Skip the record-key column(s) and `_hoodie_` meta columns: base and
+ // log rows are matched BY key, so these are provably equal on both
+ // sides (substitution was a no-op), but the explicit skip documents
+ // the invariant and guards future call sites.
+ if is_key_or_meta_column(field.name(), key_field_names) {
+ cols.push(log_col.clone());
+ continue;
+ }
+ if array_value_equals_sentinel(log_col.as_ref(), sentinel)
+ && let Ok(bidx) = base_schema.index_of(field.name())
+ {
+ let base_col = base_row.column(bidx);
+ // Only substitute when the base carries the same-typed column;
+ // otherwise leave the (sentinel) log value rather than risk a
+ // type-mismatched batch.
+ if base_col.data_type() == field.data_type() {
+ cols.push(base_col.clone());
+ changed = true;
+ continue;
+ }
+ }
+ cols.push(log_col.clone());
+ }
+
+ if !changed {
+ return Ok((log_rec, false));
+ }
+ let blended = RecordBatch::try_new(log_schema.clone(), cols).map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "fill_unavailable_from_base failed to rebuild record: {e}"
+ ))
+ })?;
+ Ok((
+ BufferedRecord::new_data(log_rec.record_key, blended, log_rec.ordering_value),
+ true,
+ ))
+}
+
+/// Partial-update (`IGNORE_DEFAULTS`) blend for the merge WINNER: each column of
+/// `log_rec` whose value equals the field's declared default is replaced with the
+/// `prior` row's value for that same-named column. `default_retains` carries the
+/// per-field rule (see [`DefaultRetain`]); a field absent from it falls back to
+/// `OnNull`. Mirrors Java `PartialUpdateHandler.reconcileDefaultValues`.
+///
+/// Direction-agnostic: `log_rec` is the WINNER (kept) record and `prior` is the
+/// LOSER row to fill from. Used by every merge path:
+/// - log-vs-log: winner is the higher-ordering log record, `prior` the other one;
+/// - base-vs-log, log wins: `log_rec` = log record, `prior` = base row;
+/// - base-vs-log, base wins: `log_rec` = base row, `prior` = losing log record.
+///
+/// Java's `EventTimePartialRecordMerger.finalMerge`/`deltaMerge` fill the winner
+/// from the loser in BOTH directions, so "prior" means the loser on either path,
+/// not specifically the base file.
+///
+/// Returns `(record, changed)`; the fast path (no default-valued column present)
+/// returns `(log_rec, false)` untouched, so a partial-update table pays the
+/// rebuild cost only on rows that actually carry a default column.
+fn reconcile_defaults_from_prior(
+ log_rec: BufferedRecord,
+ prior: &RecordBatch,
+ prior_row_idx: usize,
+ default_retains: &HashMap<String, DefaultRetain>,
+ key_field_names: &[String],
+) -> Result<(BufferedRecord, bool)> {
+ let log_batch = match log_rec.get_record() {
+ Some(b) => b,
+ None => return Ok((log_rec, false)),
+ };
+ let prior_row = prior.slice(prior_row_idx, 1);
+ let prior_schema = prior_row.schema();
+ let log_schema = log_batch.schema();
+
+ let mut changed = false;
+ let mut cols: Vec<ArrayRef> = Vec::with_capacity(log_batch.num_columns());
+ for (i, field) in log_schema.fields().iter().enumerate() {
+ let log_col = log_batch.column(i);
+ // Skip the record-key column(s) and `_hoodie_` meta columns: prior and
+ // incoming rows are matched BY key, so these are provably equal on
+ // both sides (substitution was a no-op), but the explicit skip
+ // documents the invariant and guards future call sites.
+ if is_key_or_meta_column(field.name(), key_field_names) {
+ cols.push(log_col.clone());
+ continue;
+ }
+ // A field with no declared default retains on null (OnNull); one with a
+ // declared default retains when the incoming value equals that default.
+ let retain = default_retains
+ .get(field.name())
+ .unwrap_or(&DefaultRetain::OnNull);
+ if value_is_default(log_col.as_ref(), retain)
+ && let Ok(pidx) = prior_schema.index_of(field.name())
+ {
+ let prior_col = prior_row.column(pidx);
+ // Only substitute when the prior carries the same-typed column;
+ // otherwise leave the (default) log value rather than risk a
+ // type-mismatched batch.
+ if prior_col.data_type() == field.data_type() {
+ cols.push(prior_col.clone());
+ changed = true;
+ continue;
+ }
+ }
+ cols.push(log_col.clone());
+ }
+
+ if !changed {
+ return Ok((log_rec, false));
+ }
+ let blended = RecordBatch::try_new(log_schema.clone(), cols).map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "reconcile_defaults_from_prior failed to rebuild record: {e}"
+ ))
+ })?;
+ Ok((
+ BufferedRecord::new_data(log_rec.record_key, blended, log_rec.ordering_value),
+ true,
+ ))
+}
+
+/// True if the array's value at row 0 equals the field's declared default under
+/// `retain` (so IGNORE_DEFAULTS should retain the prior value for this column).
+fn value_is_default(arr: &dyn Array, retain: &DefaultRetain) -> bool {
+ match retain {
+ DefaultRetain::OnNull => arr.is_null(0),
+ DefaultRetain::Never => false,
+ DefaultRetain::OnBool(b) => {
+ !arr.is_null(0)
+ && arr
+ .as_any()
+ .downcast_ref::<BooleanArray>()
+ .is_some_and(|a| a.value(0) == *b)
+ }
+ DefaultRetain::OnInt(d) => {
+ if arr.is_null(0) {
+ return false;
+ }
+ // Avro int → Arrow Int32, Avro long → Arrow Int64. (No Avro byte/short.)
+ match arr.data_type() {
+ DataType::Int32 => arr
+ .as_any()
+ .downcast_ref::<Int32Array>()
+ .is_some_and(|a| i64::from(a.value(0)) == *d),
+ DataType::Int64 => arr
+ .as_any()
+ .downcast_ref::<Int64Array>()
+ .is_some_and(|a| a.value(0) == *d),
+ _ => false,
+ }
+ }
+ }
+}
+
+/// Build the per-field IGNORE_DEFAULTS retention rules from a record Avro schema
+/// JSON. A field absent from the returned map (or a schema that fails to parse /
+/// isn't a record) is treated as [`DefaultRetain::OnNull`] by the caller.
+fn parse_default_retains(avro_json: &str) -> HashMap<String, DefaultRetain> {
+ let mut out = HashMap::new();
+ let schema = match apache_avro::Schema::parse_str(avro_json) {
+ Ok(s) => s,
+ Err(_) => return out,
+ };
+ if let apache_avro::Schema::Record(record) = schema {
+ for field in &record.fields {
+ out.insert(field.name.clone(), default_retain_for(field));
+ }
+ }
+ out
+}
+
+/// Derive a field's [`DefaultRetain`] from its declared Avro default, faithful to
+/// Java's `toJavaDefaultValue` + reference `==` (see [`DefaultRetain`] docs).
+fn default_retain_for(field: &apache_avro::schema::RecordField) -> DefaultRetain {
+ use serde_json::Value;
+ match &field.default {
+ None | Some(Value::Null) => DefaultRetain::OnNull,
+ Some(Value::Bool(b)) => DefaultRetain::OnBool(*b),
+ Some(Value::Number(n)) => {
+ // Only integer-family defaults in the JVM cache range can match via
+ // `==`; float/double are never cached, and Avro has no byte/short.
+ match (field_is_integer(&field.schema), n.as_i64()) {
+ (true, Some(v)) if (-128..=127).contains(&v) => DefaultRetain::OnInt(v),
+ _ => DefaultRetain::Never,
+ }
+ }
+ // string / bytes / array / object defaults: Java's `==` never matches.
+ Some(_) => DefaultRetain::Never,
+ }
+}
+
+/// True if the field's schema (or the non-null branch of a nullable union) is an
+/// Avro integer family type (`int`/`long`) — the types the JVM caches boxes for.
+fn field_is_integer(schema: &apache_avro::Schema) -> bool {
+ use apache_avro::Schema as S;
+ match schema {
+ S::Int | S::Long => true,
+ S::Union(union) => union
+ .variants()
+ .iter()
+ .any(|v| matches!(v, S::Int | S::Long)),
+ _ => false,
+ }
+}
+
+/// True if `record_schema` is a partial-update (`IS_PARTIAL` / KEEP_VALUES) shape
+/// relative to `target_schema`: it is missing at least one of the target's
+/// fields. Such a record carries only the columns an upsert actually changed;
+/// the absent columns must be filled from the prior (base/older) record at merge
+/// time rather than treated as nulls. Mirrors Java
+/// `SparkRecordMergingUtils.isPartial` (`schema != mergedSchema`), specialized to
+/// "the record is missing at least one of the reader schema's fields" (it may
+/// still carry extra meta columns the reader projection pruned).
+///
+/// A non-partial log block is resolved up to the reader schema at decode time, so
+/// only `IS_PARTIAL` blocks (decoded writer-only) reach the merge with a narrow
+/// schema — making this field-set check equivalent to the block's IS_PARTIAL flag.
+///
+/// Detected by field SET, not field count. An `IS_PARTIAL` block carries Hudi
+/// meta columns (`_hoodie_record_key`, `_hoodie_commit_time`, …) alongside the
+/// updated data columns, while the projected reader (target) schema often prunes
+/// those meta columns. So a partial record can have AS MANY or MORE fields than
+/// the target yet still be missing real data columns (e.g. the `id` key) — a
+/// field-count short-circuit (`record.len() >= target.len() ⇒ not partial`) would
+/// misclassify it as full, skip the overlay, and fail downstream in
+/// `reconcile_batch_to_schema` with "column '<x>' missing". The sound check is
+/// simply "any target field absent from the record"; for full records every
+/// target field is present, so it returns false after a bounded name scan.
+fn schema_is_partial(record_schema: &Schema, target_schema: &Schema) -> bool {
+ target_schema
+ .fields()
+ .iter()
+ .any(|f| record_schema.index_of(f.name()).is_err())
+}
+
+/// Overlay a partial-update record's present columns onto a prior row, producing
+/// a full `target_schema` single-row batch: for each target field, take the
+/// PARTIAL record's column when it carries that field, otherwise the PRIOR row's
+/// column (the "keep previous value" semantics of KEEP_VALUES). A field absent
+/// from BOTH (e.g. the prior was itself written with an older schema) becomes a
+/// typed null. Mirrors Java `SparkRecordMergingUtils.mergePartialRecords`.
+///
+/// `partial` and `prior_row` are single-row batches (`prior_row` is the caller's
+/// `prior.slice(idx, 1)`); the result is one row in `target_schema`.
+fn overlay_partial_over_prior(
+ partial: &RecordBatch,
+ prior_row: &RecordBatch,
+ target_schema: &SchemaRef,
+) -> Result<RecordBatch> {
+ // Reconcile via the kernel's name-metadata reconcile (`reconcile_batch_to_schema`)
+ // rather than an exact `DataType` compare (#66 review). The old exact-compare
+ // mishandled a reconcilable NESTED-type difference (Avro-vs-Parquet child field
+ // names — same physical layout, different `DataType` tag) two ways: a *partial*
+ // column that differed spuriously HARD-ERRORED, and a *prior/base* column that
+ // differed was SILENTLY dropped to null. Name-reconcile keeps such columns; a
+ // genuine physical mismatch (e.g. Int64 vs Int32) still fails loudly inside the
+ // rebuild. (By the D5 design every batch reaching the merge is already required-
+ // typed, so only nested child NAMES legitimately differ here — a scalar type
+ // mismatch is an upstream bug and stays loud, same as the rest of the kernel.)
+ let partial_schema = partial.schema();
+ let prior_schema = prior_row.schema();
+ let mut cols: Vec<ArrayRef> = Vec::with_capacity(target_schema.fields().len());
+ for field in target_schema.fields() {
+ // Present in the partial → overlay it. Exact type: take as-is. A nested
+ // child-NAME-only difference (Avro "item"/"entries" vs Parquet
+ // "element"/"key_value" — same physical layout) is name-reconciled. A
+ // genuine physical mismatch is a LOUD error: a present value must never be
+ // silently dropped OR unsafely reinterpreted (both `reconcile_batch_to_schema`
+ // and `project_batch_to_schema` would reinterpret/narrow an Int64-vs-Int32
+ // buffer — hence the explicit `is_name_reconcilable` gate).
+ if let Ok(pidx) = partial_schema.index_of(field.name()) {
+ let col = partial.column(pidx);
+ if col.data_type() == field.data_type() {
+ cols.push(col.clone());
+ } else if is_name_reconcilable(col.data_type(), field.data_type()) {
+ cols.push(reconcile_one_column(partial, pidx, field)?);
+ } else {
+ return Err(crate::error::CoreError::Unsupported(format!(
+ "partial-update column '{}' has type {:?} but the table schema expects {:?}",
+ field.name(),
+ col.data_type(),
+ field.data_type()
+ )));
+ }
+ continue;
+ }
+ // Absent from the partial → keep the prior row's value. Same rules, except a
+ // genuine prior-side mismatch falls back to a typed null (unchanged
+ // prior-side behavior); a nested child-name-only difference is now
+ // reconciled-and-KEPT (previously silently nulled).
+ if let Ok(bidx) = prior_schema.index_of(field.name()) {
+ let col = prior_row.column(bidx);
+ if col.data_type() == field.data_type() {
+ cols.push(col.clone());
+ continue;
+ } else if is_name_reconcilable(col.data_type(), field.data_type()) {
+ cols.push(reconcile_one_column(prior_row, bidx, field)?);
+ continue;
+ }
+ }
+ // Absent from both, or a genuine prior-side type mismatch → typed null.
+ cols.push(arrow_array::new_null_array(field.data_type(), 1));
+ }
+ RecordBatch::try_new(target_schema.clone(), cols).map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "overlay_partial_over_prior failed to rebuild record: {e}"
+ ))
+ })
+}
+
+/// True iff `source` can be reconciled to `target` by name-metadata ALONE — i.e.
+/// they are structurally identical apart from nested child-FIELD names (arrow-avro
+/// "item"/"entries" vs Parquet "element"/"key_value") or field metadata, so the
+/// underlying buffers are byte-compatible. A primitive or layout difference
+/// (e.g. Int64 vs Int32, List vs LargeList) is NOT reconcilable — reconciling it
+/// would reinterpret or drop bytes — so it returns false and the caller treats it
+/// as a genuine mismatch. Conservative by construction: anything unrecognized is
+/// false.
+fn is_name_reconcilable(source: &arrow_schema::DataType, target: &arrow_schema::DataType) -> bool {
+ use arrow_schema::DataType::{LargeList, List, Map, Struct};
+ if source == target {
+ return true;
+ }
+ match (source, target) {
+ // List/LargeList/Map wrapper field NAME (+ metadata) may differ; recurse
+ // into the element/entries type. Map also requires the sorted flag to match.
+ (List(a), List(b)) | (LargeList(a), LargeList(b)) => {
+ is_name_reconcilable(a.data_type(), b.data_type())
+ }
+ (Map(a, sa), Map(b, sb)) => sa == sb && is_name_reconcilable(a.data_type(), b.data_type()),
+ // Struct field NAMES are user data (must match); recurse into each field's
+ // type in order.
+ (Struct(fa), Struct(fb)) => {
+ fa.len() == fb.len()
+ && fa.iter().zip(fb.iter()).all(|(x, y)| {
+ x.name() == y.name() && is_name_reconcilable(x.data_type(), y.data_type())
+ })
+ }
+ _ => false,
+ }
+}
+
+/// Name-reconcile a single source column (`src.column(src_idx)`) to
+/// `target_field`'s type — nested child-field-name differences only (the same
+/// [`reconcile_batch_to_schema`] name-metadata reconcile the drain uses). Callers
+/// MUST gate with [`is_name_reconcilable`] first, since the underlying rebuild
+/// reinterprets buffers and is only byte-safe when the physical layouts match.
+fn reconcile_one_column(
+ src: &RecordBatch,
+ src_idx: usize,
+ target_field: &arrow_schema::FieldRef,
+) -> Result<ArrayRef> {
+ let one = RecordBatch::try_new(
+ Arc::new(arrow_schema::Schema::new(vec![
+ src.schema().field(src_idx).clone(),
+ ])),
+ vec![src.column(src_idx).clone()],
+ )
+ .map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!("overlay reconcile setup: {e}"))
+ })?;
+ let target_one: SchemaRef = Arc::new(arrow_schema::Schema::new(vec![target_field.clone()]));
+ let recon = reconcile_batch_to_schema(&one, &target_one)?;
+ Ok(recon.column(0).clone())
+}
+
+/// Pad a partial-update record that has NO prior to merge against (a log-only
+/// insert touching only a subset of columns) up to the full `target_schema`,
+/// filling every absent field with a typed null. Equivalent to overlaying onto
+/// an all-null prior row.
+fn pad_partial_to_target(partial: &RecordBatch, target_schema: &SchemaRef) -> Result<RecordBatch> {
+ let partial_schema = partial.schema();
+ let mut cols: Vec<ArrayRef> = Vec::with_capacity(target_schema.fields().len());
+ for field in target_schema.fields() {
+ match partial_schema.index_of(field.name()) {
+ // Present column → take it; a type mismatch is a loud error (a present
+ // value must never be silently dropped to null).
+ Ok(pidx) => {
+ let col = partial.column(pidx);
+ if col.data_type() != field.data_type() {
+ return Err(crate::error::CoreError::Unsupported(format!(
+ "partial-update column '{}' has type {:?} but the table schema expects {:?}",
+ field.name(),
+ col.data_type(),
+ field.data_type()
+ )));
+ }
+ cols.push(col.clone());
+ }
+ // Absent column → typed null (no prior value for a log-only insert).
+ Err(_) => cols.push(arrow_array::new_null_array(field.data_type(), 1)),
+ }
+ }
+ RecordBatch::try_new(target_schema.clone(), cols).map_err(|e| {
+ crate::error::CoreError::ReadFileSliceError(format!(
+ "pad_partial_to_target failed to rebuild record: {e}"
+ ))
+ })
+}
+
+/// Schema union of a partial `incoming` record over a `prior` record: the
+/// prior's fields followed by any incoming field the prior lacks. Used for the
+/// log-vs-log partial merge so the buffered result carries the columns from BOTH
+/// log updates (and stays narrower than the reader schema only if their union
+/// still omits some — letting base-vs-log fill the remainder).
+fn union_schema(incoming: &Schema, prior: &Schema) -> SchemaRef {
+ let mut fields: Vec<arrow_schema::FieldRef> = prior.fields().iter().cloned().collect();
+ for f in incoming.fields() {
+ if prior.index_of(f.name()).is_err() {
+ fields.push(f.clone());
+ }
+ }
+ Arc::new(Schema::new(fields))
+}
+
+impl HoodieFileGroupRecordBuffer for KeyBasedFileGroupRecordBuffer {
+ fn get_buffer_type(&self) -> BufferType {
+ BufferType::KeyBasedMerge
+ }
+
+ /// Mirrors Java's `KeyBasedFileGroupRecordBuffer.processDataBlock(HoodieDataBlock, Option<KeySpec>)`.
+ ///
+ /// Inflates the block on demand (matching Java where inflate/deserialize/deflate
+ /// happens inside the block triggered by `getRecordsIterator`), then iterates
+ /// each record, extracts the key, creates a BufferedRecord, and calls
+ /// `process_next_data_record`.
+ fn process_data_block(&mut self, block: &mut LogBlock) -> Result<()> {
+ // Mirrors Java: getRecordsIterator → getEngineRecordIterator
+ // → readRecordsFromBlockPayload → inflate → deserializeRecords → deflate
+ let decode_start = std::time::Instant::now();
+ // Upstream fetches a lazy block's content here. This crate's log file
+ // reader returns blocks with their content already read, so there is
+ // nothing to fetch and the take below finds it present.
+ self.base.stage_decode_ms += decode_start.elapsed().as_millis() as u64;
+
+ if let LogBlockContent::Records(record_batches) = std::mem::take(&mut block.content) {
+ let total_rows: usize = record_batches
+ .data_batches
+ .iter()
+ .map(|b| b.num_rows())
+ .sum();
+ log::debug!(
+ "[KeyBasedBuffer] processDataBlock: {} data batches, {} total rows",
+ record_batches.data_batches.len(),
+ total_rows,
+ );
+ for batch in record_batches.data_batches {
+ // A2: intern the decoded block batch into a single `Arc` here
+ // (the one mint point per block batch). Every BufferedRecord
+ // produced from it shares this `Arc`, so the map pins one source
+ // batch per block — and `Arc::as_ptr` is a valid compaction
+ // grouping key. The map then pins only the block batches that
+ // have a surviving key (dropped otherwise when this loop ends).
+ let batch = Arc::new(batch);
+ let records = self
+ .record_context
+ .batch_to_buffered_records(&batch, self.base.delete_context.as_ref())?;
+ for (key, record) in records {
+ self.process_next_data_record(record, &key)?;
+ }
+ }
+ }
+ // No explicit deflate: the `std::mem::take(&mut block.content)` above
+ // already replaced the content with `LogBlockContent::Empty` (its
+ // Default), which is exactly what `deflate()` does — a trailing
+ // `block.deflate()` here was a dead no-op (review C8).
+ Ok(())
+ }
+
+ /// Mirrors Java's `KeyBasedFileGroupRecordBuffer.processNextDataRecord(BufferedRecord, Serializable)`.
+ ///
+ /// ```java
+ /// BufferedRecord<T> existingRecord = records.get(recordKey);
+ /// totalLogRecords++;
+ /// bufferedRecordMerger.deltaMerge(record, existingRecord)
+ /// .ifPresent(merged -> records.put(recordKey, merged.toBinary(recordContext)));
+ /// ```
+ fn process_next_data_record(&mut self, record: BufferedRecord, key: &str) -> Result<()> {
+ self.base.total_log_records += 1;
+
+ // Two mutually-exclusive log-vs-log fix-ups can rewrite the incoming record
+ // before the merge — a record is either a narrow KEEP_VALUES record or a
+ // full-schema FILL_UNAVAILABLE one, so partial is checked first and toasted
+ // is the `else`:
+ //
+ // Partial-update (IS_PARTIAL / KEEP_VALUES): the incoming record carries only
+ // a subset of columns; overlay its present columns onto the prior buffered
+ // record for this key, producing the UNION of their columns. The result
+ // stays narrower than the reader schema until a base row or later update
+ // supplies the rest, so base-vs-log can still fill the gaps. With no prior
+ // data record it stays narrow and is padded at the base/drain step.
+ let inc_is_partial = match (record.get_record(), self.base.reader_schema.as_ref()) {
+ (Some(ib), Some(target)) => schema_is_partial(&ib.schema(), target),
+ _ => false,
+ };
+ if inc_is_partial {
+ let record = match self.base.records.get(key)? {
+ Some(prior) => match (record.get_record(), prior.get_record()) {
+ (Some(ib), Some(prior_batch)) => {
+ let union = union_schema(&ib.schema(), &prior_batch.schema());
+ let merged = overlay_partial_over_prior(&ib, &prior_batch, &union)?;
+ BufferedRecord::new_data(record.record_key, merged, record.ordering_value)
+ }
+ // Prior is a delete tombstone (no columns) → keep incoming as-is.
+ _ => record,
+ },
+ None => record,
+ };
+ // Single-probe merge (perf): probe `key` once via `merge_in_place` and
+ // overwrite in place, instead of get(probe+clone) → delta_merge → insert
+ // (probe+clone+probe). The merger only reads `existing` by reference. A2
+ // semantics preserved: the merged BatchRef payload is stored in-memory
+ // (IPC serialization deferred to spill only).
+ let merger = &self.base.buffered_record_merger;
+ self.base
+ .records
+ .merge_in_place(key, |existing| merger.delta_merge(&record, existing))?;
Review Comment:
**Non-blocking (follow-up):** when the existing record wins this delta merge (event-time, higher ordering), the prior is stored unchanged, so a stale partial's unique columns are dropped — Java folds the loser in both directions (`DefaultSparkRecordMerger.partialMerge` calls `mergePartialRecords(newer, older)` when older wins). E.g. C1 partial-updates `a` at ts=9, then C2 partial-updates `b` at ts=2: Hudi keeps C2's `b`, this keeps the base's. The IGNORE_DEFAULTS/FILL_UNAVAILABLE branch below does fill both directions, so it's just this narrow-schema path; a fixture with out-of-order partials would pin it.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/src/file_group/reader_v2/log_record_reader.rs:
##########
@@ -0,0 +1,1805 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
Review Comment:
**Non-blocking (nit):** this module doc still says "Nothing consumes it yet / unreachable from the crate's call graph", but this PR wires it into the production path — same stale line in `merged_log_record_reader.rs`, `buffer/loader.rs`, `record_merger.rs`, `delete_context.rs`, `update_processor.rs`, and `record_buffer.rs`. Worth a sweep so the docs don't contradict the wiring.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
##########
crates/core/tests/gold_parity_tests.rs:
##########
@@ -0,0 +1,986 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Every fixture, both file group reader versions, against what Hudi returns.
+//!
+//! Each fixture ships a `gold_data/` snapshot produced by Spark reading the same
+//! table. This walks the whole corpus and reads each one twice — once with the
+//! table twice — once with file group reader version 1, once with version 2 — and compares both
+//! against that snapshot.
+//!
+//! The point is the second column. Individual tests assert values we wrote down
+//! ourselves; this asserts agreement with the reference implementation, across
+//! the version / keygen / partitioning / log-format grid rather than the handful
+//! of fixtures that happened to get attention.
+//!
+//! # The option matrix
+//!
+//! A `SELECT *` snapshot only proves the two versions agree when nothing is asked of
+//! them. Projection is where they differ structurally — it reaches the base file
+//! read, the log-block decode and the post-merge trim by different routes — and
+//! read-optimized and incremental change which files are even considered. So a
+//! fixture may additionally ship `gold_options/`: one Spark snapshot per read
+//! option case, plus a manifest naming the cases.
+//!
+//! That manifest is written by the generator that produces the snapshots
+//! (`crates/test/data/quickstart_trips_table/mor/avro/gold_options.scala`) and
+//! is only *read* here. A case is therefore never spelled out twice, so this
+//! test and the generator cannot come to disagree about what a case means. Which
+//! cases a fixture ships depends on the fixture — a table declaring no ordering
+//! field has no `drop_ordering` case — so nothing here may assume a fixed set.
+//!
+//! # Skipping
+//!
+//! A fixture with no `gold_data/` is skipped, and so is one with no
+//! `gold_options/`. Skipping silently is how a corpus ends up looking covered
+//! when it is not, so both sets are asserted against a written-down expectation
+//! rather than merely printed.
+
+use std::collections::HashMap;
+use std::path::Path;
+
+use arrow::compute::concat_batches;
+use arrow_array::RecordBatch;
+use arrow_cast::display::{ArrayFormatter, FormatOptions};
+use hudi_core::config::read::HudiReadConfig;
+use hudi_core::table::{QueryType, ReadOptions, Table};
+use hudi_test::gold_options::{OptionCase, has_option_manifest, read_option_manifest};
+use hudi_test::{QuickstartTripsTable, SampleTable, TableFormat};
+use strum::IntoEnumIterator;
+
+/// The file group reader versions under comparison. Version 1 is what served
+/// every read before this port.
+const READER_VERSIONS: [&str; 2] = ["1", "2"];
+
+/// Rendered stand-in for a NULL, at any depth.
+///
+/// Rendering with an explicit token is what lets a null be told apart from an
+/// empty value: with the default options both a NULL cell and `""` render to
+/// `""`, and — the case that matters for the container fixtures — so do a null
+/// list element and an empty one. `FormatOptions` is handed to nested child
+/// formatters too, so this reaches inside lists, maps and structs, where a
+/// positional `is_null` check cannot.
+const NULL_TOKEN: &str = "<null>";
+
+/// Fixtures shipping no `gold_data/`, and why.
+///
+/// Lance fixtures have no Spark snapshot; `table_hfile_log_block` was dumped
+/// against a reader with no HFile support, so what it should assert is still an
+/// open question. Asserted rather than printed so a fixture cannot quietly stop
+/// being compared.
+const EXPECTED_WITHOUT_GOLD: &[&str] = &[
+ "table_hfile_log_block [MorAvro]",
+ "v9_lance_nonhivestyle [MorAvro]",
+ "v9_lance_nonpartitioned [Cow]",
+ "v9_lance_txns_nonpart [Cow]",
+ "v9_lance_txns_simple [Cow]",
+ "v9_trips_lance [Cow]",
+ "v9_trips_lance [MorAvro]",
+];
+
+/// Fixtures shipping a `gold_options/` manifest.
+///
+/// The option matrix currently covers the merge-on-read corpus; a fixture that
+/// stops shipping a manifest, or one that starts, has to be reflected here.
+const EXPECTED_OPTION_FIXTURES: &[&str] = &[
+ "table_all_data_types [MorAvro]",
+ "table_column_projection [MorAvro]",
+ "table_corrupt_tail_block [MorAvro]",
+ "table_delete_ord_decimal [MorAvro]",
+ "table_delete_ord_double [MorAvro]",
+ "table_delete_ord_int [MorAvro]",
+ "table_delete_ord_long [MorAvro]",
+ "table_delete_ord_string [MorAvro]",
+ "table_delete_ord_timestamp [MorAvro]",
+ "table_evo_add_col [MorAvro]",
+ "table_evo_promotion [MorAvro]",
+ "table_log_compaction [MorAvro]",
+ "table_log_only [MorAvro]",
+ "table_null_containers [MorAvro]",
+ "table_parquet_log_block [MorAvro]",
+ "table_partial_update [MorAvro]",
+ "v6_trips_8i1u [MorAvro]",
+ "v6_trips_8i3d [MorAvro]",
+ "v8_mor_boundary_windows [MorAvro]",
+ "v8_trips_8i3u1d [MorAvro]",
+ "v9_mor_8i4u_commit_time [MorAvro]",
+ "v9_mor_compacted_incremental [MorAvro]",
+ "v9_mor_nonpart_3commits [MorAvro]",
+];
+
+/// One fixture in one on-disk format.
+struct Fixture {
+ name: String,
+ format: TableFormat,
+ table_path: String,
+ gold_dir_owned: String,
+ options_dir_owned: String,
+ /// The snapshot Hudi returns when merging by record position, for the one
+ /// fixture where that differs from merging by key. Absent means the two
+ /// agree and `gold_dir_owned` is the answer for both.
+ gold_positions_dir: Option<String>,
+}
+
+impl Fixture {
+ fn label(&self) -> String {
+ format!("{} [{:?}]", self.name, self.format)
+ }
+
+ /// The snapshot to compare against for one way of merging.
+ fn gold_dir(&self, merge_by_position: bool) -> String {
+ match (merge_by_position, &self.gold_positions_dir) {
+ (true, Some(dir)) => dir.clone(),
+ _ => self.gold_dir_owned.clone(),
+ }
+ }
+
+ fn has_gold(&self) -> bool {
+ Path::new(&self.gold_dir_owned).is_dir()
+ }
+
+ fn has_option_cases(&self) -> bool {
+ has_option_manifest(&self.options_dir_owned)
+ }
+}
+
+/// Every fixture on disk, in every format it ships in.
+fn all_fixtures() -> Vec<Fixture> {
+ let mut fixtures = Vec::new();
+ for table in SampleTable::iter() {
+ for &format in table.available_formats() {
+ fixtures.push(Fixture {
+ name: table.as_ref().to_string(),
+ format,
+ table_path: table.path(format),
+ gold_dir_owned: table.gold_dir(format),
+ options_dir_owned: table.option_cases_dir(format),
+ gold_positions_dir: None,
+ });
+ }
+ }
+ for table in QuickstartTripsTable::iter() {
+ for &format in table.available_formats() {
+ fixtures.push(Fixture {
+ name: table.as_ref().to_string(),
+ format,
+ table_path: table.path(format),
+ gold_dir_owned: table.gold_dir(format),
+ options_dir_owned: table.option_cases_dir(format),
+ gold_positions_dir: table.gold_positions_dir(format),
+ });
+ }
+ }
+ fixtures
+}
+
+/// Build the read for a case: the reader version under test, how it merges, plus
+/// whichever options the case sets. `None` is the full `SELECT *` read.
+fn options_for(
+ reader_version: &str,
+ merge_by_position: bool,
+ case: Option<&OptionCase>,
+) -> ReadOptions {
+ let mut options = ReadOptions::new()
+ .with_hudi_option(
+ HudiReadConfig::FileGroupReaderVersion.as_ref(),
+ reader_version,
+ )
+ .with_hudi_option(
+ HudiReadConfig::MergeUseRecordPositions.as_ref(),
+ merge_by_position.to_string(),
+ );
+ let Some(case) = case else { return options };
+ if let Some(columns) = &case.projection {
+ options = options.with_projection(columns.clone());
+ }
+ if case.read_optimized {
+ options = options.with_hudi_option(HudiReadConfig::UseReadOptimizedMode.as_ref(), "true");
+ }
+ if let (Some(start), Some(end)) = (&case.start_timestamp, &case.end_timestamp) {
+ options = options
+ .with_query_type(QueryType::Incremental)
+ .with_start_timestamp(start)
+ .with_end_timestamp(end);
+ }
+ options
+}
+
+async fn read_with(
+ reader_version: &str,
+ merge_by_position: bool,
+ table_path: &str,
+ case: Option<&OptionCase>,
+) -> Result<RecordBatch, String> {
+ let table = Table::new(table_path)
+ .await
+ .map_err(|e| format!("open failed: {e}"))?;
+ let options = options_for(reader_version, merge_by_position, case);
+ let batches = table
+ .read(&options)
+ .await
+ .map_err(|e| format!("read failed: {e}"))?;
+ if batches.is_empty() {
+ return Ok(RecordBatch::new_empty(std::sync::Arc::new(
+ arrow_schema::Schema::empty(),
+ )));
+ }
+ // `concat_batches` panics rather than erroring when a batch has fewer
+ // columns than the schema it is handed, so check first: a reader returning
+ // batches of differing shape is a finding, not a crash.
+ let schema = batches[0].schema();
+ let shape = |s: &arrow_schema::Schema| -> Vec<(String, arrow_schema::DataType)> {
+ s.fields()
+ .iter()
+ .map(|f| (f.name().clone(), f.data_type().clone()))
+ .collect()
+ };
+ let first = shape(&schema);
+ if let Some(odd) = batches.iter().find(|b| shape(&b.schema()) != first) {
+ let diffs: Vec<String> = first
+ .iter()
+ .zip(shape(&odd.schema()))
+ .filter(|(a, b)| *a != b)
+ .map(|(a, b)| format!("{}: {:?} vs {:?}", a.0, a.1, b.1))
+ .collect();
+ return Err(format!(
+ "reader returned batches of differing shape: {}",
+ diffs.join("; ")
+ ));
+ }
+ concat_batches(&schema, &batches).map_err(|e| format!("concat failed: {e}"))
+}
+
+/// Compare one fixture's full read under one reader version, returning a failure
+/// description.
+///
+/// Rows are compared as a multiset — every user column rendered, then both
+/// sides sorted — rather than positionally against a key. Several fixtures
+/// carry duplicate record keys, so there is no column that identifies a row.
+async fn compare_owned(
+ table_path: &str,
+ gold_dir: &str,
+ reader_version: &str,
+ merge_by_position: bool,
+) -> Result<(), String> {
+ let actual = read_with(reader_version, merge_by_position, table_path, None).await?;
+ let gold = match hudi_test::gold::read_gold_parquet(gold_dir) {
+ Ok(gold) => gold,
+ // Spark writes a snapshot with no row groups for an empty table, which
+ // the gold reader reports as producing no batches. An empty table and
+ // an empty read agree.
+ Err(e) if e.contains("produced no batches") && actual.num_rows() == 0 => return Ok(()),
+ Err(e) => return Err(e),
+ };
+
+ let columns: Vec<String> = gold
+ .schema()
+ .fields()
+ .iter()
+ .map(|f| f.name().clone())
+ .filter(|n| !n.starts_with("_hoodie_"))
+ .collect();
+
+ compare_rows(&actual, &gold, &columns)
+}
+
+/// Compare one read-option case under one reader version.
+///
+/// When the case projects, the column *set* is itself under test: the
+/// projection is the exact contract, so both sides must have precisely those
+/// columns in precisely that order. Comparing only the columns gold happens to
+/// carry would let a reader that fails to strip an internal column — the record
+/// key it merged on, say — pass unnoticed, which is the main thing the
+/// projection cases exist to catch.
+///
+/// A case that does not project (read-optimized, incremental) has no such
+/// contract, so its columns come from gold the way the full read's do.
+async fn compare_option_case_owned(
+ table_path: &str,
+ options_dir: String,
+ case: OptionCase,
+ reader_version: &str,
+) -> Result<(), String> {
+ let actual = read_with(reader_version, false, table_path, Some(&case)).await?;
+ let gold_dir = hudi_test::gold_options::case_gold_dir(&options_dir, &case.name);
+ let gold = match hudi_test::gold::read_gold_parquet(&gold_dir) {
+ Ok(gold) => gold,
+ Err(e) if e.contains("produced no batches") && actual.num_rows() == 0 => return Ok(()),
+ Err(e) => return Err(e),
+ };
+
+ let Some(projection) = &case.projection else {
+ let columns: Vec<String> = gold
+ .schema()
+ .fields()
+ .iter()
+ .map(|f| f.name().clone())
+ .filter(|n| !n.starts_with("_hoodie_"))
+ .collect();
+ return compare_rows(&actual, &gold, &columns);
+ };
+
+ exact_columns(&gold, projection, "gold")?;
+ exact_columns(&actual, projection, "actual")?;
+ compare_rows(&actual, &gold, projection)
+}
+
+/// Fail unless `batch` has exactly `expected` columns, in order.
+fn exact_columns(batch: &RecordBatch, expected: &[String], side: &str) -> Result<(), String> {
+ let schema = batch.schema();
+ let actual: Vec<&str> = schema.fields().iter().map(|f| f.name().as_str()).collect();
+ if actual != expected {
+ return Err(format!(
+ "{side} columns mismatch: {side}={actual:?} projection={expected:?}"
+ ));
+ }
+ Ok(())
+}
+
+/// Render both sides and compare them row for row.
+fn compare_rows(
+ actual: &RecordBatch,
+ gold: &RecordBatch,
+ columns: &[String],
+) -> Result<(), String> {
+ let gold_rows = render_rows(gold, columns, "gold")?;
+ let actual_rows = render_rows(actual, columns, "actual")?;
+
+ if gold_rows.len() != actual_rows.len() {
+ // Show the rows themselves: a bare count leaves no way to tell a
+ // dropped row from a duplicated one without rerunning by hand.
+ let sample = |rows: &[String]| {
+ rows.iter()
+ .take(6)
+ .map(|r| format!("\n {r}"))
+ .collect::<String>()
+ };
+ return Err(format!(
+ "row count mismatch: actual={} gold={}\n actual rows:{}\n gold rows:{}",
+ actual_rows.len(),
+ gold_rows.len(),
+ sample(&actual_rows),
+ sample(&gold_rows)
+ ));
+ }
+ for (actual_row, gold_row) in actual_rows.iter().zip(gold_rows.iter()) {
+ if actual_row != gold_row {
+ return Err(format!(
+ "row differs:\n actual {actual_row}\n gold {gold_row}"
+ ));
+ }
+ }
+ Ok(())
+}
+
+/// Every row rendered to a comparable string, sorted.
+fn render_rows(batch: &RecordBatch, columns: &[String], side: &str) -> Result<Vec<String>, String> {
+ // One formatter per column, built with an explicit null token so a NULL is
+ // never conflated with an empty value — at the top level or nested inside a
+ // list, map or struct.
+ let format_options = FormatOptions::new().with_null(NULL_TOKEN);
+ let mut formatters = Vec::with_capacity(columns.len());
+ for name in columns {
+ let idx = batch.schema().index_of(name).map_err(|_| {
+ format!("column '{name}' present in gold but missing from {side} output")
+ })?;
+ formatters.push(
+ ArrayFormatter::try_new(batch.column(idx).as_ref(), &format_options)
+ .map_err(|e| format!("build {side} formatter for '{name}': {e}"))?,
+ );
+ }
+ let mut rows = Vec::with_capacity(batch.num_rows());
+ for row in 0..batch.num_rows() {
+ let mut cells = Vec::with_capacity(formatters.len());
+ for formatter in &formatters {
+ let rendered = formatter.value(row).to_string();
+ // A timestamp carrying a zone renders with a trailing `Z` and one
+ // without does not, for the same instant. The zone is schema
+ // metadata, not a value difference.
+ cells.push(rendered.strip_suffix('Z').unwrap_or(&rendered).to_string());
+ }
+ rows.push(cells.join(" | "));
+ }
+ rows.sort();
+ Ok(rows)
+}
+
+/// One comparison that disagreed with Hudi.
+struct Failure {
+ fixture: String,
+ /// The option case, or `None` for the full `SELECT *` read.
+ case: Option<String>,
+ reader_version: &'static str,
+ /// Whether the read merged by record position rather than by record key.
+ merge_by_position: bool,
+ message: String,
+}
+
+impl Failure {
+ /// The reader version as the sweep reports it, distinguishing the two merges.
+ fn version_label(&self) -> String {
+ if self.merge_by_position {
+ format!("{}+positions", self.reader_version)
+ } else {
+ self.reader_version.to_string()
+ }
+ }
+
+ fn describe(&self, format: TableFormat) -> String {
+ let version = self.version_label();
+ match &self.case {
+ Some(case) => format!(
+ "{} [{format:?}] case '{case}' reader version '{version}': {}",
+ self.fixture, self.message
+ ),
+ None => format!(
+ "{} [{format:?}] reader version '{version}': {}",
+ self.fixture, self.message
+ ),
+ }
+ }
+}
+
+/// Which comparisons a [`Known`] entry speaks for.
+enum CaseScope {
+ /// The full `SELECT *` read only. An option case on the same fixture is
+ /// still held to account.
+ SelectStar,
+ /// One named option case.
+ Case(&'static str),
+ /// Every comparison for this fixture.
+ ///
+ /// Deliberate blanket coverage, and only defensible when the reader fails
+ /// on the fixture *whatever* is asked of it — every case failing for the one
+ /// cause. Where a fixture fails only for some cases, name them with
+ /// [`CaseScope::Case`] instead: a blanket there would absorb a later
+ /// regression in the cases that pass today, which is the hole this ratchet
+ /// exists to close.
+ Any,
+}
+
+/// A disagreement we have already explained.
+///
+/// The reader version is part of the identity: nearly every entry below describes
+/// a *version 1* failure, and without naming the version the same entry would
+/// quietly absorb a version 2 regression on that fixture too.
+struct Known {
+ fixture: &'static str,
+ scope: CaseScope,
+ reader_version: &'static str,
+ reason: &'static str,
+}
+
+impl Known {
+ fn covers(&self, failure: &Failure) -> bool {
+ if self.fixture != failure.fixture || self.reader_version != failure.reader_version {
+ return false;
+ }
+ match self.scope {
+ CaseScope::Any => true,
+ CaseScope::SelectStar => failure.case.is_none(),
+ CaseScope::Case(name) => failure.case.as_deref() == Some(name),
+ }
+ }
+}
+
+/// Known disagreements, each with a cause. The sweep is a ratchet: a new
+/// disagreement fails the build, and one that starts passing has to be removed
+/// from here, so this list cannot quietly go stale.
+const KNOWN: &[Known] = &[
+ // Version 1 only: it now reads a log-only slice but returns it without the
+ // base columns. Nothing it is asked to project changes that, so the
+ // fixture is blanketed.
+ Known {
+ fixture: "table_log_compaction",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 drops the base columns",
+ },
+ Known {
+ fixture: "table_log_only",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 drops the base columns",
+ },
+ // Version 1 only: the partition column comes back as Utf8 from one file slice
+ // and Int64 from another, so the batches cannot be concatenated. The
+ // version 2 reads both as Int64 now that it takes the table's
+ // schema rather than each base file's.
+ Known {
+ fixture: "v9_timebasedkeygen_epochmillis",
+ scope: CaseScope::SelectStar,
+ reader_version: "1",
+ reason: "batches of differing shape",
+ },
+ Known {
+ fixture: "v9_timebasedkeygen_unixtimestamp",
+ scope: CaseScope::SelectStar,
+ reader_version: "1",
+ reason: "batches of differing shape",
+ },
+ // Version 1 only, all fixed in version 2: it concatenates a
+ // partial-update log block onto a wider base batch without checking and
+ // panics; it models an avro map's entries field differently between base
+ // and log; and it keeps a superseded row.
+ Known {
+ fixture: "table_partial_update",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 panics on a partial-update block",
+ },
+ Known {
+ fixture: "table_all_data_types",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 models an avro map's entries field",
+ },
+ Known {
+ fixture: "table_column_projection",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 models an avro map's entries field",
+ },
+ Known {
+ fixture: "table_null_containers",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 models an avro map's entries field",
+ },
+ Known {
+ fixture: "table_evo_promotion",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 cannot widen a promoted column",
+ },
+ // Version 1 only, and only partly: it never sees the column a later writer
+ // added, so the full read comes back without it and any projection naming it
+ // fails outright. The cases that don't name `extra` read correctly, so they
+ // are held to account rather than blanketed.
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::SelectStar,
+ reader_version: "1",
+ reason: "version 1 drops a column added by a later writer",
+ },
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("drop_key"),
+ reader_version: "1",
+ reason: "version 1 cannot project a column added by a later writer",
+ },
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("drop_ordering"),
+ reader_version: "1",
+ reason: "version 1 cannot project a column added by a later writer",
+ },
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("reordered"),
+ reader_version: "1",
+ reason: "version 1 cannot project a column added by a later writer",
+ },
+ // Same cause again — the column is simply absent from what version 1 returns,
+ // whatever query type asks for it.
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("read_optimized"),
+ reader_version: "1",
+ reason: "version 1 drops a column added by a later writer",
+ },
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("incr_all"),
+ reader_version: "1",
+ reason: "version 1 drops a column added by a later writer",
+ },
+ Known {
+ fixture: "table_evo_add_col",
+ scope: CaseScope::Case("incr_through_penultimate"),
+ reader_version: "1",
+ reason: "version 1 drops a column added by a later writer",
+ },
+ Known {
+ fixture: "v9_mor_nonpart_3commits",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 returns a stale row",
+ },
+ // Version 1 only: it merges whole batches by sorting and de-duplicating on
+ // record key, so a file group holding a key more than once collapses to one
+ // row per key. Hudi keeps every base row. Version 2 matches
+ // Hudi both by key and by position.
+ Known {
+ fixture: "table_duplicate_keys",
+ scope: CaseScope::Any,
+ reader_version: "1",
+ reason: "version 1 collapses duplicate record keys",
+ },
+];
+
+/// Run one comparison on its own task, so a reader that panics is reported like
+/// any other disagreement rather than aborting the sweep and hiding every
+/// fixture after it.
+async fn record<F>(failures: &mut Vec<Failure>, compared: &mut usize, failure: Failure, work: F)
+where
+ F: std::future::Future<Output = Result<(), String>> + Send + 'static,
+{
+ match tokio::spawn(work).await {
+ Ok(Ok(())) => *compared += 1,
+ Ok(Err(message)) => failures.push(Failure { message, ..failure }),
+ Err(join) if join.is_panic() => failures.push(Failure {
+ message: "PANICKED during read".to_string(),
+ ..failure
+ }),
+ Err(join) => failures.push(Failure {
+ message: join.to_string(),
+ ..failure
+ }),
+ }
+}
+
+/// A null *inside* a container must not render the same as an empty value.
+///
+/// This is the one thing the sweep cannot prove by passing: if both sides render
+/// a null element and an empty one to the same text, a genuine null-vs-empty
+/// divergence compares equal and the container fixtures pass vacuously. The
+/// check has to be made directly, on two batches that differ only in that way.
+#[test]
+fn a_null_list_element_renders_differently_from_an_empty_one() {
+ use arrow_array::Array;
+ use arrow_array::builder::{ListBuilder, StringBuilder};
+
+ let list_batch = |empty_instead_of_null: bool| {
+ let mut builder = ListBuilder::new(StringBuilder::new());
+ builder.values().append_value("a");
+ if empty_instead_of_null {
+ builder.values().append_value("");
+ } else {
+ builder.values().append_null();
+ }
+ builder.append(true);
+ let array = builder.finish();
+ let schema = arrow_schema::Schema::new(vec![arrow_schema::Field::new(
+ "arr",
+ array.data_type().clone(),
+ true,
+ )]);
+ RecordBatch::try_new(
+ std::sync::Arc::new(schema),
+ vec![std::sync::Arc::new(array)],
+ )
+ .unwrap()
+ };
+
+ let columns = vec!["arr".to_string()];
+ let with_null = render_rows(&list_batch(false), &columns, "test").unwrap();
+ let with_empty = render_rows(&list_batch(true), &columns, "test").unwrap();
+
+ assert!(
+ with_null[0].contains(NULL_TOKEN),
+ "a null element must render as the null token, got {:?}",
+ with_null[0]
+ );
+ assert_ne!(
+ with_null, with_empty,
+ "a null list element and an empty one rendered identically, so a \
+ null-vs-empty divergence inside a container would compare equal"
+ );
+}
+
+/// What one pass over the corpus found: the disagreements no [`KNOWN`] entry
+/// explains, which entries did explain something, and what was skipped.
+struct SweepResult {
+ reported: Vec<String>,
+ entry_used: Vec<bool>,
+ without_gold: Vec<String>,
+ with_options: Vec<String>,
+}
+
+/// Read every fixture that ships gold under each of `reader_versions`, where an
+/// entry is a reader version and whether it merges by record position.
+///
+/// Option cases run only for an entry that merges by key: a case's gold is a
+/// single Spark snapshot taken without record positions, so there is nothing to
+/// compare a position-merged option read against.
+async fn sweep(reader_versions: &[(&'static str, bool)]) -> SweepResult {
+ let fixtures = all_fixtures();
+ let mut without_gold: Vec<String> = Vec::new();
+ let mut with_options: Vec<String> = Vec::new();
+ let mut all_failures: Vec<(TableFormat, Failure)> = Vec::new();
+ let mut compared: HashMap<String, usize> = HashMap::new();
+ let mut case_compared: HashMap<&str, usize> = HashMap::new();
+
+ for fixture in &fixtures {
+ if !fixture.has_gold() {
+ without_gold.push(fixture.label());
+ continue;
+ }
+
+ let mut failures: Vec<Failure> = Vec::new();
+
+ for &(reader_version, merge_by_position) in reader_versions {
+ let label = if merge_by_position {
+ format!("{reader_version}+positions")
+ } else {
+ reader_version.to_string()
+ };
+ let path = fixture.table_path.clone();
+ let gold_dir = fixture.gold_dir(merge_by_position);
+ let counter = compared.entry(label).or_default();
+ record(
+ &mut failures,
+ counter,
+ Failure {
+ fixture: fixture.name.clone(),
+ case: None,
+ reader_version,
+ merge_by_position,
+ message: String::new(),
+ },
+ async move { compare_owned(&path, &gold_dir, reader_version, merge_by_position).await },
+ )
+ .await;
+ }
+
+ // Option cases compare against snapshots taken without record positions,
+ // so they run once per key-merging version and not at all under position
+ // merge.
+ if fixture.has_option_cases() && reader_versions.iter().any(|&(_, positions)| !positions) {
+ with_options.push(fixture.label());
+ let manifest = match read_option_manifest(&fixture.options_dir_owned) {
+ Ok(manifest) => manifest,
+ Err(e) => {
+ // A malformed manifest is not a comparison failure that can
+ // be explained away; it means the fixture is wrong.
+ panic!("{}: {e}", fixture.label());
+ }
+ };
+ for case in &manifest.cases {
+ for &(reader_version, positions) in reader_versions {
+ if positions {
+ continue;
+ }
+ let path = fixture.table_path.clone();
+ let options_dir = fixture.options_dir_owned.clone();
+ let case = case.clone();
+ let case_name = case.name.clone();
+ let counter = case_compared.entry(reader_version).or_default();
+ record(
+ &mut failures,
+ counter,
+ Failure {
+ fixture: fixture.name.clone(),
+ case: Some(case_name),
+ reader_version,
+ merge_by_position: false,
+ message: String::new(),
+ },
+ async move {
+ compare_option_case_owned(&path, options_dir, case, reader_version)
+ .await
+ },
+ )
+ .await;
+ }
+ }
+ }
+
+ all_failures.extend(failures.into_iter().map(|f| (fixture.format, f)));
+ }
+
+ // Match each disagreement against the known list, tracking which entries
+ // actually spoke for something. An entry that matches nothing describes a
+ // disagreement that no longer happens, and has to go — otherwise the list
+ // accumulates explanations for behaviour that has since been fixed, and
+ // stops being a description of what is actually wrong.
+ let mut entry_used = vec![false; KNOWN.len()];
+ let mut reported: Vec<String> = Vec::new();
+ for (format, failure) in &all_failures {
+ match KNOWN.iter().position(|known| known.covers(failure)) {
+ Some(index) => entry_used[index] = true,
+ None => reported.push(failure.describe(*format)),
+ }
+ }
+
+ let counts: Vec<String> = reader_versions
+ .iter()
+ .map(|&(reader_version, positions)| {
+ let label = if positions {
+ format!("{reader_version}+positions")
+ } else {
+ reader_version.to_string()
+ };
+ let n = compared.get(&label).copied().unwrap_or(0);
+ format!("{label} {n}")
+ })
+ .collect();
+ println!(
+ "compared {} fixtures — {}; {} without gold",
+ fixtures.len() - without_gold.len(),
+ counts.join(", "),
+ without_gold.len(),
+ );
+ println!(
+ "option cases on {} fixtures — version 1 {}, version 2 {}",
+ with_options.len(),
+ case_compared.get("1").copied().unwrap_or(0),
+ case_compared.get("2").copied().unwrap_or(0),
+ );
+
+ SweepResult {
+ reported,
+ entry_used,
+ without_gold,
+ with_options,
+ }
+}
+
+#[tokio::test(flavor = "multi_thread")]
+async fn every_fixture_matches_hudi_on_both_reader_versions() {
Review Comment:
**Non-blocking (test gap):** no fixture in the corpus exercises event-time "stale loses" — neither a lower-precombine log update nor a stale delete — so a regression flipping version 2 to last-writer-wins under EVENT_TIME_ORDERING would pass this entire sweep. That semantics currently lives only in unit/buffer tests; one fixture with base ts=100 and a later log upsert at ts=50 would close the corpus's biggest blind spot.
⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua merged PR #660:
URL: https://github.com/apache/hudi-rs/pull/660
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3781318351
##########
crates/core/src/table/mod.rs:
##########
@@ -155,6 +158,19 @@ impl Clone for Table {
}
}
+/// One tick below `instant_time`, as a same-width instant string.
+///
+/// Used to turn an exclusive start bound into an inclusive one. Instant times are
+/// fixed-width numeric strings, so this is a plain decrement; a value that does
+/// not parse (the bootstrap sentinels, for instance) is returned unchanged, which
+/// keeps the bound no narrower than it was.
+fn instant_time_minus_one(instant_time: &str) -> String {
Review Comment:
This decrements the instant as an integer, so it can land on a string that sorts correctly but isn't a valid time — `20250713010500000 - 1` is `20250713010499999`, i.e. `SS=99`. That value becomes `StartTimestamp`, and `actions_in_range` parses it, so an incremental read whose earliest admitted commit has `ss=00` and `SSS=000` fails with `Timeline("Invalid epoch millis")` (I reproduced this by renaming a fixture's first commit). Could the exclusive bound be kept as a flag on the selector instead of being encoded as a "one tick below" string?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
.github/workflows/ci.yml:
##########
@@ -81,6 +83,29 @@ jobs:
path: ./cov-reports
if-no-files-found: 'error'
+ # hudi-core without `spill-rocksdb` must keep building and passing. That build
+ # is the reason the feature exists — it needs no libclang and no C++ toolchain,
+ # so a consumer that cannot take a native build (or does not want one in a
+ # Python wheel) has a configuration to use. Nothing else in CI compiles it, so
+ # without this job it would break silently.
+ rust-tests-no-spill-backend:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v6
+ - name: Cache dependencies
+ uses: swatinem/rust-cache@6323deb1c3ea99e6f4dae6b7d3ac1b73ac21ff77 # v2.9.2
Review Comment:
This SHA doesn't resolve — `6323deb1c3ea99e6f4dae6b7d3ac1b73ac21ff77` isn't a commit in swatinem/rust-cache. v2.9.2 is `6323deb102c322ba6fcbdcafc7e3dddab59af2b6`, which is what the other two jobs in this file pin. The job currently fails at "Set up job", so the no-spill leg never runs. (The build itself is fine — I ran both commands locally.)
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/gold_tests.rs:
##########
@@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Reads each merge-on-read layout fixture and checks the result against the
+//! Spark snapshot shipped inside it.
+//!
+//! The fixtures carry their own `gold_data/` — a `SELECT *` of the table taken
+//! from Spark — so what is asserted here is agreement with Hudi's reference
+//! reader, not with this reader's own prior output.
+//!
+//! The file slice is discovered from the extracted fixture rather than written
+//! down per case: each of these tables is a single file group, so the base file
+//! and log files are whatever is on disk. A fixture that gains a file therefore
+//! cannot silently stop being covered.
+
+#![cfg(test)]
+
+use crate::config::HudiConfigs;
+use crate::config::read::HudiReadConfig;
+use crate::file_group::reader_v2::MAX_INSTANT_TIME;
+use crate::file_group::reader_v2::engine::HoodieFileGroupReader;
+use crate::file_group::reader_v2::input_split::InputSplit;
+use crate::file_group::reader_v2::reader_parameters::ReaderParameters;
+use crate::file_group::reader_v2::resolver::resolve_reader_context;
+use crate::storage::Storage;
+use arrow_array::RecordBatch;
+use std::path::Path;
+use std::sync::Arc;
+
+/// The base file and log files of a fixture's single file group, relative to
+/// the table root, plus the partition they live in.
+struct Slice {
+ base: Option<String>,
+ logs: Vec<String>,
+ partition: String,
+}
+
+/// Walk an extracted fixture and pick out its one file group.
+///
+/// Skips `.hoodie` (table metadata) and `gold_data` (the Spark snapshot this
+/// compares against, which is not table data).
+fn discover_slice(table_root: &Path) -> Slice {
+ fn walk(dir: &Path, root: &Path, out: &mut Vec<(String, String)>) {
+ let Ok(entries) = std::fs::read_dir(dir) else {
+ return;
+ };
+ for entry in entries.flatten() {
+ let path = entry.path();
+ let name = entry.file_name().to_string_lossy().to_string();
+ if path.is_dir() {
+ if name == ".hoodie" || name == "gold_data" {
+ continue;
+ }
+ walk(&path, root, out);
+ } else {
+ let rel = path
+ .strip_prefix(root)
+ .expect("walked path is under the table root")
+ .to_string_lossy()
+ .to_string();
+ let partition = Path::new(&rel)
+ .parent()
+ .map(|p| p.to_string_lossy().to_string())
+ .unwrap_or_default();
+ out.push((rel, partition));
+ }
+ }
+ }
+
+ let mut files = Vec::new();
+ walk(table_root, table_root, &mut files);
+
+ let mut base = None;
+ let mut logs = Vec::new();
+ let mut partition = String::new();
+ for (rel, part) in files {
+ let name = Path::new(&rel)
+ .file_name()
+ .map(|n| n.to_string_lossy().to_string())
+ .unwrap_or_default();
+ // Hadoop writes a `..<file>.crc` checksum sidecar next to each data
+ // file. It is not table data and must not reach the readers.
+ if name.ends_with(".crc") {
+ continue;
+ }
+ if name.ends_with(".parquet") && !name.starts_with('.') {
+ base = Some(rel);
+ partition = part;
+ } else if name.contains(".log.") {
+ logs.push(rel);
+ partition = part;
+ }
+ }
+ // Log files are appended in order; the scan relies on that ordering.
+ logs.sort();
+
+ Slice {
+ base,
+ logs,
+ partition,
+ }
+}
+
+/// Read a fixture's file group and return the merged rows.
+async fn read_fixture(table_path: &str) -> crate::Result<RecordBatch> {
+ let slice = discover_slice(Path::new(table_path));
+
+ // Load the table's own properties rather than inventing them: the merge
+ // semantics live in `hoodie.properties`, and inventing configs would test
+ // the reader against a table that does not exist.
+ let mut resolver = crate::table::builder::OptionResolver::new_with_options(
+ table_path,
+ [(
+ HudiReadConfig::EndTimestamp.as_ref(),
+ MAX_INSTANT_TIME.to_string(),
+ )],
+ );
+ resolver.resolve_options().await?;
+ let configs = HudiConfigs::new(resolver.hudi_options.clone());
+ let storage = Storage::new(
+ Arc::new(resolver.storage_options),
+ Arc::new(configs.clone()),
+ )?;
+
+ let has_logs = !slice.logs.is_empty();
+ let mut context = resolve_reader_context(&configs, has_logs)?;
+ // The fixtures are read whole, so nothing bounds the log scan.
+ context.instant_range = None;
+ context.rebuild_record_context(slice.partition.clone());
+
+ let input_split = InputSplit::new(slice.base, None, slice.logs, slice.partition);
+
+ // The table's own schema, the way a real read resolves it. Passing `None`
+ // here — as this harness used to — makes the reader fall back to the base
+ // file's schema, which is stale on any table whose columns were widened
+ // after that file was written. The fixtures are then read into the older
+ // shape and the comparison below, which only counts rows, cannot see it.
+ let table = crate::table::Table::new(table_path).await?;
+ let data_schema = Arc::new(table.data_schema_for_read().await?);
+
+ let mut reader = HoodieFileGroupReader::new(
+ Arc::new(context),
+ storage,
+ input_split,
+ ReaderParameters::default(),
+ Some(data_schema),
+ None,
+ )?;
+ reader.read().await
+}
+
+/// Fixtures the reader reproduces today.
+const GOLD_FIXTURES: &[&str] = &[
+ "table_log_only",
+ "table_log_compaction",
+ "table_parquet_log_block",
+ "table_partial_update",
+ "table_evo_add_col",
+ "table_evo_promotion",
+];
+
+/// Fixtures the reader does not read yet, with what stops each one.
+///
+/// Two causes remain, neither in the merge:
+///
+/// **Avro maps are modelled as a malformed Arrow type.** `avro_to_arrow` turns
+/// an Avro map into `Dictionary(Utf8, V)`, but an Arrow dictionary key must be
+/// an integer, so the array builder rejects it. Avro maps belong in Arrow's
+/// `Map` type. Changing that alters the schema of every table with a map
+/// column, including on the existing read path.
+///
+/// **The delete record's ordering value has two shapes in the wild.** Hudi
+/// moved it into per-type wrapper records — `IntWrapper`, `DecimalWrapper` and
+/// so on — because Avro forbids a union holding two branches of the same
+/// underlying type. This crate carries only the older primitive union, so a log
+/// file written the newer way resolves each branch index to the wrong type.
+///
+/// The two shapes cannot be told apart by decoding: a wrapper is a record whose
+/// single field is the primitive, so `IntWrapper` and a bare `long` encode to
+/// the same bytes. Only the branch index differs. Reading both therefore needs
+/// to key off the writer version rather than the payload, which is a design
+/// question rather than a decoding one — and swapping to the newer schema
+/// outright breaks the fixtures written the older way.
+///
+/// **A corrupt tail block is not recognised.** `LogFileReader` has no
+/// corrupt-block detection — `create_corrupted_block_if_needed` returns `None`
+/// — so it parses the trailing garbage as a block.
+const KNOWN_GAPS: &[(&str, &str)] = &[
Review Comment:
Running this with `--ignored` fails its own second assertion: all ten `KNOWN_GAPS` entries now read correctly, including the four `table_delete_ord_*` ones. The doc above ("two causes remain — avro maps as Dictionary, wrapped delete ordering values") is stale in the same way. Since `tests/gold_parity_tests.rs` covers the same fixtures with a live ratchet, is this harness still earning its keep, or could it be folded in?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/tests/reader_concurrency_tests.rs:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Reader-vs-writer concurrency repros.
+//!
+//! Most of these are `#[ignore]`d, for one of two reasons: they document a
+//! divergence from Hudi's Java reader that is not fixed yet, or they are probes
+//! that print what the reader does without asserting anything. Neither belongs
+//! in the suite — the first would fail it, and the second would pass whatever
+//! happened. Run them with
+//! `cargo test -p hudi-core --test reader_concurrency_tests -- --ignored --nocapture`.
+//!
+//! A divergence that gets fixed should leave this file, or become an ordinary
+//! test that pins the fixed behavior. An `#[ignore]` reason describing something
+//! that no longer happens is worse than no note at all.
+
+use hudi_core::table::{ReadOptions, Table};
+use std::path::{Path, PathBuf};
+
+fn rows(batches: &[arrow_array::RecordBatch]) -> usize {
+ batches.iter().map(|b| b.num_rows()).sum()
+}
+
+/// Copy an existing base file under a NEW file id at `instant`, leaving no
+/// commit for that instant — a writer that wrote its data file and then died.
+fn plant_orphan_base_file(table_dir: &Path, instant: &str) -> PathBuf {
+ let existing = std::fs::read_dir(table_dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.extension().is_some_and(|x| x == "parquet"))
+ .expect("fixture has a base file to copy");
+ let orphan = table_dir.join(format!(
+ "b179bdb3-731c-4894-b855-abfcd6921008-0_0-1-1_{instant}.parquet"
+ ));
+ std::fs::copy(&existing, &orphan).unwrap();
+ orphan
+}
+
+fn fixture_zip(relative: &str) -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR").replace("/core", "/test")).join(relative)
+}
+
+async fn probe(zip: &str, label: &str) {
+ probe_with_reader_version(zip, label, "1").await;
+ probe_with_reader_version(zip, label, "2").await;
+}
+
+async fn probe_with_reader_version(zip: &str, label: &str, reader_version: &str) {
+ use hudi_core::config::read::HudiReadConfig;
+ let dir = hudi_test::extract_test_table_fresh(&fixture_zip(zip));
+ let table_dir = std::fs::read_dir(&dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.is_dir() && p.join(".hoodie").exists())
+ .expect("extracted fixture holds one table dir");
+
+ let opts = ReadOptions::new().with_hudi_option(
+ HudiReadConfig::FileGroupReaderVersion.as_ref(),
+ reader_version,
+ );
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let before = rows(&table.read(&opts).await.unwrap());
+
+ // One tick below the latest commit: above the archival boundary (so it reads
+ // as pending, not as archived) and at or below the snapshot bound (so the
+ // as-of filter does not exclude it either). That is an in-flight commit.
+ let latest_ts = table
+ .get_timeline()
+ .get_latest_commit_timestamp()
+ .unwrap()
+ .to_string();
+ let orphan_instant = format!(
+ "{:0width$}",
+ latest_ts.parse::<u64>().unwrap() - 1,
+ width = latest_ts.len()
+ );
+ let orphan_instant = orphan_instant.as_str();
+ plant_orphan_base_file(&table_dir, orphan_instant);
+
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let after = rows(&table.read(&opts).await.unwrap());
+ println!(
+ "{label:34} reader version={reader_version} rows before={before} after={after} {}",
+ if after == before {
+ "OK (orphan excluded)"
+ } else {
+ "*** UNCOMMITTED ROWS VISIBLE ***"
+ }
+ );
+}
+
+#[ignore = "documents a divergence from Java: layout-v1 tables do not filter uncommitted files"]
Review Comment:
Running this with `--ignored` prints "OK (orphan excluded)" for all six combinations, so the reason on the `#[ignore]` no longer reproduces — which the module doc above says should be updated when it happens. Did the divergence get fixed somewhere in this change, or does it need a different fixture to show up?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader.rs:
##########
@@ -264,9 +430,13 @@ impl FileGroupReader {
/// selecting a version cannot turn a working read into a failing one. Each
/// reason is logged, because a fallback nobody can observe is
/// indistinguishable from a reader that is never used.
+ ///
+ /// Refusals are the exception, not the fallthrough: version 2 is the reader
Review Comment:
The paragraph just above ("Every reason here means version 1 serves the read instead, so selecting a version cannot turn a working read into a failing one") is contradicted by the CUSTOM branch below, which returns `Unsupported`. Since version 2 is now the default, a MOR table with `hoodie.record.merge.mode=CUSTOM` that reads today will start failing. Could the doc carve that case out, and the error message name `hoodie.read.file.group.reader.version=1` as the way back?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/mod.rs:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Context types for the merge-on-read file group reader.
+//!
+//! This module holds the inputs the MOR reader needs and the resolver that
+//! derives them from a table's configs. Nothing consumes it yet — the reader
Review Comment:
This doc and the file-scope `allow(dead_code)`s were written for the stacked PRs, but this is the change that wires the reader in, so they're now hiding real dead code in a live module. Dropping the 15 allows surfaces 38 warnings — `HoodieFileGroupReaderBuilder` and `ProjectingBatchReader` are never constructed, `RecordContext::partition_path` is threaded through three layers and never read, and the spill codec's `to_spill_bytes`/`from_spill_bytes` are unused. Worth clearing before merge while it's still cheap to tell intentional from accidental?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/Cargo.toml:
##########
@@ -101,12 +107,32 @@ percent-encoding = { workspace = true }
[dev-dependencies]
hudi-test = { path = "../test" }
serial_test = { workspace = true }
+# Also a regular dependency, but only an optional one behind `spill-rocksdb`.
+# Repeated here because tests outside the spill tier use it too, and must keep
+# building in the `--no-default-features` configuration.
tempfile = { workspace = true }
[lints.clippy]
result_large_err = "allow"
[features]
+default = ["spill-rocksdb"]
Review Comment:
The feature's rationale is that the Python wheel and the cxx bridge shouldn't have to build rocksdb, but it's default-on and `crates/hudi` takes `hudi-core` with default features, so both still do. Should the umbrella crate forward the feature so those two can opt out, or is the intent that they keep it and only external consumers opt out?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/record_merger.rs:
##########
@@ -0,0 +1,799 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.BufferedRecordMerger` and
+//! `org.apache.hudi.common.table.read.BufferedRecordMergerFactory`.
+//!
+//! The `BufferedRecordMerger` trait defines how records are merged during
+//! log scanning (delta merge: log-vs-log) and at read time (final merge:
+//! base-vs-log).
+
+use super::buffered_record::{BufferedRecord, DeleteRecord, OrderingValue};
+use crate::Result;
+
+/// Mirrors Java `OrderingValues.isCommitTimeOrderingValue` (`orderingValue == null
+/// || OrderingValues.isDefault(orderingValue)`, where `isDefault` ==
+/// `Integer(0).equals(orderingValue)`). A DELETE carrying the default ordering
+/// value is a "commit-time ordering delete" and wins unconditionally, regardless
+/// of the existing record's ordering value (mirrors
+/// `BufferedRecord.isCommitTimeOrderingDelete`).
+///
+/// The default is ONLY the null-coerced [`OrderingValue::Default`] sentinel (Java
+/// `Integer(0)`) or an absent (`None`) ordering. A GENUINE `Long(0)` field value
+/// is NOT default (GAP-2): `Integer(0).equals(Long(0))` is `false` in Java, so a
+/// delete carrying a real ordering value of `0` is ordering-compared, not treated
+/// as natural-order.
+fn is_default_ordering(ordering_value: &Option<OrderingValue>) -> bool {
+ matches!(ordering_value, None | Some(OrderingValue::Default))
+}
+
+/// Mirrors Java `BufferedRecord.isCommitTimeOrderingDelete`: a DELETE carrying
+/// the default ordering value. Such deletes win unconditionally (natural order).
+fn is_commit_time_ordering_delete(record: &BufferedRecord) -> bool {
+ record.is_delete() && is_default_ordering(&record.ordering_value)
+}
+
+/// Mirrors Java `OrderingValues.isSameClass`: ordering values are only compared
+/// when they are the same concrete type. Java throws on a cross-type
+/// `compareTo`; we instead skip the comparison (caller lets the newer/delete
+/// record win), avoiding the arbitrary cross-variant `Ord` on [`OrderingValue`].
+///
+/// Thin free-function wrapper over [`OrderingValue::is_same_class`] so the four
+/// buffer-side call sites below read as `is_same_class(a, b)`; the single source
+/// of truth is the method.
+fn is_same_class(a: &OrderingValue, b: &OrderingValue) -> bool {
+ a.is_same_class(b)
+}
+
+/// Trait for merging buffered records during the file group read pipeline.
+///
+/// Mirrors Java's `BufferedRecordMerger<T>` interface.
+///
+/// Consumed by `KeyBasedFileGroupRecordBuffer`:
+/// - `delta_merge` in `process_next_data_record` (log-vs-log within buffer)
+/// - `delta_merge_delete` in `process_next_deleted_record` (delete-vs-existing)
+/// - `final_merge` in `has_next_base_record` (base-vs-log at read time)
+pub trait BufferedRecordMerger: Send + Sync + std::fmt::Debug {
+ /// Merge a new log record with an existing buffered record.
+ ///
+ /// Returns `Some(merged)` if the record should be kept, `None` if dropped.
+ ///
+ /// Called during log scanning when a new record for the same key arrives.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>>;
+
+ /// Merge a delete record with an existing buffered record.
+ ///
+ /// Returns `Some(delete)` if the delete wins, `None` if the existing record survives.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>>;
+
+ /// Merge a base file record with a log record (final merge at read time).
+ ///
+ /// The `older_record` is from the base file, `newer_record` is from the log buffer.
+ fn final_merge(
+ &self,
+ older_record: &BufferedRecord,
+ newer_record: &BufferedRecord,
+ ) -> Result<BufferedRecord>;
+}
+
+/// Event-time based record merger.
+///
+/// Resolves conflicts by comparing ordering values: the record with the
+/// higher ordering value wins. This is the merge strategy for the
+/// `EVENT_TIME_ORDERING` mode only; `COMMIT_TIME_ORDERING` uses
+/// [`CommitTimeRecordMerger`] (last writer wins, no ordering comparison).
+///
+/// Created by `BufferedRecordMergerFactory` based on the merge mode. The read
+/// path now accepts `EVENT_TIME_ORDERING` (`buffer/loader.rs::get_record_buffer`),
+/// so this merger is live in production — it drives the scalar base-vs-log
+/// `final_merge`, and the vectorized `pick_winner` kernel mirrors its semantics.
+#[derive(Debug)]
+pub struct EventTimeRecordMerger;
+
+impl BufferedRecordMerger for EventTimeRecordMerger {
+ /// Mirrors Java `EventTimeRecordMerger.deltaMerge` via `shouldKeepNewerRecord`.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>> {
+ match existing_record {
+ None => Ok(Some(new_record.clone())),
+ Some(existing) if should_keep_newer_record(existing, new_record) => {
+ Ok(Some(new_record.clone()))
+ }
+ Some(existing) => Ok(Some(existing.clone())),
+ }
+ }
+
+ /// Mirrors Java `BufferedRecordMergerFactory.deltaMergeDeleteRecord`.
+ ///
+ /// The DELETE is obsolete (existing survives) ONLY when the existing record
+ /// has a strictly greater ordering value of the SAME type AND the delete
+ /// carries a non-default ordering value (`0` == natural order = always
+ /// delete). A default-ordering delete, or one against an existing record
+ /// that is itself a commit-time-ordering delete, follows Java's branches.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>> {
+ match existing_record {
+ None => Ok(Some(delete_record.clone())),
+ Some(existing) if is_commit_time_ordering_delete(existing) => Ok(None),
+ Some(existing) => {
+ let choose_existing = !is_default_ordering(&delete_record.ordering_value)
+ && match (&delete_record.ordering_value, &existing.ordering_value) {
+ (Some(del_val), Some(existing_val)) => {
+ is_same_class(del_val, existing_val) && existing_val > del_val
+ }
+ _ => false,
+ };
+ if choose_existing {
+ Ok(None) // existing record survives; DELETE is obsolete
+ } else {
+ Ok(Some(delete_record.clone()))
+ }
+ }
+ }
+ }
+
+ /// Mirrors Java `EventTimeRecordMerger.finalMerge`.
+ ///
+ /// The newer (log) record wins unless it is NOT a commit-time-ordering delete
+ /// and the older record (also not such a delete) has a strictly greater,
+ /// same-typed ordering value.
+ fn final_merge(
Review Comment:
Java's `finalMerge` delegates straight to `shouldKeepNewerRecord`; this re-derives the same rule inline. They agree today (I worked through the CTO-delete and cross-type cases), but it's one rule in two places. Any reason not to call `should_keep_newer_record(older_record, newer_record)` here?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/buffer/spillable_map.rs:
##########
@@ -0,0 +1,2627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Size-tracked, RocksDB-spillable merge map (A1, ENG-42993).
Review Comment:
There are a few dozen `ENG-XXXXX` ids and phase labels (A1, A2, A3c, A6e, B2, I-3, I-33, G-16) left in the source — e.g. this line, `record_context.rs:1790`, `read_stats.rs:72`. They don't resolve for anyone reading this on the ASF side. Could they be replaced with the behaviour they're standing in for, or a hudi-rs issue number?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3786603345
##########
crates/core/src/file_group/reader_v2/mod.rs:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Context types for the merge-on-read file group reader.
+//!
+//! This module holds the inputs the MOR reader needs and the resolver that
+//! derives them from a table's configs. Nothing consumes it yet — the reader
Review Comment:
Done — swept, and it found more than a lint.
Every file-scope `#![allow(dead_code)]` in `reader_v2` is gone (23 of them). Deleted outright:
- **The per-record spill codec.** `to_spill_bytes` / `from_spill_bytes`, `encode_`/`decode_ordering_value` and nine `SPILL_*` tags. Its doc claimed *"Used by `SpillableRecordMap` to store an entry in the on-disk (RocksDB) tier"* — it is not. `spillable_map` spills whole Arrow batches through `row_serde::to_binary_row_body` and keeps ordering values in the in-memory `DiskEntry` index. A superseded parallel implementation whose doc asserted the opposite. Eight tests went with it, including three I had added the round before — they were covering dead code.
- `ProjectingBatchReader`, constructed nowhere in the lib or under `cfg(test)`.
- `InputSplit::{start,length}` (always `0`/`-1`), `DeleteRecord::partition_path` (every construction site passed `String::new()`), `RecordPayload::into_record`, `BufferedRecords::from_engine_record`.
On your two specifically: `ProjectingBatchReader` is gone; `HoodieFileGroupReaderBuilder` turned out to be **used by the test harness**, which is why I checked rather than deleted — my first attempt removed it and broke the harness build.
What remains is 48 targeted allows, each naming why that item is kept, so the argument about any one of them can be had per item rather than per file. Mostly Java-parity surface reached only by the harness.
One thing worth flagging separately: `allow_inflight_instants` is plumbed from the loader into a field nothing reads, so the knob is **inert**. Nothing sets it true, so behaviour is correct today — I annotated it saying so rather than deleting it, since deleting would hide that the setting does nothing. Happy to make the builder refuse `true` instead if you'd prefer it loud.
`make check-rust` clean with zero dead-code warnings; 1267 lib tests green.
##########
crates/core/src/file_group/reader_v2/gold_tests.rs:
##########
@@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Reads each merge-on-read layout fixture and checks the result against the
+//! Spark snapshot shipped inside it.
+//!
+//! The fixtures carry their own `gold_data/` — a `SELECT *` of the table taken
+//! from Spark — so what is asserted here is agreement with Hudi's reference
+//! reader, not with this reader's own prior output.
+//!
+//! The file slice is discovered from the extracted fixture rather than written
+//! down per case: each of these tables is a single file group, so the base file
+//! and log files are whatever is on disk. A fixture that gains a file therefore
+//! cannot silently stop being covered.
+
+#![cfg(test)]
+
+use crate::config::HudiConfigs;
+use crate::config::read::HudiReadConfig;
+use crate::file_group::reader_v2::MAX_INSTANT_TIME;
+use crate::file_group::reader_v2::engine::HoodieFileGroupReader;
+use crate::file_group::reader_v2::input_split::InputSplit;
+use crate::file_group::reader_v2::reader_parameters::ReaderParameters;
+use crate::file_group::reader_v2::resolver::resolve_reader_context;
+use crate::storage::Storage;
+use arrow_array::RecordBatch;
+use std::path::Path;
+use std::sync::Arc;
+
+/// The base file and log files of a fixture's single file group, relative to
+/// the table root, plus the partition they live in.
+struct Slice {
+ base: Option<String>,
+ logs: Vec<String>,
+ partition: String,
+}
+
+/// Walk an extracted fixture and pick out its one file group.
+///
+/// Skips `.hoodie` (table metadata) and `gold_data` (the Spark snapshot this
+/// compares against, which is not table data).
+fn discover_slice(table_root: &Path) -> Slice {
+ fn walk(dir: &Path, root: &Path, out: &mut Vec<(String, String)>) {
+ let Ok(entries) = std::fs::read_dir(dir) else {
+ return;
+ };
+ for entry in entries.flatten() {
+ let path = entry.path();
+ let name = entry.file_name().to_string_lossy().to_string();
+ if path.is_dir() {
+ if name == ".hoodie" || name == "gold_data" {
+ continue;
+ }
+ walk(&path, root, out);
+ } else {
+ let rel = path
+ .strip_prefix(root)
+ .expect("walked path is under the table root")
+ .to_string_lossy()
+ .to_string();
+ let partition = Path::new(&rel)
+ .parent()
+ .map(|p| p.to_string_lossy().to_string())
+ .unwrap_or_default();
+ out.push((rel, partition));
+ }
+ }
+ }
+
+ let mut files = Vec::new();
+ walk(table_root, table_root, &mut files);
+
+ let mut base = None;
+ let mut logs = Vec::new();
+ let mut partition = String::new();
+ for (rel, part) in files {
+ let name = Path::new(&rel)
+ .file_name()
+ .map(|n| n.to_string_lossy().to_string())
+ .unwrap_or_default();
+ // Hadoop writes a `..<file>.crc` checksum sidecar next to each data
+ // file. It is not table data and must not reach the readers.
+ if name.ends_with(".crc") {
+ continue;
+ }
+ if name.ends_with(".parquet") && !name.starts_with('.') {
+ base = Some(rel);
+ partition = part;
+ } else if name.contains(".log.") {
+ logs.push(rel);
+ partition = part;
+ }
+ }
+ // Log files are appended in order; the scan relies on that ordering.
+ logs.sort();
+
+ Slice {
+ base,
+ logs,
+ partition,
+ }
+}
+
+/// Read a fixture's file group and return the merged rows.
+async fn read_fixture(table_path: &str) -> crate::Result<RecordBatch> {
+ let slice = discover_slice(Path::new(table_path));
+
+ // Load the table's own properties rather than inventing them: the merge
+ // semantics live in `hoodie.properties`, and inventing configs would test
+ // the reader against a table that does not exist.
+ let mut resolver = crate::table::builder::OptionResolver::new_with_options(
+ table_path,
+ [(
+ HudiReadConfig::EndTimestamp.as_ref(),
+ MAX_INSTANT_TIME.to_string(),
+ )],
+ );
+ resolver.resolve_options().await?;
+ let configs = HudiConfigs::new(resolver.hudi_options.clone());
+ let storage = Storage::new(
+ Arc::new(resolver.storage_options),
+ Arc::new(configs.clone()),
+ )?;
+
+ let has_logs = !slice.logs.is_empty();
+ let mut context = resolve_reader_context(&configs, has_logs)?;
+ // The fixtures are read whole, so nothing bounds the log scan.
+ context.instant_range = None;
+ context.rebuild_record_context(slice.partition.clone());
+
+ let input_split = InputSplit::new(slice.base, None, slice.logs, slice.partition);
+
+ // The table's own schema, the way a real read resolves it. Passing `None`
+ // here — as this harness used to — makes the reader fall back to the base
+ // file's schema, which is stale on any table whose columns were widened
+ // after that file was written. The fixtures are then read into the older
+ // shape and the comparison below, which only counts rows, cannot see it.
+ let table = crate::table::Table::new(table_path).await?;
+ let data_schema = Arc::new(table.data_schema_for_read().await?);
+
+ let mut reader = HoodieFileGroupReader::new(
+ Arc::new(context),
+ storage,
+ input_split,
+ ReaderParameters::default(),
+ Some(data_schema),
+ None,
+ )?;
+ reader.read().await
+}
+
+/// Fixtures the reader reproduces today.
+const GOLD_FIXTURES: &[&str] = &[
+ "table_log_only",
+ "table_log_compaction",
+ "table_parquet_log_block",
+ "table_partial_update",
+ "table_evo_add_col",
+ "table_evo_promotion",
+];
+
+/// Fixtures the reader does not read yet, with what stops each one.
+///
+/// Two causes remain, neither in the merge:
+///
+/// **Avro maps are modelled as a malformed Arrow type.** `avro_to_arrow` turns
+/// an Avro map into `Dictionary(Utf8, V)`, but an Arrow dictionary key must be
+/// an integer, so the array builder rejects it. Avro maps belong in Arrow's
+/// `Map` type. Changing that alters the schema of every table with a map
+/// column, including on the existing read path.
+///
+/// **The delete record's ordering value has two shapes in the wild.** Hudi
+/// moved it into per-type wrapper records — `IntWrapper`, `DecimalWrapper` and
+/// so on — because Avro forbids a union holding two branches of the same
+/// underlying type. This crate carries only the older primitive union, so a log
+/// file written the newer way resolves each branch index to the wrong type.
+///
+/// The two shapes cannot be told apart by decoding: a wrapper is a record whose
+/// single field is the primitive, so `IntWrapper` and a bare `long` encode to
+/// the same bytes. Only the branch index differs. Reading both therefore needs
+/// to key off the writer version rather than the payload, which is a design
+/// question rather than a decoding one — and swapping to the newer schema
+/// outright breaks the fixtures written the older way.
+///
+/// **A corrupt tail block is not recognised.** `LogFileReader` has no
+/// corrupt-block detection — `create_corrupted_block_if_needed` returns `None`
+/// — so it parses the trailing garbage as a block.
+const KNOWN_GAPS: &[(&str, &str)] = &[
Review Comment:
Deleted, agreeing with your read.
All three of its tests were subsumed: `merged_reads_match_the_spark_snapshot` covered 6 fixtures where `tests/gold_parity_tests.rs` covers 62 with a live ratchet; `known_gaps_still_fail` was `#[ignore]`d with all ten entries stale; and `fixtures_without_gold_are_present` is the same assertion `EXPECTED_WITHOUT_GOLD` already makes.
Your framing was the deciding argument — a second copy that cannot fail is worse than no copy, and its rotting data is exactly what that predicts.
##########
crates/core/src/file_group/reader_v2/buffer/spillable_map.rs:
##########
@@ -0,0 +1,2627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Size-tracked, RocksDB-spillable merge map (A1, ENG-42993).
Review Comment:
Done — zero `ENG-*` references left, plus the phase labels (`A1`, `A2`, `B2`) and three pointers to an internal design doc.
103 references, but only **15 distinct ids**, so each one meant a single thing and could be replaced consistently with the behaviour it stood for: `ENG-42992` → the lazy base-file source, `ENG-42993` → the spillable merge map, `ENG-43009` → the vectorized merge path, `ENG-38318` → event-time ordering, and so on. Where the id was a bare parenthetical the prose already stood on its own, so it just goes.
Worth admitting how it went: my first pass carried a whitespace-normalising rule that destroyed indentation across all 15 files — code bodies, Apache license headers, and doc-comment list continuations. `rustfmt` recovered the code, and I restored the rest by diffing each file against the previous commit and taking the original wherever only whitespace differed. It is clean and verified now, but it was self-inflicted and I would not script prose edits that way again.
##########
crates/core/src/file_group/reader_v2/record_merger.rs:
##########
@@ -0,0 +1,799 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.BufferedRecordMerger` and
+//! `org.apache.hudi.common.table.read.BufferedRecordMergerFactory`.
+//!
+//! The `BufferedRecordMerger` trait defines how records are merged during
+//! log scanning (delta merge: log-vs-log) and at read time (final merge:
+//! base-vs-log).
+
+use super::buffered_record::{BufferedRecord, DeleteRecord, OrderingValue};
+use crate::Result;
+
+/// Mirrors Java `OrderingValues.isCommitTimeOrderingValue` (`orderingValue == null
+/// || OrderingValues.isDefault(orderingValue)`, where `isDefault` ==
+/// `Integer(0).equals(orderingValue)`). A DELETE carrying the default ordering
+/// value is a "commit-time ordering delete" and wins unconditionally, regardless
+/// of the existing record's ordering value (mirrors
+/// `BufferedRecord.isCommitTimeOrderingDelete`).
+///
+/// The default is ONLY the null-coerced [`OrderingValue::Default`] sentinel (Java
+/// `Integer(0)`) or an absent (`None`) ordering. A GENUINE `Long(0)` field value
+/// is NOT default (GAP-2): `Integer(0).equals(Long(0))` is `false` in Java, so a
+/// delete carrying a real ordering value of `0` is ordering-compared, not treated
+/// as natural-order.
+fn is_default_ordering(ordering_value: &Option<OrderingValue>) -> bool {
+ matches!(ordering_value, None | Some(OrderingValue::Default))
+}
+
+/// Mirrors Java `BufferedRecord.isCommitTimeOrderingDelete`: a DELETE carrying
+/// the default ordering value. Such deletes win unconditionally (natural order).
+fn is_commit_time_ordering_delete(record: &BufferedRecord) -> bool {
+ record.is_delete() && is_default_ordering(&record.ordering_value)
+}
+
+/// Mirrors Java `OrderingValues.isSameClass`: ordering values are only compared
+/// when they are the same concrete type. Java throws on a cross-type
+/// `compareTo`; we instead skip the comparison (caller lets the newer/delete
+/// record win), avoiding the arbitrary cross-variant `Ord` on [`OrderingValue`].
+///
+/// Thin free-function wrapper over [`OrderingValue::is_same_class`] so the four
+/// buffer-side call sites below read as `is_same_class(a, b)`; the single source
+/// of truth is the method.
+fn is_same_class(a: &OrderingValue, b: &OrderingValue) -> bool {
+ a.is_same_class(b)
+}
+
+/// Trait for merging buffered records during the file group read pipeline.
+///
+/// Mirrors Java's `BufferedRecordMerger<T>` interface.
+///
+/// Consumed by `KeyBasedFileGroupRecordBuffer`:
+/// - `delta_merge` in `process_next_data_record` (log-vs-log within buffer)
+/// - `delta_merge_delete` in `process_next_deleted_record` (delete-vs-existing)
+/// - `final_merge` in `has_next_base_record` (base-vs-log at read time)
+pub trait BufferedRecordMerger: Send + Sync + std::fmt::Debug {
+ /// Merge a new log record with an existing buffered record.
+ ///
+ /// Returns `Some(merged)` if the record should be kept, `None` if dropped.
+ ///
+ /// Called during log scanning when a new record for the same key arrives.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>>;
+
+ /// Merge a delete record with an existing buffered record.
+ ///
+ /// Returns `Some(delete)` if the delete wins, `None` if the existing record survives.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>>;
+
+ /// Merge a base file record with a log record (final merge at read time).
+ ///
+ /// The `older_record` is from the base file, `newer_record` is from the log buffer.
+ fn final_merge(
+ &self,
+ older_record: &BufferedRecord,
+ newer_record: &BufferedRecord,
+ ) -> Result<BufferedRecord>;
+}
+
+/// Event-time based record merger.
+///
+/// Resolves conflicts by comparing ordering values: the record with the
+/// higher ordering value wins. This is the merge strategy for the
+/// `EVENT_TIME_ORDERING` mode only; `COMMIT_TIME_ORDERING` uses
+/// [`CommitTimeRecordMerger`] (last writer wins, no ordering comparison).
+///
+/// Created by `BufferedRecordMergerFactory` based on the merge mode. The read
+/// path now accepts `EVENT_TIME_ORDERING` (`buffer/loader.rs::get_record_buffer`),
+/// so this merger is live in production — it drives the scalar base-vs-log
+/// `final_merge`, and the vectorized `pick_winner` kernel mirrors its semantics.
+#[derive(Debug)]
+pub struct EventTimeRecordMerger;
+
+impl BufferedRecordMerger for EventTimeRecordMerger {
+ /// Mirrors Java `EventTimeRecordMerger.deltaMerge` via `shouldKeepNewerRecord`.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>> {
+ match existing_record {
+ None => Ok(Some(new_record.clone())),
+ Some(existing) if should_keep_newer_record(existing, new_record) => {
+ Ok(Some(new_record.clone()))
+ }
+ Some(existing) => Ok(Some(existing.clone())),
+ }
+ }
+
+ /// Mirrors Java `BufferedRecordMergerFactory.deltaMergeDeleteRecord`.
+ ///
+ /// The DELETE is obsolete (existing survives) ONLY when the existing record
+ /// has a strictly greater ordering value of the SAME type AND the delete
+ /// carries a non-default ordering value (`0` == natural order = always
+ /// delete). A default-ordering delete, or one against an existing record
+ /// that is itself a commit-time-ordering delete, follows Java's branches.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>> {
+ match existing_record {
+ None => Ok(Some(delete_record.clone())),
+ Some(existing) if is_commit_time_ordering_delete(existing) => Ok(None),
+ Some(existing) => {
+ let choose_existing = !is_default_ordering(&delete_record.ordering_value)
+ && match (&delete_record.ordering_value, &existing.ordering_value) {
+ (Some(del_val), Some(existing_val)) => {
+ is_same_class(del_val, existing_val) && existing_val > del_val
+ }
+ _ => false,
+ };
+ if choose_existing {
+ Ok(None) // existing record survives; DELETE is obsolete
+ } else {
+ Ok(Some(delete_record.clone()))
+ }
+ }
+ }
+ }
+
+ /// Mirrors Java `EventTimeRecordMerger.finalMerge`.
+ ///
+ /// The newer (log) record wins unless it is NOT a commit-time-ordering delete
+ /// and the older record (also not such a delete) has a strictly greater,
+ /// same-typed ordering value.
+ fn final_merge(
Review Comment:
Done — `final_merge` now delegates, matching Java's `finalMerge`.
I worked the case analysis first rather than trusting that they looked the same. `final_merge` returned the older record iff `!ctod(newer) && !ctod(older) && same_class && old > new`; `should_keep_newer_record` returns false under exactly that condition — the `new >= old` negation lines up. So:
```rust
if should_keep_newer_record(older_record, newer_record) {
Ok(newer_record.clone())
} else {
Ok(older_record.clone())
}
```
The gold corpus is green behind it: 62 fixtures and 183 option cases against Spark's output, plus the position-merge sweep. Thanks for confirming the CTO-delete and cross-type cases — that made this a rename rather than an investigation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788530913
##########
crates/core/src/schema/delete.rs:
##########
@@ -37,9 +37,54 @@ static DELETE_RECORD_AVRO_SCHEMA_IN_JSON: Lazy<Result<JsonValue>> = Lazy::new(||
.map_err(|e| CoreError::Schema(format!("Failed to parse schema to JSON: {e}")))
});
+/// Union position of `ArrayWrapper`, which carries a list rather than a scalar.
+/// Nothing orders records by a list, so it is rejected rather than mapped.
+const ARRAY_WRAPPER_POSITION: u32 = 12;
+
+/// Replace a wrapped ordering value with the primitive inside it.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records — `LongWrapper`
+/// is a record whose single `value` field is a `long`. The Arrow side wants the
+/// primitive, and [`avro_schema_for_delete_record`] narrows the schema to match,
+/// so the value has to be unwrapped to agree with it.
+///
+/// The narrowed schema has two branches, `[null, <primitive>]`, so the surviving
+/// branch is position 1 regardless of where the wrapper sat in the full union.
+///
+/// Records that are already primitives pass through: this runs over decoded
+/// values, and only the wrapper shape is rewritten.
+pub fn unwrap_ordering_value(delete_record: AvroValue) -> Result<AvroValue> {
+ let AvroValue::Record(mut fields) = delete_record else {
+ return Err(CoreError::Schema(
+ "Expected a record for delete record".to_string(),
+ ));
+ };
+ let Some((_, ordering_val)) = fields.get_mut(2) else {
+ return Err(CoreError::Schema(
+ "Delete record has no orderingVal field".to_string(),
+ ));
+ };
+ if let AvroValue::Union(pos, inner) = ordering_val {
+ if *pos == ARRAY_WRAPPER_POSITION {
Review Comment:
Correction: rather than degrading, the union column now passes through whole and `delete_batch_to_keys_with_ordering` decodes ArrayWrapper per row into the composite ordering value the merge compares, so a stale composite delete loses correctly, with regression tests for the decode and the merge outcome.
##########
crates/core/src/file_group/log_file/content.rs:
##########
@@ -17,35 +17,128 @@
* under the License.
*/
use crate::Result;
-use crate::avro_to_arrow::arrow_array_reader::AvroArrowArrayReader;
use crate::config::HudiConfigs;
use crate::error::CoreError;
-use crate::file_group::log_file::avro::AvroDataBlockContentReader;
+use crate::file_group::log_file::avro::AvroBlockDecoder;
use crate::file_group::log_file::log_block::{
BlockMetadataKey, BlockType, LogBlockContent, LogBlockVersion,
};
use crate::file_group::log_file::log_format::LogFormatVersion;
use crate::file_group::record_batches::RecordBatches;
use crate::hfile::{HFileReader, HFileRecord};
-use crate::schema::delete::{avro_schema_for_delete_record, avro_schema_for_delete_record_list};
-use apache_avro::{Schema as AvroSchema, from_avro_datum};
+use crate::schema::delete::delete_record_list_schema_json;
+use crate::schema::extended_promotion::record_needs_rewrite_for_extended_promotion;
+use crate::schema::parquet_list_norm::normalize_parquet_metadata;
+use crate::schema::resolver::avro_json_to_arrow_schema;
+use crate::storage::RowFilterBuilder;
+use arrow_array::{Array, ArrayRef, ListArray, RecordBatch, StructArray, UnionArray};
+use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
-use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::file::metadata::ParquetMetaDataReader;
use std::collections::HashMap;
use std::io::{Read, Seek};
use std::sync::Arc;
+/// Turn the wrapped ordering values into a plain column.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records, so a decode
+/// against that schema yields a union of one-field structs. The merge wants the
+/// value itself.
+///
+/// A block writes one ordering type, so exactly one branch is populated; that
+/// branch's `value` child is the column. A block mixing branches is rejected
+/// rather than silently reduced to one of them.
+fn unwrap_ordering_values(ordering: &ArrayRef) -> Result<ArrayRef> {
+ let union = ordering
+ .as_any()
+ .downcast_ref::<UnionArray>()
+ .ok_or_else(|| {
+ CoreError::LogBlockError(format!(
+ "Expected orderingVal to be a union, got {}",
+ ordering.data_type()
+ ))
+ })?;
+
+ let mut active: Option<i8> = None;
+ for i in 0..union.len() {
+ let type_id = union.type_id(i);
+ match active {
+ None => active = Some(type_id),
+ Some(seen) if seen == type_id => {}
+ Some(seen) => {
+ return Err(CoreError::LogBlockError(format!(
+ "Delete block mixes ordering types (union branches {seen} and {type_id})"
+ )));
+ }
+ }
+ }
+ let Some(active) = active else {
+ return Ok(ordering.clone());
+ };
+
+ let child = union.child(active);
+ // Null is a branch like any other; there is nothing to unwrap out of it.
Review Comment:
Correction to the above: ArrayWrapper rows now decode to composite ordering values (union passthrough) rather than natural order; null branches still decode as natural-order deletes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788042170
##########
crates/core/src/file_group/reader_v2/log_record_reader.rs:
##########
@@ -0,0 +1,1805 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+
+//! Mirrors `org.apache.hudi.common.table.log.BaseHoodieLogRecordReader`.
+//!
+//! The 2-pass log scanning engine that reads log blocks, filters them through
+//! 5 gates, resolves compaction, and dispatches to the record buffer.
+//!
+//! ## 3-pass algorithm (matching Java's `scanInternal`):
+//!
+//! **Pass 1 — Forward scan with 5 gates:**
+//! - Gate 1: Corrupt → skip
+//! - Gate 2: Future (instant > latestInstantTime) → skip
+//! - Gate 3: Completed/inflight → skip uncommitted/inflight data/delete blocks (table version < 8;
+//! applied only when the completion sets are supplied)
+//! - Gate 4: Instant range filter → skip
+//! - Rollback command → remove target instant
+//! - Data/Delete → add to `instant_to_blocks_map`
+//!
+//! **Pass 2 — Reverse iteration with compaction resolution:**
+//! Iterate instants newest→oldest, `Collections.reverse(logBlocks)` then `addLast`.
+//! Result: deque has latest-first at head, earliest-last at tail.
+//!
+//! **Pass 3 — `processQueuedBlocksForInstant`:**
+//! Drain deque via `pollLast` (tail-first) = oldest instant processed first.
+
+use crate::Result;
+use crate::file_group::log_file::log_block::{BlockMetadataKey, BlockType, LogBlock};
+use crate::file_group::log_file::reader::LogFileReader;
+use crate::file_group::reader_v2::buffer::HoodieFileGroupRecordBuffer;
+use crate::file_group::reader_v2::profiling::profile_once;
+use crate::file_group::reader_v2::reader_context::{CompletionGateInputs, ReaderContext};
+use crate::storage::Storage;
+use crate::timeline::selector::InstantRange;
+use std::collections::{HashMap, HashSet, VecDeque};
+use std::sync::Arc;
+
+// =========================================================================
+// Pass 1 result
+// =========================================================================
+
+/// Output of Pass 1 (forward scan with 5 gates).
+#[derive(Debug)]
+pub struct Pass1Result {
+ /// Instants in first-seen order (no duplicates).
+ pub ordered_instants_list: Vec<String>,
+ /// instant time → blocks for that instant (in file-read order).
+ pub instant_to_blocks_map: HashMap<String, Vec<LogBlock>>,
+ /// Stats
+ pub total_log_blocks: u64,
+ pub total_corrupt_blocks: u64,
+ pub total_rollbacks: u64,
+}
+
+/// Borrowed view of [`CompletionGateInputs`] used inside a single scan pass.
+pub struct CompletionGate<'a> {
+ completed_instants: &'a HashSet<String>,
+ inflight_instants: &'a HashSet<String>,
+ archived_boundary: Option<&'a str>,
+}
+
+impl<'a> CompletionGate<'a> {
+ /// Build a borrowing gate from owned inputs.
+ pub fn new(inputs: &'a CompletionGateInputs) -> Self {
+ Self {
+ completed_instants: &inputs.completed_instants,
+ inflight_instants: &inputs.inflight_instants,
+ archived_boundary: inputs.archived_boundary.as_deref(),
+ }
+ }
+
+ /// Whether a data/delete block at `instant_time` is admitted (committed and not inflight).
+ ///
+ /// Mirrors `TimelineView::is_committed` (completed-set membership OR older than the archived
+ /// boundary) plus the explicit inflight exclusion, matching Java's scanInternalV1 filter.
+ ///
+ /// # No-positive-information fallback
+ ///
+ /// When the gate carries NO positive completion information — an empty
+ /// `completed_instants` set AND no `archived_boundary` — it cannot establish
+ /// committedness for any instant, so it admits everything (a no-op) rather
+ /// than excluding all blocks. Java always builds the gate from a non-empty
+ /// active timeline (`filterCompletedInstants()`), so an empty set here means
+ /// the inputs were not populated by the caller (e.g. the FFI bridge forwarded
+ /// an empty list). Excluding on that basis would silently drop EVERY log delta
+ /// — including committed ones — and return base-file-only data (the
+ /// C-INFLIGHT silent-wrong: a committed later delta wrongly dropped). Deferring
+ /// to the other gates preserves the pre-gate behavior for a mis-wired gate; a
+ /// correctly-populated gate is unaffected because its completed set is non-empty.
+ fn admits(&self, instant_time: &str) -> bool {
+ if self.is_unpopulated() {
+ return true;
+ }
+ let is_committed = self.completed_instants.contains(instant_time)
+ || self
+ .archived_boundary
+ .is_some_and(|boundary| instant_time < boundary);
+ is_committed && !self.inflight_instants.contains(instant_time)
+ }
+
+ /// Whether the gate carries NO positive completion information — an empty
+ /// `completed_instants` set AND no `archived_boundary`. Such a gate cannot establish
+ /// committedness for any instant, so [`admits`](Self::admits) degrades to admit-all.
+ ///
+ /// When the gate was supplied deliberately (the caller passes `Some` only for a table
+ /// version < 8 snapshot), being unpopulated is a mis-wire signal, not a genuine empty
+ /// timeline: Java always builds the gate from a non-empty active timeline
+ /// (`filterCompletedInstants()`), and a v1 snapshot always has >= 1 completed instant.
+ /// `forward_scan_pass1` warns once when this holds so a gate whose inputs were dropped
+ /// across the FFI boundary is diagnosable rather than a silent no-op.
+ fn is_unpopulated(&self) -> bool {
+ self.completed_instants.is_empty() && self.archived_boundary.is_none()
+ }
+}
+
+/// Run Pass 1: forward scan with 5 gates.
+///
+/// Processes a flat list of blocks (from all log files, in file-read order)
+/// and classifies them into `instant_to_blocks_map` + `ordered_instants_list`.
+///
+/// ## 5 Gates (per-block):
+/// 1. Corrupt → skip
+/// 2. Future (instant > latest_instant_time) → skip
+/// 3. Completed/inflight → skip data/delete blocks whose instant is uncommitted or inflight
+/// (C-INFLIGHT-DELTA). Applied only when `completion_gate` is `Some` — callers pass it for
+/// table version < 8 and leave it `None` for v8+ (and when no timeline is available), which
+/// preserves the prior behavior of admitting such blocks.
+/// 4. Instant range → skip if not in range
+/// 5. Rollback command → remove target instant from maps
+pub fn forward_scan_pass1(
+ all_blocks: Vec<LogBlock>,
+ latest_instant_time: &str,
+ instant_range: &Option<InstantRange>,
+ timezone: &str,
+ completion_gate: Option<&CompletionGate>,
+) -> Result<Pass1Result> {
+ let mut target_rollback_instants: HashSet<String> = HashSet::new();
+ let mut instant_to_blocks_map: HashMap<String, Vec<LogBlock>> = HashMap::new();
+ let mut ordered_instants_list: Vec<String> = Vec::new();
+ let mut total_log_blocks: u64 = 0;
+ let mut total_corrupt_blocks: u64 = 0;
+ let mut total_rollbacks: u64 = 0;
+
+ log::debug!(
+ "[Pass1] forward_scan: {} total blocks, latest_instant_time={}, has_instant_range={}",
+ all_blocks.len(),
+ latest_instant_time,
+ instant_range.is_some(),
+ );
+
+ // The completion gate was supplied (table version < 8 snapshot) but carries no positive
+ // completion info — its inputs were not populated across the FFI boundary. Rather than
+ // silently degrade to admit-all (Gate 3 becomes a no-op, so a straddling uncommitted delta
+ // could be merged), warn once so the mis-wire is diagnosable. Behavior is unchanged (still
+ // fail-open); this only makes the condition visible.
+ if completion_gate.is_some_and(CompletionGate::is_unpopulated) {
+ log::warn!(
+ "[Pass1] completion gate supplied but unpopulated (empty completed set, no archived \
+ boundary): Gate 3 (C-INFLIGHT) admits all delta blocks. A table version < 8 snapshot \
+ always has >= 1 completed instant, so this indicates the gate inputs were dropped \
+ across the FFI boundary. latest_instant_time={latest_instant_time}"
+ );
+ }
+
+ for block in all_blocks {
+ total_log_blocks += 1;
+
+ // Gate 1: Corrupt blocks → skip
+ if block.block_type == BlockType::Corrupted {
+ log::debug!("[Pass1] Gate1: corrupt block #{total_log_blocks} skipped");
+ total_corrupt_blocks += 1;
+ continue;
+ }
+
+ let instant_time = match block.instant_time() {
+ Ok(t) => t.to_string(),
+ Err(_) => {
+ log::debug!("[Pass1] block #{total_log_blocks} has no instant time, skipping");
+ continue;
+ }
+ };
+
+ // Gate 2: Future blocks → skip (instant > latestInstantTime)
+ if block.block_type != BlockType::Command && instant_time.as_str() > latest_instant_time {
+ log::debug!(
+ "[Pass1] Gate2: future block #{total_log_blocks} instant={instant_time} > {latest_instant_time}, skipped"
+ );
+ continue;
+ }
+
+ // Gate 3: Completed/inflight check (Java scanInternalV1, tableVersion < 8).
+ // Applied only when the caller supplies the timeline sets (None for v8+ or
+ // when no timeline is available). A data/delete block whose instant is not
+ // committed, or is inflight, is skipped -- without this an uncommitted
+ // "straddling" instant (time < latest completed, no in-log rollback block)
+ // is wrongly merged (C-INFLIGHT-DELTA). Command (rollback) blocks are never
+ // gated here so they can still remove their target instant below.
+ if block.block_type != BlockType::Command
+ && let Some(gate) = completion_gate
+ && !gate.admits(&instant_time)
+ {
+ log::debug!(
+ "[Pass1] Gate3: block #{total_log_blocks} instant={instant_time} uncommitted/inflight, skipped"
+ );
+ continue;
+ }
+
+ // Gate 4: Instant range filter (not for command blocks)
+ if block.block_type != BlockType::Command
+ && let Some(range) = instant_range
+ && range.not_in_range(&instant_time, timezone)?
+ {
+ log::debug!(
+ "[Pass1] Gate4: block #{total_log_blocks} instant={instant_time} out of range, skipped"
+ );
+ continue;
+ }
+
+ log::debug!(
+ "[Pass1] block #{total_log_blocks} passed all gates: type={:?} instant={instant_time}",
+ block.block_type,
+ );
+
+ // Classify the block
+ match block.block_type {
+ BlockType::AvroData | BlockType::ParquetData | BlockType::Delete => {
+ let blocks_list = instant_to_blocks_map
+ .entry(instant_time.clone())
+ .or_default();
+ if blocks_list.is_empty() {
+ ordered_instants_list.push(instant_time.clone());
+ }
+ blocks_list.push(block);
+ }
+ BlockType::Command if block.is_rollback_block() => {
+ total_rollbacks += 1;
+ if let Ok(target) = block.target_instant_time() {
+ let target = target.to_string();
+ log::debug!("[Pass1] ROLLBACK: removing instant={target}");
+ target_rollback_instants.insert(target.clone());
+ ordered_instants_list.retain(|t| t != &target);
+ instant_to_blocks_map.remove(&target);
+ }
+ }
+ _ => {}
+ }
+ }
+
+ log::debug!(
+ "[Pass1] complete: ordered_instants={ordered_instants_list:?} \
+ total_blocks={total_log_blocks} corrupt={total_corrupt_blocks} \
+ rollbacks={total_rollbacks}",
+ );
+ for (instant, blocks) in &instant_to_blocks_map {
+ log::debug!(
+ "[Pass1] instant={instant}: {} block(s) [{:?}]",
+ blocks.len(),
+ blocks
+ .iter()
+ .map(|b| format!("{:?}", b.block_type))
+ .collect::<Vec<_>>(),
+ );
+ }
+
+ Ok(Pass1Result {
+ ordered_instants_list,
+ instant_to_blocks_map,
+ total_log_blocks,
+ total_corrupt_blocks,
+ total_rollbacks,
+ })
+}
+
+// =========================================================================
+// Pass 2 result
+// =========================================================================
+
+/// Output of Pass 2 (reverse iteration with compaction resolution).
+#[derive(Debug)]
+pub struct Pass2Result {
+ /// The final ordered deque of blocks to be processed.
+ /// Latest instant at head (front), earliest at tail (back).
+ /// Drain via `pop_back` for oldest-first processing.
+ pub current_instant_log_blocks: VecDeque<LogBlock>,
+ /// Dedup guard — instants whose blocks are enqueued.
+ /// Recorded for parity with Java's scan result; nothing reads it back here.
+ #[allow(dead_code)]
+ pub instant_times_included: HashSet<String>,
+ /// Which instant times actually contributed blocks (ordered output).
+ pub valid_block_instants: Vec<String>,
+}
+
+/// Run Pass 2: reverse iteration with compaction resolution.
+///
+/// Iterates `ordered_instants_list` from newest→oldest, applying:
+/// - `Collections.reverse(logBlocks)` then `addLast` (Rust: `reverse()` + `push_back`)
+/// - Compaction: if block has `COMPACTED_BLOCK_TIMES` header, register mapping
+/// - Deduplication: `instant_times_included` prevents double-enqueue
+///
+/// ## Invariants (from FS_logFileReadInputs_fileOrderRequestTime.md):
+///
+/// **Invariant 1**: `instant_times_included` and `valid_block_instants` contain
+/// exactly the same instant strings (Set vs List).
+///
+/// **Invariant 2**: Every instant in `instant_times_included`/`valid_block_instants`
+/// has its blocks in `current_instant_log_blocks`, and vice versa.
+///
+/// **Invariant 3**: Each instant appears in `instant_times_included` at most once (dedup).
+///
+/// **Invariant 4**: `current_instant_log_blocks` is ordered latest-first (reverse chronological).
+/// Drain via `pop_back` produces oldest-first processing order.
+pub fn reverse_scan_pass2(pass1: &mut Pass1Result) -> Pass2Result {
+ log::debug!(
+ "[Pass2] reverse_scan: {} instants to process (newest→oldest)",
+ pass1.ordered_instants_list.len(),
+ );
+
+ let mut current_instant_log_blocks: VecDeque<LogBlock> = VecDeque::new();
+ let mut instant_times_included: HashSet<String> = HashSet::new();
+ let mut valid_block_instants: Vec<String> = Vec::new();
+ let mut block_time_to_compaction_block_time_map: HashMap<String, String> = HashMap::new();
+
+ for i in (0..pass1.ordered_instants_list.len()).rev() {
+ let instant_time = &pass1.ordered_instants_list[i];
+ let instants_blocks = match pass1.instant_to_blocks_map.get(instant_time) {
+ Some(blocks) if !blocks.is_empty() => blocks,
+ _ => continue,
+ };
+
+ let first_block = &instants_blocks[0];
+
+ // Check for compacted blocks (has COMPACTED_BLOCK_TIMES header)
+ if let Some(compacted_times) = first_block
+ .header
+ .get(&BlockMetadataKey::CompactedBlockTimes)
+ {
+ for original_instant in compacted_times.split(',') {
+ let original_instant = original_instant.trim().to_string();
+ let final_instant = block_time_to_compaction_block_time_map
+ .get(instant_time)
+ .cloned()
+ .unwrap_or_else(|| instant_time.clone());
+ block_time_to_compaction_block_time_map.insert(original_instant, final_instant);
+ }
+ } else {
+ // Normal data block — check if it's been compacted
+ let compacted_final = block_time_to_compaction_block_time_map.get(instant_time);
+
+ if let Some(final_instant) = compacted_final {
+ let final_instant = final_instant.clone();
+ if instant_times_included.contains(&final_instant) {
+ continue;
+ }
+ if let Some(blocks) = pass1.instant_to_blocks_map.get(&final_instant) {
+ let mut reversed = blocks.clone();
+ reversed.reverse();
+ for block in reversed {
+ current_instant_log_blocks.push_back(block);
+ }
+ }
+ instant_times_included.insert(final_instant.clone());
+ valid_block_instants.push(final_instant);
+ } else {
+ // Not compacted — add blocks directly
+ // Java: Collections.reverse(logBlocks) then forEach(addLast)
+ let blocks = pass1
+ .instant_to_blocks_map
+ .get(instant_time)
+ .cloned()
+ .unwrap_or_default();
+ let mut reversed = blocks;
+ reversed.reverse();
+ for block in reversed {
+ current_instant_log_blocks.push_back(block);
+ }
+ instant_times_included.insert(instant_time.clone());
+ valid_block_instants.push(instant_time.clone());
+ }
+ }
+ }
+
+ log::debug!(
+ "[Pass2] complete: {} blocks in deque, valid_instants={:?}",
+ current_instant_log_blocks.len(),
+ valid_block_instants,
+ );
+
+ Pass2Result {
+ current_instant_log_blocks,
+ instant_times_included,
+ valid_block_instants,
+ }
+}
+
+// =========================================================================
+// BaseHoodieLogRecordReader
+// =========================================================================
+
+/// The base log record reader implementing the 3-pass scanning algorithm.
+///
+/// Mirrors Java's `BaseHoodieLogRecordReader<T>` (abstract base class).
+///
+/// ## Java hierarchy:
+/// ```text
+/// BaseHoodieLogRecordReader<T> (abstract — this struct)
+/// └─ HoodieMergedLogRecordReader<T> (concrete — wraps this via composition)
+/// ```
+///
+/// ## Fields matching Java:
+/// - `reader_context` ↔ `readerContext`
+/// - `storage` ↔ `storage`
+/// - `log_file_paths` ↔ `logFiles`
+/// - `latest_instant_time` ↔ `latestInstantTime`
+/// - `instant_range` ↔ `instantRange`
+/// - `force_full_scan` ↔ `forceFullScan`
+/// - `record_buffer` ↔ `recordBuffer`
+/// - `allow_inflight_instants` ↔ `allowInflightInstants`
+pub struct BaseHoodieLogRecordReader {
+ pub reader_context: Arc<ReaderContext>,
+ pub storage: Arc<Storage>,
+ pub log_file_paths: Vec<String>,
+ pub latest_instant_time: String,
+ /// Mirrors Java's `private final Option<InstantRange> instantRange`.
+ pub instant_range: Option<InstantRange>,
+ /// Mirrors Java's `protected final boolean forceFullScan`.
+ /// When true, scanning happens eagerly in the constructor.
+ pub force_full_scan: bool,
+ pub record_buffer: Box<dyn HoodieFileGroupRecordBuffer>,
+ /// Carried for parity. Nothing reads it, so it is inert — the completion gate
+ /// below is what actually decides whether an inflight instant is admitted,
+ /// and `ReaderParameters` never sets this true.
+ #[allow(dead_code)]
+ pub allow_inflight_instants: bool,
+ /// Inputs for the Gate-3 completed/inflight check (C-INFLIGHT-DELTA). `Some` only for
+ /// table version < 8 (v1 timeline layout); `None` for v8+ and when no timeline is available,
+ /// in which case Gate 3 is a no-op. Populated by the builder / FFI wiring from the active
+ /// timeline (completed + inflight instant sets + the first active instant).
+ pub completion_gate_inputs: Option<CompletionGateInputs>,
+
+ // ── Stats / state (mirrors Java's AtomicLong counters + progress) ──
+ pub valid_block_instants: Vec<String>,
+ pub total_log_files: u64,
+ pub total_log_blocks: u64,
+ pub total_log_records: u64,
+ pub total_corrupt_blocks: u64,
+ pub total_rollbacks: u64,
+ /// Mirrors Java's `private float progress` (0.0 → 1.0).
+ pub progress: f32,
+
+ // ── Stage timings (perf harness) ──────────────────────
+ /// Wall ms spent reading log-block metadata + bytes off storage (Pass 1).
+ pub log_block_read_ms: u64,
+ /// Wall ms spent in Pass-3 block dispatch (inflate/decode + merge insert).
+ pub merge_insert_ms: u64,
+}
+
+impl BaseHoodieLogRecordReader {
+ /// Mirrors Java's `scanInternal(Option<KeySpec> keySpecOpt, boolean skipProcessingBlocks)`.
+ ///
+ /// # Arguments
+ /// - `skip_processing_blocks` — when `true`, Pass 3 (block processing) is skipped.
+ /// Java's `HoodieMergedLogRecordReader.scan(true)` uses this to collect block
+ /// metadata without actually merging records.
+ pub async fn scan_internal(&mut self, skip_processing_blocks: bool) -> Result<()> {
+ // Reset state (mirrors Java: currentInstantLogBlocks = new ArrayDeque<>(), progress = 0.0f, ...)
+ self.valid_block_instants.clear();
+ self.progress = 0.0;
+ self.total_log_files = 0;
+ self.total_log_blocks = 0;
+ self.total_log_records = 0;
+ self.total_corrupt_blocks = 0;
+ self.total_rollbacks = 0;
+
+ let timezone = self.reader_context.timezone();
+
+ // Read all blocks as metadata-only (no content decoding).
Review Comment:
This comment describes the lazy sweep, but the loop below uses the eager whole-file reader — the lazy tier (`new_streaming`, `read_all_blocks_metadata_only`, `LogBlockFetcher`/`DeferredContent`, and the `hoodie.memory.dfs.buffer.max.size` window knob) has no production caller, only the `#[cfg(test)]` memory_bench; the one production `load_content` call in Pass 3 no-ops because content is always already read. Since the range here is deliberately unbounded (rollback command blocks must always process), eager decode means out-of-window and rolled-back blocks pay a full Arrow decode before the gates discard them — v1 skips those at the header. Is wiring the sweep in a follow-up, or should the tier and the knob wait until it lands?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/log_file/content.rs:
##########
@@ -17,35 +17,128 @@
* under the License.
*/
use crate::Result;
-use crate::avro_to_arrow::arrow_array_reader::AvroArrowArrayReader;
use crate::config::HudiConfigs;
use crate::error::CoreError;
-use crate::file_group::log_file::avro::AvroDataBlockContentReader;
+use crate::file_group::log_file::avro::AvroBlockDecoder;
use crate::file_group::log_file::log_block::{
BlockMetadataKey, BlockType, LogBlockContent, LogBlockVersion,
};
use crate::file_group::log_file::log_format::LogFormatVersion;
use crate::file_group::record_batches::RecordBatches;
use crate::hfile::{HFileReader, HFileRecord};
-use crate::schema::delete::{avro_schema_for_delete_record, avro_schema_for_delete_record_list};
-use apache_avro::{Schema as AvroSchema, from_avro_datum};
+use crate::schema::delete::delete_record_list_schema_json;
+use crate::schema::extended_promotion::record_needs_rewrite_for_extended_promotion;
+use crate::schema::parquet_list_norm::normalize_parquet_metadata;
+use crate::schema::resolver::avro_json_to_arrow_schema;
+use crate::storage::RowFilterBuilder;
+use arrow_array::{Array, ArrayRef, ListArray, RecordBatch, StructArray, UnionArray};
+use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
-use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::file::metadata::ParquetMetaDataReader;
use std::collections::HashMap;
use std::io::{Read, Seek};
use std::sync::Arc;
+/// Turn the wrapped ordering values into a plain column.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records, so a decode
+/// against that schema yields a union of one-field structs. The merge wants the
+/// value itself.
+///
+/// A block writes one ordering type, so exactly one branch is populated; that
+/// branch's `value` child is the column. A block mixing branches is rejected
+/// rather than silently reduced to one of them.
+fn unwrap_ordering_values(ordering: &ArrayRef) -> Result<ArrayRef> {
+ let union = ordering
+ .as_any()
+ .downcast_ref::<UnionArray>()
+ .ok_or_else(|| {
+ CoreError::LogBlockError(format!(
+ "Expected orderingVal to be a union, got {}",
+ ordering.data_type()
+ ))
+ })?;
+
+ let mut active: Option<i8> = None;
+ for i in 0..union.len() {
+ let type_id = union.type_id(i);
+ match active {
+ None => active = Some(type_id),
+ Some(seen) if seen == type_id => {}
+ Some(seen) => {
+ return Err(CoreError::LogBlockError(format!(
+ "Delete block mixes ordering types (union branches {seen} and {type_id})"
Review Comment:
Null is a union branch too, so a delete block over a nullable ordering column where some deleted rows carry null mixes branch 0 with a wrapper branch and fails here. Java resolves the branch per record and treats a null as the default ordering, so the delete wins naturally. Not a regression — the old per-record narrowing couldn't read this shape either — but the message reads as if two value types mixed, when null+value is the realistic case. Could the null branch be exempted from the mixing check and those slots treated as default-ordering deletes?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3786608594
##########
crates/core/tests/reader_concurrency_tests.rs:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Reader-vs-writer concurrency repros.
+//!
+//! Most of these are `#[ignore]`d, for one of two reasons: they document a
+//! divergence from Hudi's Java reader that is not fixed yet, or they are probes
+//! that print what the reader does without asserting anything. Neither belongs
+//! in the suite — the first would fail it, and the second would pass whatever
+//! happened. Run them with
+//! `cargo test -p hudi-core --test reader_concurrency_tests -- --ignored --nocapture`.
+//!
+//! A divergence that gets fixed should leave this file, or become an ordinary
+//! test that pins the fixed behavior. An `#[ignore]` reason describing something
+//! that no longer happens is worse than no note at all.
+
+use hudi_core::table::{ReadOptions, Table};
+use std::path::{Path, PathBuf};
+
+fn rows(batches: &[arrow_array::RecordBatch]) -> usize {
+ batches.iter().map(|b| b.num_rows()).sum()
+}
+
+/// Copy an existing base file under a NEW file id at `instant`, leaving no
+/// commit for that instant — a writer that wrote its data file and then died.
+fn plant_orphan_base_file(table_dir: &Path, instant: &str) -> PathBuf {
+ let existing = std::fs::read_dir(table_dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.extension().is_some_and(|x| x == "parquet"))
+ .expect("fixture has a base file to copy");
+ let orphan = table_dir.join(format!(
+ "b179bdb3-731c-4894-b855-abfcd6921008-0_0-1-1_{instant}.parquet"
+ ));
+ std::fs::copy(&existing, &orphan).unwrap();
+ orphan
+}
+
+fn fixture_zip(relative: &str) -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR").replace("/core", "/test")).join(relative)
+}
+
+async fn probe(zip: &str, label: &str) {
+ probe_with_reader_version(zip, label, "1").await;
+ probe_with_reader_version(zip, label, "2").await;
+}
+
+async fn probe_with_reader_version(zip: &str, label: &str, reader_version: &str) {
+ use hudi_core::config::read::HudiReadConfig;
+ let dir = hudi_test::extract_test_table_fresh(&fixture_zip(zip));
+ let table_dir = std::fs::read_dir(&dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.is_dir() && p.join(".hoodie").exists())
+ .expect("extracted fixture holds one table dir");
+
+ let opts = ReadOptions::new().with_hudi_option(
+ HudiReadConfig::FileGroupReaderVersion.as_ref(),
+ reader_version,
+ );
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let before = rows(&table.read(&opts).await.unwrap());
+
+ // One tick below the latest commit: above the archival boundary (so it reads
+ // as pending, not as archived) and at or below the snapshot bound (so the
+ // as-of filter does not exclude it either). That is an in-flight commit.
+ let latest_ts = table
+ .get_timeline()
+ .get_latest_commit_timestamp()
+ .unwrap()
+ .to_string();
+ let orphan_instant = format!(
+ "{:0width$}",
+ latest_ts.parse::<u64>().unwrap() - 1,
+ width = latest_ts.len()
+ );
+ let orphan_instant = orphan_instant.as_str();
+ plant_orphan_base_file(&table_dir, orphan_instant);
+
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let after = rows(&table.read(&opts).await.unwrap());
+ println!(
+ "{label:34} reader version={reader_version} rows before={before} after={after} {}",
+ if after == before {
+ "OK (orphan excluded)"
+ } else {
+ "*** UNCOMMITTED ROWS VISIBLE ***"
+ }
+ );
+}
+
+#[ignore = "documents a divergence from Java: layout-v1 tables do not filter uncommitted files"]
Review Comment:
Bisected, and the answer is the first of your two possibilities: **it got fixed, in the very commit that added the probe.**
`cd0d5be` ("fix(core): address the merge-on-read port and concurrency reviews") says so itself:
> A file was readable whenever a completion timestamp could be found for its commit, which asked the wrong question twice. Timeline layout v1 records no completion timestamps at all, so nothing was filtered there and a snapshot returned rows from commits that never completed — 4 rows became 7 on the v6 fixtures.
That commit replaced the layout-v2-only check with `completed_requests: HashSet<String>`, populated for **both** layouts, plus the archival boundary. The probe was written as the reproduction, the fix landed alongside it, and the `#[ignore]` reason was never updated — which is why it now prints "OK (orphan excluded)" for all six combinations.
So it is an ordinary test now, exactly as this file's own module doc prescribes: `test_a_base_file_from_an_uncommitted_instant_is_not_readable`, un-ignored, asserting `after == before` instead of printing it, on both timeline layouts and both reader versions.
I also corrected the module doc, which claimed the `#[ignore]`s exist for two reasons (unfixed divergence, or non-asserting probe). Only the second applies now — there are no divergence ignores left in this file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788139984
##########
crates/core/src/file_group/file_slice.rs:
##########
@@ -29,7 +29,20 @@ use std::path::PathBuf;
/// a [FileSlice] is a logical group of [BaseFile] and [LogFile]s.
#[derive(Clone, Debug)]
pub struct FileSlice {
- pub base_file: BaseFile,
+ /// The base file, when the slice has one.
+ ///
+ /// A slice written by inserts that went straight to log files has none —
+ /// Flink ingestion, a bucket index's first write to a bucket, and any
+ /// merge-on-read file group before its first compaction all produce these.
+ pub base_file: Option<BaseFile>,
Review Comment:
`base_file` becoming `Option<BaseFile>` (and `base_file_relative_path()` returning `Result<Option<String>>`) is a breaking change to hudi-core's public Rust API — the field is `pub` and external code compiles against the old shapes directly. All in-repo consumers (datafusion, python, cpp) are updated and CI is green, so this is only about communication. Could the PR description call it out alongside the timestamp-semantics breaking change, for crates.io consumers upgrading to 0.5?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#issuecomment-5288554004
## [Codecov](https://app.codecov.io/gh/apache/hudi-rs/pull/660?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `65.52078%` with `672 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 79.02%. Comparing base ([`9b09398`](https://app.codecov.io/gh/apache/hudi-rs/commit/9b09398d2848eeae722847c5b6276c719a294c40?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`2c684a6`](https://app.codecov.io/gh/apache/hudi-rs/commit/2c684a67fa80f4cafb351ae243fff30ec42d71a4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
| [Files with missing lines](https://app.codecov.io/gh/apache/hudi-rs/pull/660?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [crates/core/src/file\_group/reader\_v2/engine.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fengine.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2VuZ2luZS5ycw==) | 57.50% | [150 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/file\_group/log\_file/reader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Flog_file%2Freader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvbG9nX2ZpbGUvcmVhZGVyLnJz) | 49.47% | [97 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [.../src/file\_group/reader\_v2/buffer/position\_based.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fbuffer%2Fposition_based.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2J1ZmZlci9wb3NpdGlvbl9iYXNlZC5ycw==) | 57.92% | [85 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/file\_group/log\_file/content.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Flog_file%2Fcontent.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvbG9nX2ZpbGUvY29udGVudC5ycw==) | 46.15% | [63 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...core/src/file\_group/reader\_v2/log\_record\_reader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Flog_record_reader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2xvZ19yZWNvcmRfcmVhZGVyLnJz) | 68.27% | [59 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...tes/core/src/file\_group/reader\_v2/buffer/loader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fbuffer%2Floader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2J1ZmZlci9sb2FkZXIucnM=) | 48.95% | [49 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/file\_group/reader.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyLnJz) | 75.48% | [38 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...s/core/src/file\_group/reader\_v2/buffered\_record.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fbuffered_record.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2J1ZmZlcmVkX3JlY29yZC5ycw==) | 77.98% | [35 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [crates/core/src/file\_group/reader\_v2/buffer/mod.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fbuffer%2Fmod.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2J1ZmZlci9tb2QucnM=) | 36.36% | [14 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...e/src/file\_group/reader\_v2/buffer/record\_buffer.rs](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&filepath=crates%2Fcore%2Fsrc%2Ffile_group%2Freader_v2%2Fbuffer%2Frecord_buffer.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-Y3JhdGVzL2NvcmUvc3JjL2ZpbGVfZ3JvdXAvcmVhZGVyX3YyL2J1ZmZlci9yZWNvcmRfYnVmZmVyLnJz) | 72.09% | [12 Missing :warning: ](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| ... and [13 more](https://app.codecov.io/gh/apache/hudi-rs/pull/660?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## main #660 +/- ##
==========================================
- Coverage 82.38% 79.02% -3.37%
==========================================
Files 83 118 +35
Lines 6365 11135 +4770
==========================================
+ Hits 5244 8799 +3555
- Misses 1121 2336 +1215
```
</details>
[:umbrella: View full report in Codecov by Harness](https://app.codecov.io/gh/apache/hudi-rs/pull/660?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3786087285
##########
crates/core/src/table/mod.rs:
##########
@@ -155,6 +158,19 @@ impl Clone for Table {
}
}
+/// One tick below `instant_time`, as a same-width instant string.
+///
+/// Used to turn an exclusive start bound into an inclusive one. Instant times are
+/// fixed-width numeric strings, so this is a plain decrement; a value that does
+/// not parse (the bootstrap sentinels, for instance) is returned unchanged, which
+/// keeps the bound no narrower than it was.
+fn instant_time_minus_one(instant_time: &str) -> String {
Review Comment:
Confirmed and fixed — thank you, this was a real bug and the reproduction was exact. I reproduced it before changing anything:
```
20250713010500000 -> 20250713010499999 parse: ERR Timeline("Invalid epoch millis")
20250713010501000 -> 20250713010500999 parse: OK 2025-07-13 01:05:00.999 UTC
```
The `ss` field becomes 99. It sorts correctly, so every range comparison looked right, and it is not a time, so the selector rejects it.
**I tried your suggested fix first — carrying the exclusivity as a flag on the selector — and it broke `hudi_table_get_file_slices_incremental` (2 slices instead of 3).** `StartTimestamp` has four consumers building ranges, not one: `create_instant_range_for_log_file_scan`, `resolver::resolve_instant_range`, the row-mask path, and the metadata-table reader. Making only the log-scan range inclusive dropped a slice from the file-slice selection path, which never goes through it. So the flag approach is right in principle but needs all four moved together, and I did not want to land a partial version of it inside this PR.
What I did instead removes the invalid value rather than the encoding: the decrement now steps in the **time** domain, so the result is both a real instant and lexicographically just below the input.
```rust
const FORMATS: [&str; 2] = ["%Y%m%d%H%M%S%3f", "%Y%m%d%H%M%S"];
if let Ok(dt) = Instant::parse_datetime(instant_time, "UTC") {
for format in FORMATS {
if dt.format(format).to_string() == instant_time
&& let Some(stepped) = dt.checked_sub_signed(TimeDelta::milliseconds(1))
{ ... }
}
}
```
The round-trip is what distinguishes a date-formatted instant from an epoch-millis one (a metadata table's `00000000000000000`) — both parse, only the former renders back to itself — so the sentinels keep the integer decrement.
Two tests, both mutation-checked against the old integer decrement:
- `test_instant_time_minus_one_stays_a_valid_instant` — every borrow boundary: whole second, whole minute, whole hour, midnight (the date steps back), second-precision instants, and the epoch-millis sentinels.
- `test_resolve_incremental_window_start_bound_is_a_real_instant` — the bound sorts below the earliest admitted instant **and** parses. No fixture has a commit on a whole second, which is why this went unseen, so it asserts the property rather than reproducing the one input that broke.
I'd still like the flag version eventually — it removes the dual contract instead of satisfying it — but as its own change, with all four consumers moved at once.
##########
.github/workflows/ci.yml:
##########
@@ -81,6 +83,29 @@ jobs:
path: ./cov-reports
if-no-files-found: 'error'
+ # hudi-core without `spill-rocksdb` must keep building and passing. That build
+ # is the reason the feature exists — it needs no libclang and no C++ toolchain,
+ # so a consumer that cannot take a native build (or does not want one in a
+ # Python wheel) has a configuration to use. Nothing else in CI compiles it, so
+ # without this job it would break silently.
+ rust-tests-no-spill-backend:
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v6
+ - name: Cache dependencies
+ uses: swatinem/rust-cache@6323deb1c3ea99e6f4dae6b7d3ac1b73ac21ff77 # v2.9.2
Review Comment:
Fixed — you and I found this within about an hour of each other; it was pushed just before your comment landed.
Confirmed the same way you did: `gh api repos/Swatinem/rust-cache/commits/6323deb1c3ea99e6f4dae6b7d3ac1b73ac21ff77` returns `422 No commit found`. It shares the `6323deb1` prefix with the real v2.9.2 SHA and carries the same `# v2.9.2` comment, so it reads as correct at a glance — and the job died in "Set up job" after two seconds, before checkout, so nothing of ours ever ran.
Now pinned to `6323deb102c322ba6fcbdcafc7e3dddab59af2b6`, matching the other two jobs. I also checked every pinned action SHA across all workflow files resolves against the API — `astral-sh/setup-uv` and `docker/setup-buildx-action` both do. The `rust-tests-no-spill-backend` leg passes on the current head.
##########
crates/core/src/file_group/reader.rs:
##########
@@ -264,9 +430,13 @@ impl FileGroupReader {
/// selecting a version cannot turn a working read into a failing one. Each
/// reason is logged, because a fallback nobody can observe is
/// indistinguishable from a reader that is never used.
+ ///
+ /// Refusals are the exception, not the fallthrough: version 2 is the reader
Review Comment:
You're right, and it was my paragraph. Fixed both halves.
The doc now carves the case out explicitly rather than claiming something the code below contradicts:
> Every reason here **but one** means version 1 serves the read instead… The exception is a `CUSTOM` record merge mode with a merge to perform, which errors. That one *does* turn a working read into a failing one: version 2 is the default, so a merge-on-read table declaring `CUSTOM` that reads today stops reading. It is deliberate — falling back would merge with version 1's own derivation, which drops deletes, and wrong rows are worse than a refusal — but it is a behaviour change, and the error names the way back.
And the error now names the escape hatch:
> A table with a CUSTOM record merge mode needs its own merger, which no reader here implements. Set `hoodie.read.file.group.reader.version=1` to read it with the reader that served it before, which merges without that merger.
Worth recording alongside this: the only merge-on-read tables in the fixture corpus with `record.merge.mode=CUSTOM` are the metadata tables, which route to version 1 before the capability check runs. So this refusal is covered by unit tests only — no gold fixture exercises it.
##########
crates/core/src/file_group/reader_v2/mod.rs:
##########
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Context types for the merge-on-read file group reader.
+//!
+//! This module holds the inputs the MOR reader needs and the resolver that
+//! derives them from a table's configs. Nothing consumes it yet — the reader
Review Comment:
Confirmed, and I've made a start rather than a sweep.
You're right that the framing is stale: 25 files still say "nothing consumes it yet", written when these were stacked PRs, and this is the change that wires the reader in. I removed the file-scope `allow(dead_code)` from `input_split.rs` and `iterator_mode.rs` (their module docs needed unsplitting anyway — see the two nits on those files), and it immediately surfaced exactly what you describe: `InputSplit::start` and `InputSplit::length`, always `0` and `-1`, never read. Both removed.
I have **not** done the other 13 allows. Clearing them properly means deciding per item whether something is scaffolding the port still needs or an accident, and I would rather that be a deliberate pass — with your read on `HoodieFileGroupReaderBuilder` and `ProjectingBatchReader` in particular, since "never constructed" could mean either. Happy to do it in this PR if you'd prefer it cleared before merge; my instinct is a follow-up so the diff stays reviewable, but it's your call and I don't feel strongly.
##########
crates/core/src/file_group/reader_v2/gold_tests.rs:
##########
@@ -0,0 +1,335 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Reads each merge-on-read layout fixture and checks the result against the
+//! Spark snapshot shipped inside it.
+//!
+//! The fixtures carry their own `gold_data/` — a `SELECT *` of the table taken
+//! from Spark — so what is asserted here is agreement with Hudi's reference
+//! reader, not with this reader's own prior output.
+//!
+//! The file slice is discovered from the extracted fixture rather than written
+//! down per case: each of these tables is a single file group, so the base file
+//! and log files are whatever is on disk. A fixture that gains a file therefore
+//! cannot silently stop being covered.
+
+#![cfg(test)]
+
+use crate::config::HudiConfigs;
+use crate::config::read::HudiReadConfig;
+use crate::file_group::reader_v2::MAX_INSTANT_TIME;
+use crate::file_group::reader_v2::engine::HoodieFileGroupReader;
+use crate::file_group::reader_v2::input_split::InputSplit;
+use crate::file_group::reader_v2::reader_parameters::ReaderParameters;
+use crate::file_group::reader_v2::resolver::resolve_reader_context;
+use crate::storage::Storage;
+use arrow_array::RecordBatch;
+use std::path::Path;
+use std::sync::Arc;
+
+/// The base file and log files of a fixture's single file group, relative to
+/// the table root, plus the partition they live in.
+struct Slice {
+ base: Option<String>,
+ logs: Vec<String>,
+ partition: String,
+}
+
+/// Walk an extracted fixture and pick out its one file group.
+///
+/// Skips `.hoodie` (table metadata) and `gold_data` (the Spark snapshot this
+/// compares against, which is not table data).
+fn discover_slice(table_root: &Path) -> Slice {
+ fn walk(dir: &Path, root: &Path, out: &mut Vec<(String, String)>) {
+ let Ok(entries) = std::fs::read_dir(dir) else {
+ return;
+ };
+ for entry in entries.flatten() {
+ let path = entry.path();
+ let name = entry.file_name().to_string_lossy().to_string();
+ if path.is_dir() {
+ if name == ".hoodie" || name == "gold_data" {
+ continue;
+ }
+ walk(&path, root, out);
+ } else {
+ let rel = path
+ .strip_prefix(root)
+ .expect("walked path is under the table root")
+ .to_string_lossy()
+ .to_string();
+ let partition = Path::new(&rel)
+ .parent()
+ .map(|p| p.to_string_lossy().to_string())
+ .unwrap_or_default();
+ out.push((rel, partition));
+ }
+ }
+ }
+
+ let mut files = Vec::new();
+ walk(table_root, table_root, &mut files);
+
+ let mut base = None;
+ let mut logs = Vec::new();
+ let mut partition = String::new();
+ for (rel, part) in files {
+ let name = Path::new(&rel)
+ .file_name()
+ .map(|n| n.to_string_lossy().to_string())
+ .unwrap_or_default();
+ // Hadoop writes a `..<file>.crc` checksum sidecar next to each data
+ // file. It is not table data and must not reach the readers.
+ if name.ends_with(".crc") {
+ continue;
+ }
+ if name.ends_with(".parquet") && !name.starts_with('.') {
+ base = Some(rel);
+ partition = part;
+ } else if name.contains(".log.") {
+ logs.push(rel);
+ partition = part;
+ }
+ }
+ // Log files are appended in order; the scan relies on that ordering.
+ logs.sort();
+
+ Slice {
+ base,
+ logs,
+ partition,
+ }
+}
+
+/// Read a fixture's file group and return the merged rows.
+async fn read_fixture(table_path: &str) -> crate::Result<RecordBatch> {
+ let slice = discover_slice(Path::new(table_path));
+
+ // Load the table's own properties rather than inventing them: the merge
+ // semantics live in `hoodie.properties`, and inventing configs would test
+ // the reader against a table that does not exist.
+ let mut resolver = crate::table::builder::OptionResolver::new_with_options(
+ table_path,
+ [(
+ HudiReadConfig::EndTimestamp.as_ref(),
+ MAX_INSTANT_TIME.to_string(),
+ )],
+ );
+ resolver.resolve_options().await?;
+ let configs = HudiConfigs::new(resolver.hudi_options.clone());
+ let storage = Storage::new(
+ Arc::new(resolver.storage_options),
+ Arc::new(configs.clone()),
+ )?;
+
+ let has_logs = !slice.logs.is_empty();
+ let mut context = resolve_reader_context(&configs, has_logs)?;
+ // The fixtures are read whole, so nothing bounds the log scan.
+ context.instant_range = None;
+ context.rebuild_record_context(slice.partition.clone());
+
+ let input_split = InputSplit::new(slice.base, None, slice.logs, slice.partition);
+
+ // The table's own schema, the way a real read resolves it. Passing `None`
+ // here — as this harness used to — makes the reader fall back to the base
+ // file's schema, which is stale on any table whose columns were widened
+ // after that file was written. The fixtures are then read into the older
+ // shape and the comparison below, which only counts rows, cannot see it.
+ let table = crate::table::Table::new(table_path).await?;
+ let data_schema = Arc::new(table.data_schema_for_read().await?);
+
+ let mut reader = HoodieFileGroupReader::new(
+ Arc::new(context),
+ storage,
+ input_split,
+ ReaderParameters::default(),
+ Some(data_schema),
+ None,
+ )?;
+ reader.read().await
+}
+
+/// Fixtures the reader reproduces today.
+const GOLD_FIXTURES: &[&str] = &[
+ "table_log_only",
+ "table_log_compaction",
+ "table_parquet_log_block",
+ "table_partial_update",
+ "table_evo_add_col",
+ "table_evo_promotion",
+];
+
+/// Fixtures the reader does not read yet, with what stops each one.
+///
+/// Two causes remain, neither in the merge:
+///
+/// **Avro maps are modelled as a malformed Arrow type.** `avro_to_arrow` turns
+/// an Avro map into `Dictionary(Utf8, V)`, but an Arrow dictionary key must be
+/// an integer, so the array builder rejects it. Avro maps belong in Arrow's
+/// `Map` type. Changing that alters the schema of every table with a map
+/// column, including on the existing read path.
+///
+/// **The delete record's ordering value has two shapes in the wild.** Hudi
+/// moved it into per-type wrapper records — `IntWrapper`, `DecimalWrapper` and
+/// so on — because Avro forbids a union holding two branches of the same
+/// underlying type. This crate carries only the older primitive union, so a log
+/// file written the newer way resolves each branch index to the wrong type.
+///
+/// The two shapes cannot be told apart by decoding: a wrapper is a record whose
+/// single field is the primitive, so `IntWrapper` and a bare `long` encode to
+/// the same bytes. Only the branch index differs. Reading both therefore needs
+/// to key off the writer version rather than the payload, which is a design
+/// question rather than a decoding one — and swapping to the newer schema
+/// outright breaks the fixtures written the older way.
+///
+/// **A corrupt tail block is not recognised.** `LogFileReader` has no
+/// corrupt-block detection — `create_corrupted_block_if_needed` returns `None`
+/// — so it parses the trailing garbage as a block.
+const KNOWN_GAPS: &[(&str, &str)] = &[
Review Comment:
Confirmed — running it with `--ignored` fails exactly as you say:
```
these now read correctly and should move to GOLD_FIXTURES:
["table_column_projection", "table_all_data_types", "table_null_containers",
"table_corrupt_tail_block", "table_delete_ord_int", "table_delete_ord_long",
"table_delete_ord_double", "table_delete_ord_decimal",
"table_delete_ord_string", "table_delete_ord_timestamp"]
```
All ten, including the four `table_delete_ord_*` you named, and the "two causes remain" doc above is stale with them.
On whether it earns its keep: I think you're right that it doesn't. `tests/gold_parity_tests.rs` covers the same fixtures against the same Spark snapshots with a live ratchet — a new disagreement fails the build and a fixed one has to be removed — whereas this harness is `#[ignore]`d, so its data can rot without anything noticing, which is precisely what happened. A second copy that cannot fail is worse than no copy.
I've left it alone for now rather than deleting it unilaterally, since removing a test harness on someone else's read is not my call to make silently. Say the word and I'll fold what it uniquely covers into `gold_parity_tests.rs` and delete it.
##########
crates/core/tests/reader_concurrency_tests.rs:
##########
@@ -0,0 +1,341 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+//! Reader-vs-writer concurrency repros.
+//!
+//! Most of these are `#[ignore]`d, for one of two reasons: they document a
+//! divergence from Hudi's Java reader that is not fixed yet, or they are probes
+//! that print what the reader does without asserting anything. Neither belongs
+//! in the suite — the first would fail it, and the second would pass whatever
+//! happened. Run them with
+//! `cargo test -p hudi-core --test reader_concurrency_tests -- --ignored --nocapture`.
+//!
+//! A divergence that gets fixed should leave this file, or become an ordinary
+//! test that pins the fixed behavior. An `#[ignore]` reason describing something
+//! that no longer happens is worse than no note at all.
+
+use hudi_core::table::{ReadOptions, Table};
+use std::path::{Path, PathBuf};
+
+fn rows(batches: &[arrow_array::RecordBatch]) -> usize {
+ batches.iter().map(|b| b.num_rows()).sum()
+}
+
+/// Copy an existing base file under a NEW file id at `instant`, leaving no
+/// commit for that instant — a writer that wrote its data file and then died.
+fn plant_orphan_base_file(table_dir: &Path, instant: &str) -> PathBuf {
+ let existing = std::fs::read_dir(table_dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.extension().is_some_and(|x| x == "parquet"))
+ .expect("fixture has a base file to copy");
+ let orphan = table_dir.join(format!(
+ "b179bdb3-731c-4894-b855-abfcd6921008-0_0-1-1_{instant}.parquet"
+ ));
+ std::fs::copy(&existing, &orphan).unwrap();
+ orphan
+}
+
+fn fixture_zip(relative: &str) -> PathBuf {
+ PathBuf::from(env!("CARGO_MANIFEST_DIR").replace("/core", "/test")).join(relative)
+}
+
+async fn probe(zip: &str, label: &str) {
+ probe_with_reader_version(zip, label, "1").await;
+ probe_with_reader_version(zip, label, "2").await;
+}
+
+async fn probe_with_reader_version(zip: &str, label: &str, reader_version: &str) {
+ use hudi_core::config::read::HudiReadConfig;
+ let dir = hudi_test::extract_test_table_fresh(&fixture_zip(zip));
+ let table_dir = std::fs::read_dir(&dir)
+ .unwrap()
+ .filter_map(|e| e.ok())
+ .map(|e| e.path())
+ .find(|p| p.is_dir() && p.join(".hoodie").exists())
+ .expect("extracted fixture holds one table dir");
+
+ let opts = ReadOptions::new().with_hudi_option(
+ HudiReadConfig::FileGroupReaderVersion.as_ref(),
+ reader_version,
+ );
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let before = rows(&table.read(&opts).await.unwrap());
+
+ // One tick below the latest commit: above the archival boundary (so it reads
+ // as pending, not as archived) and at or below the snapshot bound (so the
+ // as-of filter does not exclude it either). That is an in-flight commit.
+ let latest_ts = table
+ .get_timeline()
+ .get_latest_commit_timestamp()
+ .unwrap()
+ .to_string();
+ let orphan_instant = format!(
+ "{:0width$}",
+ latest_ts.parse::<u64>().unwrap() - 1,
+ width = latest_ts.len()
+ );
+ let orphan_instant = orphan_instant.as_str();
+ plant_orphan_base_file(&table_dir, orphan_instant);
+
+ let table = Table::new(table_dir.to_str().unwrap()).await.unwrap();
+ let after = rows(&table.read(&opts).await.unwrap());
+ println!(
+ "{label:34} reader version={reader_version} rows before={before} after={after} {}",
+ if after == before {
+ "OK (orphan excluded)"
+ } else {
+ "*** UNCOMMITTED ROWS VISIBLE ***"
+ }
+ );
+}
+
+#[ignore = "documents a divergence from Java: layout-v1 tables do not filter uncommitted files"]
Review Comment:
Confirmed — `--ignored` gives 3 passed, 0 failed, and the orphan-exclusion probe prints "OK (orphan excluded)" for all six combinations. The reason on the `#[ignore]` no longer reproduces, and the module doc above says that is exactly when it should be updated.
I don't yet know which of your two possibilities it is — fixed somewhere in this change, or needing a different fixture to show up. The honest answer is that I would have to bisect the stack to tell, and I did not want to guess in the ignore reason. Left as-is pending that; if you'd rather it not merge in this state I'll dig into which commit changed the behaviour and either drop the test or re-point it at a fixture that still shows the divergence.
##########
crates/core/Cargo.toml:
##########
@@ -101,12 +107,32 @@ percent-encoding = { workspace = true }
[dev-dependencies]
hudi-test = { path = "../test" }
serial_test = { workspace = true }
+# Also a regular dependency, but only an optional one behind `spill-rocksdb`.
+# Repeated here because tests outside the spill tier use it too, and must keep
+# building in the `--no-default-features` configuration.
tempfile = { workspace = true }
[lints.clippy]
result_large_err = "allow"
[features]
+default = ["spill-rocksdb"]
Review Comment:
Your reading is right, and it makes the feature's stated rationale not currently true for the two consumers it names. `spill-rocksdb` is default-on, `crates/hudi` takes `hudi-core` with default features, and both the Python wheel and the cxx bridge go through the umbrella crate — so both still build rocksdb, libclang and all.
The intent was the second of your two options — external consumers opt out — but the doc reads as though the wheel and the bridge already benefit, which they do not. Two ways to make it honest: forward the feature from `crates/hudi` so those two can actually opt out, or correct the rationale to say only external consumers benefit today.
I'd rather not pick between those inside this PR — forwarding the feature changes the umbrella crate's public feature surface, which deserves its own diff and its own note. Which would you prefer?
##########
crates/core/src/file_group/reader_v2/buffer/spillable_map.rs:
##########
@@ -0,0 +1,2627 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Size-tracked, RocksDB-spillable merge map (A1, ENG-42993).
Review Comment:
Confirmed and agreed: 105 `ENG-XXXXX` references across 16 files, plus the phase labels. They are internal tracker ids and mean nothing to a reader on the ASF side — the project's own guidance says not to reference internal plans.
I have not done this yet. It is mechanical in shape but not by `sed`: each one is standing in for a behaviour, and replacing it well means reading the surrounding code and writing down what the id was pointing at — otherwise the comment loses information rather than gaining it. Doing that badly across 105 sites would be worse than leaving them.
So: happy to do it, and I think it should be done before this merges into an ASF repo, but as a focused pass. Would you rather it land in this PR or immediately after?
##########
crates/core/src/file_group/reader_v2/record_merger.rs:
##########
@@ -0,0 +1,799 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.BufferedRecordMerger` and
+//! `org.apache.hudi.common.table.read.BufferedRecordMergerFactory`.
+//!
+//! The `BufferedRecordMerger` trait defines how records are merged during
+//! log scanning (delta merge: log-vs-log) and at read time (final merge:
+//! base-vs-log).
+
+use super::buffered_record::{BufferedRecord, DeleteRecord, OrderingValue};
+use crate::Result;
+
+/// Mirrors Java `OrderingValues.isCommitTimeOrderingValue` (`orderingValue == null
+/// || OrderingValues.isDefault(orderingValue)`, where `isDefault` ==
+/// `Integer(0).equals(orderingValue)`). A DELETE carrying the default ordering
+/// value is a "commit-time ordering delete" and wins unconditionally, regardless
+/// of the existing record's ordering value (mirrors
+/// `BufferedRecord.isCommitTimeOrderingDelete`).
+///
+/// The default is ONLY the null-coerced [`OrderingValue::Default`] sentinel (Java
+/// `Integer(0)`) or an absent (`None`) ordering. A GENUINE `Long(0)` field value
+/// is NOT default (GAP-2): `Integer(0).equals(Long(0))` is `false` in Java, so a
+/// delete carrying a real ordering value of `0` is ordering-compared, not treated
+/// as natural-order.
+fn is_default_ordering(ordering_value: &Option<OrderingValue>) -> bool {
+ matches!(ordering_value, None | Some(OrderingValue::Default))
+}
+
+/// Mirrors Java `BufferedRecord.isCommitTimeOrderingDelete`: a DELETE carrying
+/// the default ordering value. Such deletes win unconditionally (natural order).
+fn is_commit_time_ordering_delete(record: &BufferedRecord) -> bool {
+ record.is_delete() && is_default_ordering(&record.ordering_value)
+}
+
+/// Mirrors Java `OrderingValues.isSameClass`: ordering values are only compared
+/// when they are the same concrete type. Java throws on a cross-type
+/// `compareTo`; we instead skip the comparison (caller lets the newer/delete
+/// record win), avoiding the arbitrary cross-variant `Ord` on [`OrderingValue`].
+///
+/// Thin free-function wrapper over [`OrderingValue::is_same_class`] so the four
+/// buffer-side call sites below read as `is_same_class(a, b)`; the single source
+/// of truth is the method.
+fn is_same_class(a: &OrderingValue, b: &OrderingValue) -> bool {
+ a.is_same_class(b)
+}
+
+/// Trait for merging buffered records during the file group read pipeline.
+///
+/// Mirrors Java's `BufferedRecordMerger<T>` interface.
+///
+/// Consumed by `KeyBasedFileGroupRecordBuffer`:
+/// - `delta_merge` in `process_next_data_record` (log-vs-log within buffer)
+/// - `delta_merge_delete` in `process_next_deleted_record` (delete-vs-existing)
+/// - `final_merge` in `has_next_base_record` (base-vs-log at read time)
+pub trait BufferedRecordMerger: Send + Sync + std::fmt::Debug {
+ /// Merge a new log record with an existing buffered record.
+ ///
+ /// Returns `Some(merged)` if the record should be kept, `None` if dropped.
+ ///
+ /// Called during log scanning when a new record for the same key arrives.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>>;
+
+ /// Merge a delete record with an existing buffered record.
+ ///
+ /// Returns `Some(delete)` if the delete wins, `None` if the existing record survives.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>>;
+
+ /// Merge a base file record with a log record (final merge at read time).
+ ///
+ /// The `older_record` is from the base file, `newer_record` is from the log buffer.
+ fn final_merge(
+ &self,
+ older_record: &BufferedRecord,
+ newer_record: &BufferedRecord,
+ ) -> Result<BufferedRecord>;
+}
+
+/// Event-time based record merger.
+///
+/// Resolves conflicts by comparing ordering values: the record with the
+/// higher ordering value wins. This is the merge strategy for the
+/// `EVENT_TIME_ORDERING` mode only; `COMMIT_TIME_ORDERING` uses
+/// [`CommitTimeRecordMerger`] (last writer wins, no ordering comparison).
+///
+/// Created by `BufferedRecordMergerFactory` based on the merge mode. The read
+/// path now accepts `EVENT_TIME_ORDERING` (`buffer/loader.rs::get_record_buffer`),
+/// so this merger is live in production — it drives the scalar base-vs-log
+/// `final_merge`, and the vectorized `pick_winner` kernel mirrors its semantics.
+#[derive(Debug)]
+pub struct EventTimeRecordMerger;
+
+impl BufferedRecordMerger for EventTimeRecordMerger {
+ /// Mirrors Java `EventTimeRecordMerger.deltaMerge` via `shouldKeepNewerRecord`.
+ fn delta_merge(
+ &self,
+ new_record: &BufferedRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<BufferedRecord>> {
+ match existing_record {
+ None => Ok(Some(new_record.clone())),
+ Some(existing) if should_keep_newer_record(existing, new_record) => {
+ Ok(Some(new_record.clone()))
+ }
+ Some(existing) => Ok(Some(existing.clone())),
+ }
+ }
+
+ /// Mirrors Java `BufferedRecordMergerFactory.deltaMergeDeleteRecord`.
+ ///
+ /// The DELETE is obsolete (existing survives) ONLY when the existing record
+ /// has a strictly greater ordering value of the SAME type AND the delete
+ /// carries a non-default ordering value (`0` == natural order = always
+ /// delete). A default-ordering delete, or one against an existing record
+ /// that is itself a commit-time-ordering delete, follows Java's branches.
+ fn delta_merge_delete(
+ &self,
+ delete_record: &DeleteRecord,
+ existing_record: Option<&BufferedRecord>,
+ ) -> Result<Option<DeleteRecord>> {
+ match existing_record {
+ None => Ok(Some(delete_record.clone())),
+ Some(existing) if is_commit_time_ordering_delete(existing) => Ok(None),
+ Some(existing) => {
+ let choose_existing = !is_default_ordering(&delete_record.ordering_value)
+ && match (&delete_record.ordering_value, &existing.ordering_value) {
+ (Some(del_val), Some(existing_val)) => {
+ is_same_class(del_val, existing_val) && existing_val > del_val
+ }
+ _ => false,
+ };
+ if choose_existing {
+ Ok(None) // existing record survives; DELETE is obsolete
+ } else {
+ Ok(Some(delete_record.clone()))
+ }
+ }
+ }
+ }
+
+ /// Mirrors Java `EventTimeRecordMerger.finalMerge`.
+ ///
+ /// The newer (log) record wins unless it is NOT a commit-time-ordering delete
+ /// and the older record (also not such a delete) has a strictly greater,
+ /// same-typed ordering value.
+ fn final_merge(
Review Comment:
No good reason — you're right that it is one rule in two places, and thank you for working through the CTO-delete and cross-type cases to confirm they agree today.
I have not changed it yet, only because `final_merge` is on the merge hot path and I want to confirm the delegation is behaviour-identical under the gold sweep rather than by reading, given the two spellings agree by inspection but the whole point of the finding is that they could stop agreeing. That check is cheap and I'll do it — I just did not want to slip a hot-path change in unverified alongside everything else in this round.
If you'd rather see it in this PR, say so and I'll make the change and run the corpus against it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788765172
##########
crates/core/schemas/HoodieDeleteRecord.avsc:
##########
Review Comment:
Answering my own question after tracing it: the list file is the wire schema (wrapper-record union, what Java serializes) and this file is its post-unwrap companion, where each union position holds the bare primitive inside the wrapper at the same position, which `avro_schema_for_delete_record` indexes to build the narrowed reader schema after `unwrap_ordering_value` strips the wrapper (pinned by `test_schema_for_delete_record`). Since only their own tests still call those helpers, a follow-up could either cross-reference the two files in a comment or drop this one with the helpers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "hudi-agent (via GitHub)" <gi...@apache.org>.
hudi-agent commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3780692640
##########
crates/core/src/file_group/reader_v2/input_split.rs:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.InputSplit`.
Review Comment:
🤖 nit: the `#![allow(dead_code)]` attribute is sandwiched between two `//!` module-doc blocks, splitting what reads as a single description across an attribute. Could you move `#![allow(dead_code)]` before the first `//!` line (or after the last), so the module doc is contiguous?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
crates/core/src/file_group/reader_v2/iterator_mode.rs:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.IteratorMode`.
Review Comment:
🤖 nit: same split `//!` block issue as `input_split.rs` - the `#![allow(dead_code)]` attribute breaks the module doc in two. Could you move it so the description is contiguous?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
crates/core/src/file_group/reader_v2/iterator_mode.rs:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.IteratorMode`.
+//!
+//! Controls what form the output records take when iterating.
+
+/// The mode in which the file group reader yields records.
+///
+/// In Java Hudi, this controls whether the iterator produces engine-native
+/// records, HoodieRecord wrappers, or just record keys.
+///
+/// In hudi-rs, we always work with Arrow RecordBatch, so `EngineRecord`
+/// is the primary mode. The other modes are kept for API symmetry.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum IteratorMode {
+ /// Yield engine-native records (Arrow RecordBatch in Rust).
+ #[default]
+ EngineRecord,
+
+ /// Yield HoodieRecord wrappers (not yet implemented in Rust).
+ HoodieRecord,
+
+ /// Yield only record keys (not yet implemented in Rust).
+ RecordKey,
+}
+
+impl IteratorMode {
Review Comment:
🤖 nit: `from_str_opt` is a non-idiomatic name - the `_opt` suffix for returning `Option` is not a Rust convention. Have you considered implementing `std::str::FromStr` (enabling `.parse::<IteratorMode>()`) or naming it `try_from_str` to align with how the ecosystem spells fallible conversions?
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
##########
crates/core/src/file_group/reader_v2/buffered_record.rs:
##########
@@ -0,0 +1,1094 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.BufferedRecord` and
+//! `org.apache.hudi.common.model.DeleteRecord`.
+//!
+//! In Java Hudi, `BufferedRecord<T>` wraps a single engine-native record
+//! with its key, ordering value, and operation type. In hudi-rs, the
+//! "engine record" is an Arrow `RecordBatch`.
+//!
+//! During log scanning, records are accumulated into the record buffer.
+//! At read time, base file records are merged with log records to produce
+//! the final output.
+//!
+//! ## In-memory representation (A2)
+//!
+//! Per the A2 alignment work (05-a2a1-design.md), the in-memory payload is a
+//! zero-copy [`RecordPayload::BatchRef`] into the shared source batch that the
+//! row was decoded from — NOT a per-row Arrow IPC blob. IPC serialization is a
+//! spill-only concern (A1) and lives in
+//! [`row_serde`](super::row_serde) (`to_binary_row` /
+//! `from_binary`); it is no longer paid on the in-memory merge hot path. The
+//! data-driven motivation: the per-row IPC blob cost ~27x the build memory and
+//! ~12x the build CPU of the batch-ref representation (spike Part 1).
+
+use std::sync::Arc;
+
+use arrow_array::RecordBatch;
+use arrow_buffer::i256;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::file_group::reader_v2::row_serde;
+
+// ── Spill value encoding (A1) ──────────────────────────────────────────────
+// On spill, a `BufferedRecord` is serialized to a self-describing byte blob
+// stored as the RocksDB value. The record key is the RocksDB key, so it is NOT
+// repeated in the value. The layout is:
+//
+// [payload_tag: u8]
+// 0x00 = Delete (no further payload bytes)
+// 0x01 = Data (followed by an Arrow IPC single-row stream)
+// [ordering_tag: u8]
+// 0x00 = None
+// 0x01 = Long followed by 8 bytes (i64 little-endian)
+// 0x02 = String followed by [len: u32 LE][utf8 bytes]
+// [if payload_tag == Data] Arrow IPC stream bytes (rest of the blob)
+//
+// All multi-byte integers are little-endian. The format is internal to the
+// reader's own spill round-trip (never persisted across runs), so no on-disk
+// version negotiation is needed — the spill dir is created and destroyed within
+// a single read (see `SpillableRecordMap`).
+
+/// Spill payload tag: the record is a delete tombstone (no data follows).
+const SPILL_PAYLOAD_TAG_DELETE: u8 = 0x00;
+/// Spill payload tag: the record carries data (an Arrow IPC stream follows the
+/// ordering-value header).
+const SPILL_PAYLOAD_TAG_DATA: u8 = 0x01;
+/// Spill ordering-value tag: no ordering value.
+const SPILL_ORDERING_TAG_NONE: u8 = 0x00;
+/// Spill ordering-value tag: `OrderingValue::Long` (8-byte i64 follows).
+const SPILL_ORDERING_TAG_LONG: u8 = 0x01;
+/// Spill ordering-value tag: `OrderingValue::String` (u32 length + utf8 follow).
+const SPILL_ORDERING_TAG_STRING: u8 = 0x02;
+/// Spill ordering-value tag: `OrderingValue::Composite` (u32 element count, then
+/// each element encoded with this same tag scheme).
+const SPILL_ORDERING_TAG_COMPOSITE: u8 = 0x03;
+/// Spill ordering-value tag: `OrderingValue::Double` (8-byte IEEE-754 f64 follows).
+const SPILL_ORDERING_TAG_DOUBLE: u8 = 0x04;
+/// Spill ordering-value tag: `OrderingValue::Decimal` (16-byte little-endian i128
+/// unscaled value, then a 1-byte i8 scale follow).
+const SPILL_ORDERING_TAG_DECIMAL: u8 = 0x05;
+/// Spill ordering-value tag: `OrderingValue::Default` (no payload). The null-coerced
+/// default ordering value (Java `HoodieRecord.DEFAULT_ORDERING_VALUE` == `Integer(0)`).
+const SPILL_ORDERING_TAG_DEFAULT: u8 = 0x06;
+
+/// In-memory payload of a buffered record (A2).
+///
+/// The default representation is a zero-copy reference into a shared source
+/// batch ([`BatchRef`](RecordPayload::BatchRef)). Serialization (Arrow IPC)
+/// happens only on spill (A1), never on the in-memory hot path — so this enum
+/// deliberately carries no binary variant.
+///
+/// ## Pinning caveat
+///
+/// A [`BatchRef`](RecordPayload::BatchRef) keeps its WHOLE source
+/// `Arc<RecordBatch>` alive (a 1-row [`RecordBatch::slice`] still references the
+/// parent's full buffers). When survivors are sparse across many source batches,
+/// the merge map can pin far more Arrow memory than the live rows occupy. The
+/// `compact_pinned_batches` safety valve (see `key_based.rs`) re-batches the
+/// survivors of sparsely-populated source batches and repoints their entries to
+/// release the dead-row memory.
+#[derive(Debug, Clone)]
+pub enum RecordPayload {
+ /// Zero-copy reference into a shared source batch (the A2 default). `row_idx`
+ /// addresses a single row within `batch`. Reading the payload slices that row
+ /// out via [`RecordBatch::slice`] (cheap, shares buffers).
+ BatchRef {
+ /// The shared source batch this row was decoded from. Interned once per
+ /// decoded block batch (one `Arc` per batch) so that `Arc::as_ptr`
+ /// identity is a valid grouping key for compaction.
+ batch: Arc<RecordBatch>,
+ /// Row index of this record within `batch`.
+ row_idx: usize,
+ },
+ /// Owned single-row batch. Produced by `compact_pinned_batches` (a freshly
+ /// re-batched survivor set) or for records that have no shared source batch.
+ Owned(RecordBatch),
+ /// Tombstone — the record represents a deletion and carries no payload.
+ Delete,
+}
+
+impl RecordPayload {
+ /// Returns the payload as a single-row `RecordBatch`, or `None` for a delete.
+ ///
+ /// `BatchRef` slices the addressed row out of its source batch (zero-copy
+ /// buffer slice); `Owned` clones the held batch. Mirrors the read side of
+ /// Java's `BufferedRecord.getRecord()`.
+ pub fn get_record(&self) -> Option<RecordBatch> {
+ match self {
+ RecordPayload::BatchRef { batch, row_idx } => Some(batch.slice(*row_idx, 1)),
+ RecordPayload::Owned(batch) => Some(batch.clone()),
+ RecordPayload::Delete => None,
+ }
+ }
+
+ /// Consume the payload and return its single-row `RecordBatch`, or `None`
+ /// for a delete. The move-not-clone counterpart to [`get_record`](Self::get_record).
+ pub fn into_record(self) -> Option<RecordBatch> {
+ match self {
+ RecordPayload::BatchRef { batch, row_idx } => Some(batch.slice(row_idx, 1)),
+ RecordPayload::Owned(batch) => Some(batch),
+ RecordPayload::Delete => None,
+ }
+ }
+
+ /// Returns true if this payload is a delete tombstone.
+ pub fn is_delete(&self) -> bool {
+ matches!(self, RecordPayload::Delete)
+ }
+}
+
+/// The universal record envelope flowing through the merge pipeline.
+///
+/// Mirrors Java's `BufferedRecord<T>`. In Rust/Arrow, a single "record" is a
+/// single-row view ([`RecordPayload::BatchRef`]) into a shared decoded batch;
+/// the merge map stores one of these per live key.
+///
+/// During the log scanning phase, `BufferedRecord`s are stored in the record
+/// buffer's map keyed by record key.
+#[derive(Debug, Clone)]
+pub struct BufferedRecord {
+ /// The record key extracted from the record.
+ pub record_key: String,
+
+ /// The in-memory record payload (zero-copy batch ref, owned batch, or delete).
+ pub payload: RecordPayload,
+
+ /// The ordering value used for merge conflict resolution.
+ /// Higher ordering value wins during delta merge.
+ pub ordering_value: Option<OrderingValue>,
+}
+
+impl BufferedRecord {
+ /// Create a new data record from a zero-copy reference into a shared source
+ /// batch (the A2 hot-path constructor).
+ ///
+ /// `batch` must be the interned `Arc` for the decoded block batch (one `Arc`
+ /// per batch) so that `Arc::as_ptr` grouping during compaction is valid.
+ pub fn new_batch_ref(
+ record_key: String,
+ batch: Arc<RecordBatch>,
+ row_idx: usize,
+ ordering_value: Option<OrderingValue>,
+ ) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::BatchRef { batch, row_idx },
+ ordering_value,
+ }
+ }
+
+ /// Create a new data record from an owned single-row batch.
+ ///
+ /// Used by tests and by call sites that hold a single-row batch with no
+ /// shared source (e.g. base-record construction in tests). The hot path
+ /// uses [`new_batch_ref`](Self::new_batch_ref) instead.
+ pub fn new_data(
+ record_key: String,
+ data: RecordBatch,
+ ordering_value: Option<OrderingValue>,
+ ) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::Owned(data),
+ ordering_value,
+ }
+ }
+
+ /// Create a new delete record (tombstone).
+ pub fn new_delete(record_key: String, ordering_value: Option<OrderingValue>) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::Delete,
+ ordering_value,
+ }
+ }
+
+ /// Returns true if this record represents a deletion.
+ pub fn is_delete(&self) -> bool {
+ self.payload.is_delete()
+ }
+
+ /// Returns true if this record has no data payload (a delete tombstone).
+ pub fn is_empty(&self) -> bool {
+ self.payload.is_delete()
+ }
+
+ /// Return the record data as a single-row `RecordBatch`.
+ ///
+ /// Mirrors Java's `BufferedRecord.getRecord()`. Delegates to
+ /// [`RecordPayload::get_record`]: `BatchRef` slices the row out (zero-copy),
+ /// `Owned` clones, `Delete` returns `None`.
+ pub fn get_record(&self) -> Option<RecordBatch> {
+ self.payload.get_record()
+ }
+
+ /// Consume the record and return its data batch. The move-not-clone
+ /// counterpart to [`get_record`](Self::get_record), used on the merge-output
+ /// hot path where the record is no longer needed afterward.
+ pub fn into_record(self) -> Option<RecordBatch> {
+ self.payload.into_record()
+ }
+
+ /// Serialize this record to a self-describing spill blob (**spill-only, A1**).
+ ///
+ /// Used by [`SpillableRecordMap`](super::buffer::spillable_map::SpillableRecordMap)
+ /// to store an entry in the on-disk (RocksDB) tier when the in-memory budget
+ /// is exhausted. The record key is NOT included (it is the RocksDB key); the
+ /// blob carries the payload (delete tombstone or a single-row Arrow IPC
+ /// stream) and the ordering value. See the module-level spill-encoding notes.
+ ///
+ /// Start with single-row IPC for correctness; a batched-spill optimization
+ /// (group survivors by source batch, spill multi-row blobs) is tracked as B5
+ /// in the backlog (05-a2a1-design.md) and deliberately NOT built here.
+ pub fn to_spill_bytes(&self) -> Vec<u8> {
+ let mut buf = Vec::new();
+ match self.get_record() {
+ None => buf.push(SPILL_PAYLOAD_TAG_DELETE),
+ Some(batch) => {
+ buf.push(SPILL_PAYLOAD_TAG_DATA);
+ encode_ordering_value(&mut buf, self.ordering_value.as_ref());
+ // Arrow IPC single-row stream (the spill serialization primitive,
+ // row_serde::to_binary_row). For deletes we skip this entirely.
+ let ipc = row_serde::to_binary_row(&batch.schema(), &batch);
+ buf.extend_from_slice(&ipc);
+ return buf;
+ }
+ }
+ // Delete path: still record the ordering value (it can matter when a
+ // spilled delete tombstone is later compared during the base merge).
+ encode_ordering_value(&mut buf, self.ordering_value.as_ref());
+ buf
+ }
+
+ /// Reconstruct a record from a spill blob produced by
+ /// [`to_spill_bytes`](Self::to_spill_bytes) (**spill-only, A1**).
+ ///
+ /// `record_key` is supplied by the caller (it is the RocksDB key, not stored
+ /// in the blob). Returns a typed [`CoreError`] on a malformed blob rather
+ /// than panicking — a corrupt spill entry is a recoverable read-path error,
+ /// not an internal invariant violation.
+ pub fn from_spill_bytes(record_key: String, bytes: &[u8]) -> Result<Self> {
+ let mut cursor = 0usize;
+ let payload_tag = read_u8(bytes, &mut cursor)?;
+ match payload_tag {
+ SPILL_PAYLOAD_TAG_DELETE => {
+ let ordering_value = decode_ordering_value(bytes, &mut cursor)?;
+ Ok(BufferedRecord::new_delete(record_key, ordering_value))
+ }
+ SPILL_PAYLOAD_TAG_DATA => {
+ let ordering_value = decode_ordering_value(bytes, &mut cursor)?;
+ let batch = row_serde::from_binary(&bytes[cursor..])?;
+ // Reloaded from disk: there is no shared source batch to reference,
+ // so the payload is necessarily `Owned`.
+ Ok(BufferedRecord::new_data(record_key, batch, ordering_value))
+ }
+ other => Err(CoreError::ReadFileSliceError(format!(
+ "spill decode: unknown payload tag {other:#04x}"
+ ))),
+ }
+ }
+}
+
+/// Append an ordering value to a spill blob using the tag scheme documented at
+/// the module level.
+fn encode_ordering_value(buf: &mut Vec<u8>, ordering_value: Option<&OrderingValue>) {
+ match ordering_value {
+ None => buf.push(SPILL_ORDERING_TAG_NONE),
+ Some(OrderingValue::Long(v)) => {
+ buf.push(SPILL_ORDERING_TAG_LONG);
+ buf.extend_from_slice(&v.to_le_bytes());
+ }
+ Some(OrderingValue::Double(v)) => {
+ buf.push(SPILL_ORDERING_TAG_DOUBLE);
+ buf.extend_from_slice(&v.to_le_bytes());
+ }
+ Some(OrderingValue::Decimal { unscaled, scale }) => {
+ buf.push(SPILL_ORDERING_TAG_DECIMAL);
+ buf.extend_from_slice(&unscaled.to_le_bytes());
+ buf.push(*scale as u8);
+ }
+ Some(OrderingValue::String(s)) => {
+ buf.push(SPILL_ORDERING_TAG_STRING);
+ let sb = s.as_bytes();
+ buf.extend_from_slice(&(sb.len() as u32).to_le_bytes());
+ buf.extend_from_slice(sb);
+ }
+ Some(OrderingValue::Default) => buf.push(SPILL_ORDERING_TAG_DEFAULT),
+ Some(OrderingValue::Composite(elems)) => {
+ buf.push(SPILL_ORDERING_TAG_COMPOSITE);
+ buf.extend_from_slice(&(elems.len() as u32).to_le_bytes());
+ for e in elems {
+ encode_ordering_value(buf, Some(e));
+ }
+ }
+ }
+}
+
+/// Read a single byte at `*cursor`, advancing it. Errors if out of bounds.
+fn read_u8(bytes: &[u8], cursor: &mut usize) -> Result<u8> {
+ let b = *bytes.get(*cursor).ok_or_else(|| {
+ CoreError::ReadFileSliceError("spill decode: unexpected end of blob".to_string())
+ })?;
+ *cursor += 1;
+ Ok(b)
+}
+
+/// Decode an ordering value written by [`encode_ordering_value`], advancing
+/// `*cursor` past it.
+fn decode_ordering_value(bytes: &[u8], cursor: &mut usize) -> Result<Option<OrderingValue>> {
+ let tag = read_u8(bytes, cursor)?;
+ match tag {
+ SPILL_ORDERING_TAG_NONE => Ok(None),
+ SPILL_ORDERING_TAG_DEFAULT => Ok(Some(OrderingValue::Default)),
+ SPILL_ORDERING_TAG_LONG => {
+ let end = *cursor + 8;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Long ordering value".to_string(),
+ )
+ })?;
+ let v = i64::from_le_bytes(slice.try_into().expect("8-byte slice"));
+ *cursor = end;
+ Ok(Some(OrderingValue::Long(v)))
+ }
+ SPILL_ORDERING_TAG_DOUBLE => {
+ let end = *cursor + 8;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Double ordering value".to_string(),
+ )
+ })?;
+ let v = f64::from_le_bytes(slice.try_into().expect("8-byte slice"));
+ *cursor = end;
+ Ok(Some(OrderingValue::Double(v)))
+ }
+ SPILL_ORDERING_TAG_DECIMAL => {
+ let end = *cursor + 16;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Decimal ordering value".to_string(),
+ )
+ })?;
+ let unscaled = i128::from_le_bytes(slice.try_into().expect("16-byte slice"));
+ *cursor = end;
+ let scale = read_u8(bytes, cursor)? as i8;
+ Ok(Some(OrderingValue::Decimal { unscaled, scale }))
+ }
+ SPILL_ORDERING_TAG_STRING => {
+ let len_end = *cursor + 4;
+ let len_slice = bytes.get(*cursor..len_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated String ordering length".to_string(),
+ )
+ })?;
+ let len = u32::from_le_bytes(len_slice.try_into().expect("4-byte slice")) as usize;
+ *cursor = len_end;
+ let str_end = *cursor + len;
+ let str_slice = bytes.get(*cursor..str_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated String ordering value".to_string(),
+ )
+ })?;
+ let s = std::str::from_utf8(str_slice)
+ .map_err(|e| {
+ CoreError::ReadFileSliceError(format!(
+ "spill decode: invalid utf8 in String ordering value: {e}"
+ ))
+ })?
+ .to_string();
+ *cursor = str_end;
+ Ok(Some(OrderingValue::String(s)))
+ }
+ SPILL_ORDERING_TAG_COMPOSITE => {
+ let len_end = *cursor + 4;
+ let len_slice = bytes.get(*cursor..len_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated composite ordering length".to_string(),
+ )
+ })?;
+ let n = u32::from_le_bytes(len_slice.try_into().expect("4-byte slice")) as usize;
+ *cursor = len_end;
+ let mut elems = Vec::with_capacity(n);
+ for _ in 0..n {
+ match decode_ordering_value(bytes, cursor)? {
+ Some(v) => elems.push(v),
+ None => {
+ return Err(CoreError::ReadFileSliceError(
+ "spill decode: null element in composite ordering value".to_string(),
+ ));
+ }
+ }
+ }
+ Ok(Some(OrderingValue::Composite(elems)))
+ }
+ other => Err(CoreError::ReadFileSliceError(format!(
Review Comment:
🤖 nit: `is_empty` and `is_delete` are identical — could you remove `is_empty` and let callers use `is_delete` directly? The name `is_empty` implies something subtly different (a record with no meaningful content might not be a delete in a richer model), and having two names for the same predicate creates a maintenance surface where they could drift apart.
<sub><i>⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality.</i></sub>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#issuecomment-5300989054
The lines that miss coverage are codecov artifact: tarpaulin's llvm engine misses lines in async bodies that tests demonstrably execute. The merge semantics are validated scenario-by-scenario against Spark-written reference test fixture tables on both reader versions. Merging this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3788515992
##########
crates/core/src/file_group/reader_v2/resolver.rs:
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Derives a [`ReaderContext`] from a table's resolved configs.
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::error::ConfigError;
+use crate::config::read::HudiReadConfig;
+use crate::config::table::{BaseFileFormatValue, HudiTableConfig};
+use crate::error::CoreError;
+use crate::file_group::reader_v2::buffer::spillable_map;
+use crate::file_group::reader_v2::reader_context::{CONFIG_MERGE_TYPE, MergeMode, ReaderContext};
+use crate::file_group::reader_v2::record_context::RecordContext;
+use crate::file_group::reader_v2::schema_handler::FileGroupReaderSchemaHandler;
+use crate::timeline::selector::InstantRange;
+use std::collections::HashMap;
+
+/// Resolve the MOR reader context from `hudi_configs`, which the caller has
+/// already merged read options into.
+#[allow(dead_code)]
+pub(crate) fn resolve_reader_context(
+ hudi_configs: &HudiConfigs,
+ has_log_files: bool,
+) -> Result<ReaderContext> {
+ let table_path: String = hudi_configs.get(HudiTableConfig::BasePath)?.into();
+
+ // Resolved by `Table::prepare_reader_options` for table-level reads. A
+ // `FileGroupReader` built straight from a base URI never loads the timeline,
+ // so the caller must supply it; defaulting here would silently widen the read.
+ let latest_commit_time: String = hudi_configs
+ .try_get(HudiReadConfig::EndTimestamp)?
+ .ok_or_else(|| ConfigError::NotFound(HudiReadConfig::EndTimestamp.as_ref().to_string()))?
+ .into();
+
+ let merge_mode = resolve_merge_mode(hudi_configs)?;
Review Comment:
Fixed: the resolver now tolerates an unsupported merge mode when there are no log files to merge (commit-time ordering stands in for the mode nothing reads), the merging-read error names `hoodie.read.file.group.reader.version=1`, and end-to-end tests pin the COW, read-optimized, and merging cases.
##########
crates/core/src/schema/delete.rs:
##########
@@ -37,9 +37,54 @@ static DELETE_RECORD_AVRO_SCHEMA_IN_JSON: Lazy<Result<JsonValue>> = Lazy::new(||
.map_err(|e| CoreError::Schema(format!("Failed to parse schema to JSON: {e}")))
});
+/// Union position of `ArrayWrapper`, which carries a list rather than a scalar.
+/// Nothing orders records by a list, so it is rejected rather than mapped.
+const ARRAY_WRAPPER_POSITION: u32 = 12;
+
+/// Replace a wrapped ordering value with the primitive inside it.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records — `LongWrapper`
+/// is a record whose single `value` field is a `long`. The Arrow side wants the
+/// primitive, and [`avro_schema_for_delete_record`] narrows the schema to match,
+/// so the value has to be unwrapped to agree with it.
+///
+/// The narrowed schema has two branches, `[null, <primitive>]`, so the surviving
+/// branch is position 1 regardless of where the wrapper sat in the full union.
+///
+/// Records that are already primitives pass through: this runs over decoded
+/// values, and only the wrapper shape is rewritten.
+pub fn unwrap_ordering_value(delete_record: AvroValue) -> Result<AvroValue> {
+ let AvroValue::Record(mut fields) = delete_record else {
+ return Err(CoreError::Schema(
+ "Expected a record for delete record".to_string(),
+ ));
+ };
+ let Some((_, ordering_val)) = fields.get_mut(2) else {
+ return Err(CoreError::Schema(
+ "Delete record has no orderingVal field".to_string(),
+ ));
+ };
+ if let AvroValue::Union(pos, inner) = ordering_val {
+ if *pos == ARRAY_WRAPPER_POSITION {
Review Comment:
Fixed: `unwrap_ordering_values` decodes ArrayWrapper rows as null ordering cells, so those deletes apply in natural order (warn-logged) instead of failing the read, with a regression test.
##########
crates/core/src/file_group/log_file/content.rs:
##########
@@ -17,35 +17,128 @@
* under the License.
*/
use crate::Result;
-use crate::avro_to_arrow::arrow_array_reader::AvroArrowArrayReader;
use crate::config::HudiConfigs;
use crate::error::CoreError;
-use crate::file_group::log_file::avro::AvroDataBlockContentReader;
+use crate::file_group::log_file::avro::AvroBlockDecoder;
use crate::file_group::log_file::log_block::{
BlockMetadataKey, BlockType, LogBlockContent, LogBlockVersion,
};
use crate::file_group::log_file::log_format::LogFormatVersion;
use crate::file_group::record_batches::RecordBatches;
use crate::hfile::{HFileReader, HFileRecord};
-use crate::schema::delete::{avro_schema_for_delete_record, avro_schema_for_delete_record_list};
-use apache_avro::{Schema as AvroSchema, from_avro_datum};
+use crate::schema::delete::delete_record_list_schema_json;
+use crate::schema::extended_promotion::record_needs_rewrite_for_extended_promotion;
+use crate::schema::parquet_list_norm::normalize_parquet_metadata;
+use crate::schema::resolver::avro_json_to_arrow_schema;
+use crate::storage::RowFilterBuilder;
+use arrow_array::{Array, ArrayRef, ListArray, RecordBatch, StructArray, UnionArray};
+use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
-use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::file::metadata::ParquetMetaDataReader;
use std::collections::HashMap;
use std::io::{Read, Seek};
use std::sync::Arc;
+/// Turn the wrapped ordering values into a plain column.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records, so a decode
+/// against that schema yields a union of one-field structs. The merge wants the
+/// value itself.
+///
+/// A block writes one ordering type, so exactly one branch is populated; that
+/// branch's `value` child is the column. A block mixing branches is rejected
+/// rather than silently reduced to one of them.
+fn unwrap_ordering_values(ordering: &ArrayRef) -> Result<ArrayRef> {
+ let union = ordering
+ .as_any()
+ .downcast_ref::<UnionArray>()
+ .ok_or_else(|| {
+ CoreError::LogBlockError(format!(
+ "Expected orderingVal to be a union, got {}",
+ ordering.data_type()
+ ))
+ })?;
+
+ let mut active: Option<i8> = None;
+ for i in 0..union.len() {
+ let type_id = union.type_id(i);
+ match active {
+ None => active = Some(type_id),
+ Some(seen) if seen == type_id => {}
+ Some(seen) => {
+ return Err(CoreError::LogBlockError(format!(
+ "Delete block mixes ordering types (union branches {seen} and {type_id})"
Review Comment:
Fixed: the null branch is exempt from the mixing check and decodes as null cells that merge as natural-order deletes, matching Hudi, with regression tests for the mixed and all-null shapes.
##########
crates/core/src/file_group/log_file/content.rs:
##########
@@ -17,35 +17,128 @@
* under the License.
*/
use crate::Result;
-use crate::avro_to_arrow::arrow_array_reader::AvroArrowArrayReader;
use crate::config::HudiConfigs;
use crate::error::CoreError;
-use crate::file_group::log_file::avro::AvroDataBlockContentReader;
+use crate::file_group::log_file::avro::AvroBlockDecoder;
use crate::file_group::log_file::log_block::{
BlockMetadataKey, BlockType, LogBlockContent, LogBlockVersion,
};
use crate::file_group::log_file::log_format::LogFormatVersion;
use crate::file_group::record_batches::RecordBatches;
use crate::hfile::{HFileReader, HFileRecord};
-use crate::schema::delete::{avro_schema_for_delete_record, avro_schema_for_delete_record_list};
-use apache_avro::{Schema as AvroSchema, from_avro_datum};
+use crate::schema::delete::delete_record_list_schema_json;
+use crate::schema::extended_promotion::record_needs_rewrite_for_extended_promotion;
+use crate::schema::parquet_list_norm::normalize_parquet_metadata;
+use crate::schema::resolver::avro_json_to_arrow_schema;
+use crate::storage::RowFilterBuilder;
+use arrow_array::{Array, ArrayRef, ListArray, RecordBatch, StructArray, UnionArray};
+use arrow_schema::{DataType, Field, Schema};
use bytes::Bytes;
-use parquet::arrow::arrow_reader::ParquetRecordBatchReader;
+use parquet::arrow::arrow_reader::{
+ ArrowReaderMetadata, ArrowReaderOptions, ParquetRecordBatchReaderBuilder,
+};
+use parquet::file::metadata::ParquetMetaDataReader;
use std::collections::HashMap;
use std::io::{Read, Seek};
use std::sync::Arc;
+/// Turn the wrapped ordering values into a plain column.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records, so a decode
+/// against that schema yields a union of one-field structs. The merge wants the
+/// value itself.
+///
+/// A block writes one ordering type, so exactly one branch is populated; that
+/// branch's `value` child is the column. A block mixing branches is rejected
+/// rather than silently reduced to one of them.
+fn unwrap_ordering_values(ordering: &ArrayRef) -> Result<ArrayRef> {
+ let union = ordering
+ .as_any()
+ .downcast_ref::<UnionArray>()
+ .ok_or_else(|| {
+ CoreError::LogBlockError(format!(
+ "Expected orderingVal to be a union, got {}",
+ ordering.data_type()
+ ))
+ })?;
+
+ let mut active: Option<i8> = None;
+ for i in 0..union.len() {
+ let type_id = union.type_id(i);
+ match active {
+ None => active = Some(type_id),
+ Some(seen) if seen == type_id => {}
+ Some(seen) => {
+ return Err(CoreError::LogBlockError(format!(
+ "Delete block mixes ordering types (union branches {seen} and {type_id})"
+ )));
+ }
+ }
+ }
+ let Some(active) = active else {
+ return Ok(ordering.clone());
+ };
+
+ let child = union.child(active);
+ // Null is a branch like any other; there is nothing to unwrap out of it.
Review Comment:
Addressed with the thread above: null and ArrayWrapper branches decode per row as natural-order deletes, and only a genuine two-scalar-type mix still errors.
##########
crates/core/src/file_group/file_slice.rs:
##########
@@ -29,7 +29,20 @@ use std::path::PathBuf;
/// a [FileSlice] is a logical group of [BaseFile] and [LogFile]s.
#[derive(Clone, Debug)]
pub struct FileSlice {
- pub base_file: BaseFile,
+ /// The base file, when the slice has one.
+ ///
+ /// A slice written by inserts that went straight to log files has none —
+ /// Flink ingestion, a bucket index's first write to a bucket, and any
+ /// merge-on-read file group before its first compaction all produce these.
+ pub base_file: Option<BaseFile>,
Review Comment:
Done: the PR description's behaviour-changes section now calls out the `FileSlice::base_file` / `base_file_relative_path()` API break for crates.io consumers upgrading to 0.5.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "linliu-code (via GitHub)" <gi...@apache.org>.
linliu-code commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3786088804
##########
crates/core/src/file_group/reader_v2/input_split.rs:
##########
@@ -0,0 +1,259 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.InputSplit`.
Review Comment:
Fixed. The `#![allow(dead_code)]` is gone rather than moved — see the discussion on `reader_v2/mod.rs`: this is the change that wires the reader in, so "nothing consumes it yet" was no longer true, and dropping the attribute surfaced two genuinely dead fields (`InputSplit::start` and `length`, always `0`/`-1`, never read) which are now removed. The module doc is a single contiguous block.
##########
crates/core/src/file_group/reader_v2/iterator_mode.rs:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.IteratorMode`.
Review Comment:
Fixed the same way as `input_split.rs`: the attribute is removed rather than relocated, and the module doc is contiguous.
##########
crates/core/src/file_group/reader_v2/iterator_mode.rs:
##########
@@ -0,0 +1,57 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.IteratorMode`.
+//!
+//! Controls what form the output records take when iterating.
+
+/// The mode in which the file group reader yields records.
+///
+/// In Java Hudi, this controls whether the iterator produces engine-native
+/// records, HoodieRecord wrappers, or just record keys.
+///
+/// In hudi-rs, we always work with Arrow RecordBatch, so `EngineRecord`
+/// is the primary mode. The other modes are kept for API symmetry.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
+pub enum IteratorMode {
+ /// Yield engine-native records (Arrow RecordBatch in Rust).
+ #[default]
+ EngineRecord,
+
+ /// Yield HoodieRecord wrappers (not yet implemented in Rust).
+ HoodieRecord,
+
+ /// Yield only record keys (not yet implemented in Rust).
+ RecordKey,
+}
+
+impl IteratorMode {
Review Comment:
Fixed — implemented `std::str::FromStr`, so `"ENGINE_RECORD".parse::<IteratorMode>()` works and the error is a `ConfigError::InvalidValue` naming the value rather than a bare `None`.
Worth noting it had no callers at all, so the idiomatic spelling cost nothing.
##########
crates/core/src/file_group/reader_v2/buffered_record.rs:
##########
@@ -0,0 +1,1094 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+#![allow(dead_code)]
+
+//! Mirrors `org.apache.hudi.common.table.read.BufferedRecord` and
+//! `org.apache.hudi.common.model.DeleteRecord`.
+//!
+//! In Java Hudi, `BufferedRecord<T>` wraps a single engine-native record
+//! with its key, ordering value, and operation type. In hudi-rs, the
+//! "engine record" is an Arrow `RecordBatch`.
+//!
+//! During log scanning, records are accumulated into the record buffer.
+//! At read time, base file records are merged with log records to produce
+//! the final output.
+//!
+//! ## In-memory representation (A2)
+//!
+//! Per the A2 alignment work (05-a2a1-design.md), the in-memory payload is a
+//! zero-copy [`RecordPayload::BatchRef`] into the shared source batch that the
+//! row was decoded from — NOT a per-row Arrow IPC blob. IPC serialization is a
+//! spill-only concern (A1) and lives in
+//! [`row_serde`](super::row_serde) (`to_binary_row` /
+//! `from_binary`); it is no longer paid on the in-memory merge hot path. The
+//! data-driven motivation: the per-row IPC blob cost ~27x the build memory and
+//! ~12x the build CPU of the batch-ref representation (spike Part 1).
+
+use std::sync::Arc;
+
+use arrow_array::RecordBatch;
+use arrow_buffer::i256;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::file_group::reader_v2::row_serde;
+
+// ── Spill value encoding (A1) ──────────────────────────────────────────────
+// On spill, a `BufferedRecord` is serialized to a self-describing byte blob
+// stored as the RocksDB value. The record key is the RocksDB key, so it is NOT
+// repeated in the value. The layout is:
+//
+// [payload_tag: u8]
+// 0x00 = Delete (no further payload bytes)
+// 0x01 = Data (followed by an Arrow IPC single-row stream)
+// [ordering_tag: u8]
+// 0x00 = None
+// 0x01 = Long followed by 8 bytes (i64 little-endian)
+// 0x02 = String followed by [len: u32 LE][utf8 bytes]
+// [if payload_tag == Data] Arrow IPC stream bytes (rest of the blob)
+//
+// All multi-byte integers are little-endian. The format is internal to the
+// reader's own spill round-trip (never persisted across runs), so no on-disk
+// version negotiation is needed — the spill dir is created and destroyed within
+// a single read (see `SpillableRecordMap`).
+
+/// Spill payload tag: the record is a delete tombstone (no data follows).
+const SPILL_PAYLOAD_TAG_DELETE: u8 = 0x00;
+/// Spill payload tag: the record carries data (an Arrow IPC stream follows the
+/// ordering-value header).
+const SPILL_PAYLOAD_TAG_DATA: u8 = 0x01;
+/// Spill ordering-value tag: no ordering value.
+const SPILL_ORDERING_TAG_NONE: u8 = 0x00;
+/// Spill ordering-value tag: `OrderingValue::Long` (8-byte i64 follows).
+const SPILL_ORDERING_TAG_LONG: u8 = 0x01;
+/// Spill ordering-value tag: `OrderingValue::String` (u32 length + utf8 follow).
+const SPILL_ORDERING_TAG_STRING: u8 = 0x02;
+/// Spill ordering-value tag: `OrderingValue::Composite` (u32 element count, then
+/// each element encoded with this same tag scheme).
+const SPILL_ORDERING_TAG_COMPOSITE: u8 = 0x03;
+/// Spill ordering-value tag: `OrderingValue::Double` (8-byte IEEE-754 f64 follows).
+const SPILL_ORDERING_TAG_DOUBLE: u8 = 0x04;
+/// Spill ordering-value tag: `OrderingValue::Decimal` (16-byte little-endian i128
+/// unscaled value, then a 1-byte i8 scale follow).
+const SPILL_ORDERING_TAG_DECIMAL: u8 = 0x05;
+/// Spill ordering-value tag: `OrderingValue::Default` (no payload). The null-coerced
+/// default ordering value (Java `HoodieRecord.DEFAULT_ORDERING_VALUE` == `Integer(0)`).
+const SPILL_ORDERING_TAG_DEFAULT: u8 = 0x06;
+
+/// In-memory payload of a buffered record (A2).
+///
+/// The default representation is a zero-copy reference into a shared source
+/// batch ([`BatchRef`](RecordPayload::BatchRef)). Serialization (Arrow IPC)
+/// happens only on spill (A1), never on the in-memory hot path — so this enum
+/// deliberately carries no binary variant.
+///
+/// ## Pinning caveat
+///
+/// A [`BatchRef`](RecordPayload::BatchRef) keeps its WHOLE source
+/// `Arc<RecordBatch>` alive (a 1-row [`RecordBatch::slice`] still references the
+/// parent's full buffers). When survivors are sparse across many source batches,
+/// the merge map can pin far more Arrow memory than the live rows occupy. The
+/// `compact_pinned_batches` safety valve (see `key_based.rs`) re-batches the
+/// survivors of sparsely-populated source batches and repoints their entries to
+/// release the dead-row memory.
+#[derive(Debug, Clone)]
+pub enum RecordPayload {
+ /// Zero-copy reference into a shared source batch (the A2 default). `row_idx`
+ /// addresses a single row within `batch`. Reading the payload slices that row
+ /// out via [`RecordBatch::slice`] (cheap, shares buffers).
+ BatchRef {
+ /// The shared source batch this row was decoded from. Interned once per
+ /// decoded block batch (one `Arc` per batch) so that `Arc::as_ptr`
+ /// identity is a valid grouping key for compaction.
+ batch: Arc<RecordBatch>,
+ /// Row index of this record within `batch`.
+ row_idx: usize,
+ },
+ /// Owned single-row batch. Produced by `compact_pinned_batches` (a freshly
+ /// re-batched survivor set) or for records that have no shared source batch.
+ Owned(RecordBatch),
+ /// Tombstone — the record represents a deletion and carries no payload.
+ Delete,
+}
+
+impl RecordPayload {
+ /// Returns the payload as a single-row `RecordBatch`, or `None` for a delete.
+ ///
+ /// `BatchRef` slices the addressed row out of its source batch (zero-copy
+ /// buffer slice); `Owned` clones the held batch. Mirrors the read side of
+ /// Java's `BufferedRecord.getRecord()`.
+ pub fn get_record(&self) -> Option<RecordBatch> {
+ match self {
+ RecordPayload::BatchRef { batch, row_idx } => Some(batch.slice(*row_idx, 1)),
+ RecordPayload::Owned(batch) => Some(batch.clone()),
+ RecordPayload::Delete => None,
+ }
+ }
+
+ /// Consume the payload and return its single-row `RecordBatch`, or `None`
+ /// for a delete. The move-not-clone counterpart to [`get_record`](Self::get_record).
+ pub fn into_record(self) -> Option<RecordBatch> {
+ match self {
+ RecordPayload::BatchRef { batch, row_idx } => Some(batch.slice(row_idx, 1)),
+ RecordPayload::Owned(batch) => Some(batch),
+ RecordPayload::Delete => None,
+ }
+ }
+
+ /// Returns true if this payload is a delete tombstone.
+ pub fn is_delete(&self) -> bool {
+ matches!(self, RecordPayload::Delete)
+ }
+}
+
+/// The universal record envelope flowing through the merge pipeline.
+///
+/// Mirrors Java's `BufferedRecord<T>`. In Rust/Arrow, a single "record" is a
+/// single-row view ([`RecordPayload::BatchRef`]) into a shared decoded batch;
+/// the merge map stores one of these per live key.
+///
+/// During the log scanning phase, `BufferedRecord`s are stored in the record
+/// buffer's map keyed by record key.
+#[derive(Debug, Clone)]
+pub struct BufferedRecord {
+ /// The record key extracted from the record.
+ pub record_key: String,
+
+ /// The in-memory record payload (zero-copy batch ref, owned batch, or delete).
+ pub payload: RecordPayload,
+
+ /// The ordering value used for merge conflict resolution.
+ /// Higher ordering value wins during delta merge.
+ pub ordering_value: Option<OrderingValue>,
+}
+
+impl BufferedRecord {
+ /// Create a new data record from a zero-copy reference into a shared source
+ /// batch (the A2 hot-path constructor).
+ ///
+ /// `batch` must be the interned `Arc` for the decoded block batch (one `Arc`
+ /// per batch) so that `Arc::as_ptr` grouping during compaction is valid.
+ pub fn new_batch_ref(
+ record_key: String,
+ batch: Arc<RecordBatch>,
+ row_idx: usize,
+ ordering_value: Option<OrderingValue>,
+ ) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::BatchRef { batch, row_idx },
+ ordering_value,
+ }
+ }
+
+ /// Create a new data record from an owned single-row batch.
+ ///
+ /// Used by tests and by call sites that hold a single-row batch with no
+ /// shared source (e.g. base-record construction in tests). The hot path
+ /// uses [`new_batch_ref`](Self::new_batch_ref) instead.
+ pub fn new_data(
+ record_key: String,
+ data: RecordBatch,
+ ordering_value: Option<OrderingValue>,
+ ) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::Owned(data),
+ ordering_value,
+ }
+ }
+
+ /// Create a new delete record (tombstone).
+ pub fn new_delete(record_key: String, ordering_value: Option<OrderingValue>) -> Self {
+ Self {
+ record_key,
+ payload: RecordPayload::Delete,
+ ordering_value,
+ }
+ }
+
+ /// Returns true if this record represents a deletion.
+ pub fn is_delete(&self) -> bool {
+ self.payload.is_delete()
+ }
+
+ /// Returns true if this record has no data payload (a delete tombstone).
+ pub fn is_empty(&self) -> bool {
+ self.payload.is_delete()
+ }
+
+ /// Return the record data as a single-row `RecordBatch`.
+ ///
+ /// Mirrors Java's `BufferedRecord.getRecord()`. Delegates to
+ /// [`RecordPayload::get_record`]: `BatchRef` slices the row out (zero-copy),
+ /// `Owned` clones, `Delete` returns `None`.
+ pub fn get_record(&self) -> Option<RecordBatch> {
+ self.payload.get_record()
+ }
+
+ /// Consume the record and return its data batch. The move-not-clone
+ /// counterpart to [`get_record`](Self::get_record), used on the merge-output
+ /// hot path where the record is no longer needed afterward.
+ pub fn into_record(self) -> Option<RecordBatch> {
+ self.payload.into_record()
+ }
+
+ /// Serialize this record to a self-describing spill blob (**spill-only, A1**).
+ ///
+ /// Used by [`SpillableRecordMap`](super::buffer::spillable_map::SpillableRecordMap)
+ /// to store an entry in the on-disk (RocksDB) tier when the in-memory budget
+ /// is exhausted. The record key is NOT included (it is the RocksDB key); the
+ /// blob carries the payload (delete tombstone or a single-row Arrow IPC
+ /// stream) and the ordering value. See the module-level spill-encoding notes.
+ ///
+ /// Start with single-row IPC for correctness; a batched-spill optimization
+ /// (group survivors by source batch, spill multi-row blobs) is tracked as B5
+ /// in the backlog (05-a2a1-design.md) and deliberately NOT built here.
+ pub fn to_spill_bytes(&self) -> Vec<u8> {
+ let mut buf = Vec::new();
+ match self.get_record() {
+ None => buf.push(SPILL_PAYLOAD_TAG_DELETE),
+ Some(batch) => {
+ buf.push(SPILL_PAYLOAD_TAG_DATA);
+ encode_ordering_value(&mut buf, self.ordering_value.as_ref());
+ // Arrow IPC single-row stream (the spill serialization primitive,
+ // row_serde::to_binary_row). For deletes we skip this entirely.
+ let ipc = row_serde::to_binary_row(&batch.schema(), &batch);
+ buf.extend_from_slice(&ipc);
+ return buf;
+ }
+ }
+ // Delete path: still record the ordering value (it can matter when a
+ // spilled delete tombstone is later compared during the base merge).
+ encode_ordering_value(&mut buf, self.ordering_value.as_ref());
+ buf
+ }
+
+ /// Reconstruct a record from a spill blob produced by
+ /// [`to_spill_bytes`](Self::to_spill_bytes) (**spill-only, A1**).
+ ///
+ /// `record_key` is supplied by the caller (it is the RocksDB key, not stored
+ /// in the blob). Returns a typed [`CoreError`] on a malformed blob rather
+ /// than panicking — a corrupt spill entry is a recoverable read-path error,
+ /// not an internal invariant violation.
+ pub fn from_spill_bytes(record_key: String, bytes: &[u8]) -> Result<Self> {
+ let mut cursor = 0usize;
+ let payload_tag = read_u8(bytes, &mut cursor)?;
+ match payload_tag {
+ SPILL_PAYLOAD_TAG_DELETE => {
+ let ordering_value = decode_ordering_value(bytes, &mut cursor)?;
+ Ok(BufferedRecord::new_delete(record_key, ordering_value))
+ }
+ SPILL_PAYLOAD_TAG_DATA => {
+ let ordering_value = decode_ordering_value(bytes, &mut cursor)?;
+ let batch = row_serde::from_binary(&bytes[cursor..])?;
+ // Reloaded from disk: there is no shared source batch to reference,
+ // so the payload is necessarily `Owned`.
+ Ok(BufferedRecord::new_data(record_key, batch, ordering_value))
+ }
+ other => Err(CoreError::ReadFileSliceError(format!(
+ "spill decode: unknown payload tag {other:#04x}"
+ ))),
+ }
+ }
+}
+
+/// Append an ordering value to a spill blob using the tag scheme documented at
+/// the module level.
+fn encode_ordering_value(buf: &mut Vec<u8>, ordering_value: Option<&OrderingValue>) {
+ match ordering_value {
+ None => buf.push(SPILL_ORDERING_TAG_NONE),
+ Some(OrderingValue::Long(v)) => {
+ buf.push(SPILL_ORDERING_TAG_LONG);
+ buf.extend_from_slice(&v.to_le_bytes());
+ }
+ Some(OrderingValue::Double(v)) => {
+ buf.push(SPILL_ORDERING_TAG_DOUBLE);
+ buf.extend_from_slice(&v.to_le_bytes());
+ }
+ Some(OrderingValue::Decimal { unscaled, scale }) => {
+ buf.push(SPILL_ORDERING_TAG_DECIMAL);
+ buf.extend_from_slice(&unscaled.to_le_bytes());
+ buf.push(*scale as u8);
+ }
+ Some(OrderingValue::String(s)) => {
+ buf.push(SPILL_ORDERING_TAG_STRING);
+ let sb = s.as_bytes();
+ buf.extend_from_slice(&(sb.len() as u32).to_le_bytes());
+ buf.extend_from_slice(sb);
+ }
+ Some(OrderingValue::Default) => buf.push(SPILL_ORDERING_TAG_DEFAULT),
+ Some(OrderingValue::Composite(elems)) => {
+ buf.push(SPILL_ORDERING_TAG_COMPOSITE);
+ buf.extend_from_slice(&(elems.len() as u32).to_le_bytes());
+ for e in elems {
+ encode_ordering_value(buf, Some(e));
+ }
+ }
+ }
+}
+
+/// Read a single byte at `*cursor`, advancing it. Errors if out of bounds.
+fn read_u8(bytes: &[u8], cursor: &mut usize) -> Result<u8> {
+ let b = *bytes.get(*cursor).ok_or_else(|| {
+ CoreError::ReadFileSliceError("spill decode: unexpected end of blob".to_string())
+ })?;
+ *cursor += 1;
+ Ok(b)
+}
+
+/// Decode an ordering value written by [`encode_ordering_value`], advancing
+/// `*cursor` past it.
+fn decode_ordering_value(bytes: &[u8], cursor: &mut usize) -> Result<Option<OrderingValue>> {
+ let tag = read_u8(bytes, cursor)?;
+ match tag {
+ SPILL_ORDERING_TAG_NONE => Ok(None),
+ SPILL_ORDERING_TAG_DEFAULT => Ok(Some(OrderingValue::Default)),
+ SPILL_ORDERING_TAG_LONG => {
+ let end = *cursor + 8;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Long ordering value".to_string(),
+ )
+ })?;
+ let v = i64::from_le_bytes(slice.try_into().expect("8-byte slice"));
+ *cursor = end;
+ Ok(Some(OrderingValue::Long(v)))
+ }
+ SPILL_ORDERING_TAG_DOUBLE => {
+ let end = *cursor + 8;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Double ordering value".to_string(),
+ )
+ })?;
+ let v = f64::from_le_bytes(slice.try_into().expect("8-byte slice"));
+ *cursor = end;
+ Ok(Some(OrderingValue::Double(v)))
+ }
+ SPILL_ORDERING_TAG_DECIMAL => {
+ let end = *cursor + 16;
+ let slice = bytes.get(*cursor..end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated Decimal ordering value".to_string(),
+ )
+ })?;
+ let unscaled = i128::from_le_bytes(slice.try_into().expect("16-byte slice"));
+ *cursor = end;
+ let scale = read_u8(bytes, cursor)? as i8;
+ Ok(Some(OrderingValue::Decimal { unscaled, scale }))
+ }
+ SPILL_ORDERING_TAG_STRING => {
+ let len_end = *cursor + 4;
+ let len_slice = bytes.get(*cursor..len_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated String ordering length".to_string(),
+ )
+ })?;
+ let len = u32::from_le_bytes(len_slice.try_into().expect("4-byte slice")) as usize;
+ *cursor = len_end;
+ let str_end = *cursor + len;
+ let str_slice = bytes.get(*cursor..str_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated String ordering value".to_string(),
+ )
+ })?;
+ let s = std::str::from_utf8(str_slice)
+ .map_err(|e| {
+ CoreError::ReadFileSliceError(format!(
+ "spill decode: invalid utf8 in String ordering value: {e}"
+ ))
+ })?
+ .to_string();
+ *cursor = str_end;
+ Ok(Some(OrderingValue::String(s)))
+ }
+ SPILL_ORDERING_TAG_COMPOSITE => {
+ let len_end = *cursor + 4;
+ let len_slice = bytes.get(*cursor..len_end).ok_or_else(|| {
+ CoreError::ReadFileSliceError(
+ "spill decode: truncated composite ordering length".to_string(),
+ )
+ })?;
+ let n = u32::from_le_bytes(len_slice.try_into().expect("4-byte slice")) as usize;
+ *cursor = len_end;
+ let mut elems = Vec::with_capacity(n);
+ for _ in 0..n {
+ match decode_ordering_value(bytes, cursor)? {
+ Some(v) => elems.push(v),
+ None => {
+ return Err(CoreError::ReadFileSliceError(
+ "spill decode: null element in composite ordering value".to_string(),
+ ));
+ }
+ }
+ }
+ Ok(Some(OrderingValue::Composite(elems)))
+ }
+ other => Err(CoreError::ReadFileSliceError(format!(
Review Comment:
Fixed — `is_empty` removed, callers use `is_delete`. Your reasoning is the right one: they had identical bodies, `is_empty` implies something subtly different from "is a delete tombstone", and two names for one predicate is a drift surface. The three call sites were all in tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3787843003
##########
crates/core/src/file_group/reader_v2/resolver.rs:
##########
@@ -0,0 +1,908 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Derives a [`ReaderContext`] from a table's resolved configs.
+
+use crate::Result;
+use crate::config::HudiConfigs;
+use crate::config::error::ConfigError;
+use crate::config::read::HudiReadConfig;
+use crate::config::table::{BaseFileFormatValue, HudiTableConfig};
+use crate::error::CoreError;
+use crate::file_group::reader_v2::buffer::spillable_map;
+use crate::file_group::reader_v2::reader_context::{CONFIG_MERGE_TYPE, MergeMode, ReaderContext};
+use crate::file_group::reader_v2::record_context::RecordContext;
+use crate::file_group::reader_v2::schema_handler::FileGroupReaderSchemaHandler;
+use crate::timeline::selector::InstantRange;
+use std::collections::HashMap;
+
+/// Resolve the MOR reader context from `hudi_configs`, which the caller has
+/// already merged read options into.
+#[allow(dead_code)]
+pub(crate) fn resolve_reader_context(
+ hudi_configs: &HudiConfigs,
+ has_log_files: bool,
+) -> Result<ReaderContext> {
+ let table_path: String = hudi_configs.get(HudiTableConfig::BasePath)?.into();
+
+ // Resolved by `Table::prepare_reader_options` for table-level reads. A
+ // `FileGroupReader` built straight from a base URI never loads the timeline,
+ // so the caller must supply it; defaulting here would silently widen the read.
+ let latest_commit_time: String = hudi_configs
+ .try_get(HudiReadConfig::EndTimestamp)?
+ .ok_or_else(|| ConfigError::NotFound(HudiReadConfig::EndTimestamp.as_ref().to_string()))?
+ .into();
+
+ let merge_mode = resolve_merge_mode(hudi_configs)?;
+ let base_file_format = BaseFileFormatValue::resolve_from_configs(hudi_configs, None)?;
+ let instant_range = resolve_instant_range(hudi_configs)?;
+ let (table_config, hoodie_reader_config) = partition_configs(hudi_configs);
+
+ Ok(ReaderContext {
+ table_path,
+ latest_commit_time,
+ merge_mode: merge_mode.as_ref().to_string(),
+ base_file_format: base_file_format.as_ref().to_string(),
+ has_log_files,
+ instant_range: Some(instant_range),
+ table_config,
+ hoodie_reader_config,
+ should_merge_use_record_position: resolve_use_record_position(hudi_configs)?,
+ // Only one iterator mode is implemented.
+ iterator_mode: "ENGINE_RECORD".to_string(),
+ // Dispatch is on `merge_mode`; the strategy id is carried, not consulted.
+ merge_strategy_id: String::new(),
+ // No bootstrap support in this crate.
+ has_bootstrap_base_file: false,
+ needs_bootstrap_merge: false,
+ enable_logical_timestamp_field_repair: false,
+ // Predicate pushdown into the merge path has no caller here, so no
+ // filter is installed and the primary-key-safety gate is irrelevant.
+ row_filter_builder: None,
+ mor_pk_safe: false,
+ // The table-version < 8 completion gate needs a timeline the caller
+ // has not loaded; leaving it unset keeps the gate a no-op.
+ completion_gate_inputs: None,
Review Comment:
This is the only production producer of `completion_gate_inputs`, so Gate 3 in `forward_scan_pass1` is inert for every read through this crate — the straddling case it exists for (writer A inflight at T1, writer B commits T2 > T1, so A's uncommitted blocks pass Gate 2) still merges. Not a regression — version 1 has no committed-set check either — but `Table` already holds the timeline this needs. Is wiring it through a follow-up, or deliberately off?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/schema/delete.rs:
##########
@@ -37,9 +37,54 @@ static DELETE_RECORD_AVRO_SCHEMA_IN_JSON: Lazy<Result<JsonValue>> = Lazy::new(||
.map_err(|e| CoreError::Schema(format!("Failed to parse schema to JSON: {e}")))
});
+/// Union position of `ArrayWrapper`, which carries a list rather than a scalar.
+/// Nothing orders records by a list, so it is rejected rather than mapped.
+const ARRAY_WRAPPER_POSITION: u32 = 12;
+
+/// Replace a wrapped ordering value with the primitive inside it.
+///
+/// Hudi writes `orderingVal` as a union of per-type wrapper records — `LongWrapper`
+/// is a record whose single `value` field is a `long`. The Arrow side wants the
+/// primitive, and [`avro_schema_for_delete_record`] narrows the schema to match,
+/// so the value has to be unwrapped to agree with it.
+///
+/// The narrowed schema has two branches, `[null, <primitive>]`, so the surviving
+/// branch is position 1 regardless of where the wrapper sat in the full union.
+///
+/// Records that are already primitives pass through: this runs over decoded
+/// values, and only the wrapper shape is rewritten.
+pub fn unwrap_ordering_value(delete_record: AvroValue) -> Result<AvroValue> {
+ let AvroValue::Record(mut fields) = delete_record else {
+ return Err(CoreError::Schema(
+ "Expected a record for delete record".to_string(),
+ ));
+ };
+ let Some((_, ordering_val)) = fields.get_mut(2) else {
+ return Err(CoreError::Schema(
+ "Delete record has no orderingVal field".to_string(),
+ ));
+ };
+ if let AvroValue::Union(pos, inner) = ordering_val {
+ if *pos == ARRAY_WRAPPER_POSITION {
Review Comment:
`ArrayWrapper` is what multiple ordering fields serialize to (HUDI-9569), and composite ordering is otherwise supported here — so a table with `hoodie.table.ordering.fields=ts,seq` reads fine until its first delete block, then the whole read errors. Could this degrade to natural order, as other unusable ordering values do? All six `table_delete_ord_*` fixtures are single-field, so the sweep can't see it.
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/engine.rs:
##########
@@ -0,0 +1,2077 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! The merge-on-read file group reader.
+//!
+//! Ported wholesale; nothing consumes it yet, so its items are unreachable
+//! from the crate's call graph until it is wired in.
+
+use crate::Result;
+use crate::config::table::BaseFileFormatValue;
+use crate::error::CoreError;
+use crate::file_group::base_file::reader::{
+ BaseFileReadOptions, BaseFileReader, create_base_file_reader,
+};
+use crate::file_group::reader_v2::buffer::BufferType;
+use crate::file_group::reader_v2::buffer::loader::{
+ DefaultFileGroupRecordBufferLoader, FileGroupRecordBufferLoader,
+};
+use crate::file_group::reader_v2::buffer::record_positions::ROW_INDEX_TEMPORARY_COLUMN_NAME;
+use crate::file_group::reader_v2::buffered_record_converter::BufferedRecordConverter;
+use crate::file_group::reader_v2::input_split::InputSplit;
+use crate::file_group::reader_v2::iterator_mode::IteratorMode;
+use crate::file_group::reader_v2::merge_iterator::{
+ DEFAULT_BATCH_SIZE, FileGroupMergeIterator, StreamStatsHandle, new_stream_stats_handle,
+};
+use crate::file_group::reader_v2::output_converter::OutputConverter;
+use crate::file_group::reader_v2::profiling::profile_once;
+use crate::file_group::reader_v2::read_stats::HoodieReadStats;
+use crate::file_group::reader_v2::reader_context::ReaderContext;
+use crate::file_group::reader_v2::reader_parameters::ReaderParameters;
+use crate::file_group::reader_v2::schema_handler::FileGroupReaderSchemaHandler;
+use crate::storage::{RowFilterBuilder, Storage};
+use arrow_array::RecordBatch;
+use arrow_schema::SchemaRef;
+use std::str::FromStr;
+use std::sync::Arc;
+
+/// The top-level file group reader orchestrator.
+///
+/// Mirrors Java's `org.apache.hudi.common.table.read.HoodieFileGroupReader<T>`.
+///
+/// This is the main entry point for reading a file group. It:
+/// 1. Accepts an `InputSplit` describing what to read (base file + log files)
+/// 2. Creates the [`FileGroupReaderSchemaHandler`] from `data_schema` + `requested_schema`
+/// 3. Creates base file iterators via storage
+/// 4. Delegates log scanning + buffer creation to `FileGroupRecordBufferLoader`
+/// 5. Merges base file records with log records via the buffer
+/// 6. Projects output back to `requested_schema` via `OutputConverter`
+///
+/// ## Construction
+///
+/// Use [`HoodieFileGroupReader::builder()`] for the builder pattern, or construct
+/// directly with [`HoodieFileGroupReader::new()`].
+pub struct HoodieFileGroupReader {
+ // ── Context (mirrors Java's HoodieReaderContext<T>) ────────────────
+ /// Reader context carrying merge mode, instant range, and config maps.
+ reader_context: Arc<ReaderContext>,
+
+ /// Storage for reading base files and log files.
+ storage: Arc<Storage>,
+
+ // ── Input ──────────────────────────────────────────────────────────
+ /// Describes what to read: base file, log files, partition path.
+ input_split: InputSplit,
+
+ // ── Configuration ──────────────────────────────────────────────────
+ /// Reader flags: use_record_position, emit_delete, sort_output, etc.
+ reader_parameters: ReaderParameters,
+
+ /// The current iterator mode.
+ #[allow(dead_code)]
+ iterator_mode: IteratorMode,
+
+ // ── Schema (mirrors Java's readerContext.getSchemaHandler()) ───────
+ /// Schema handler created in the constructor from `data_schema` +
+ /// `requested_schema`, exactly like Java lines 119-121.
+ /// Owns the `required_schema` used for base file projection and the
+ /// `output_converter` used for final projection.
+ schema_handler: FileGroupReaderSchemaHandler,
+
+ // ── Strategy ───────────────────────────────────────────────────────
+ /// Buffer loader: selects buffer impl + triggers log scan.
+ record_buffer_loader: DefaultFileGroupRecordBufferLoader,
+
+ // ── Mutable state (populated during read) ──────────────────────────
+ // NOTE: the record buffer and base-file batches are not stored on the
+ // reader — they are local to `init_record_iterators` and owned by the
+ // returned `FileGroupMergeIterator` for the rest of the read.
+ /// Optional converter for projecting/transforming output records.
+ /// Mirrors Java's `Option<UnaryOperator<T>> outputConverter`.
+ output_converter: Option<Box<dyn OutputConverter>>,
+
+ /// Read statistics accumulator.
+ read_stats: HoodieReadStats,
+
+ /// Stage-timing sink shared with the [`FileGroupMergeIterator`] returned by
+ /// [`Self::open`]. The streaming iterator
+ /// owns the buffer once `open()` returns, so the merge-phase timings
+ /// (final_merge_ms, output_build_ms) and the update-processor
+ /// insert/update/delete counts are accumulated through this handle during
+ /// iteration and drained back into [`Self::read_stats`] by [`Self::read`]
+ /// after the stream is exhausted. Wrapped in `Arc<Mutex<…>>` because the FFI
+ /// path requires the iterator to be `Send` (it is boxed into an
+ /// `FFI_ArrowArrayStream`); the lock is taken once per emitted chunk, so the
+ /// cost is negligible against the per-chunk merge work. The FFI path never
+ /// reads these stats back — only `read()`-based callers do.
+ stream_stats: StreamStatsHandle,
+
+ /// Valid block instants from log scanning.
+ valid_block_instants: Vec<String>,
+
+ /// Converter for engine records to [`BufferedRecord`].
+ /// Mirrors Java's `BufferedRecordConverter<T> bufferedRecordConverter`.
+ buffered_record_converter: Option<Box<dyn BufferedRecordConverter>>,
+ // NOTE: the optional parquet `RowFilter` builder used to live
+ // on this struct. It now lives on `reader_context` so the same builder is
+ // visible to (a) the base parquet read here, and (b) the parquet log
+ // block decoder in `file_group::log_file::content::Decoder`. The gate
+ // (CoW || mor_pk_safe) lives at the use sites; this file's gate is at
+ // `make_base_file_source` below.
+}
+
+/// Base-file read options carrying an optional pushdown predicate, and the
+/// row-position column when the merge is by position.
+///
+/// The three base reads below differ only in projection, so both are attached in
+/// one place — a read that silently lost the filter would return extra rows
+/// rather than fail, which is the hard kind of bug to notice, and one that lost
+/// the row-position column would fail in the buffer with the column named but
+/// not the read that dropped it.
+fn base_read_options(
+ row_filter: Option<RowFilterBuilder>,
+ use_record_position: bool,
+) -> BaseFileReadOptions {
+ let mut options = BaseFileReadOptions::new();
+ if let Some(row_filter) = row_filter {
+ options = options.with_row_filter(row_filter);
+ }
+ if use_record_position {
+ options = options.with_row_index_column(ROW_INDEX_TEMPORARY_COLUMN_NAME);
+ }
+ options
+}
+
+/// `schema` without the internal row-position column.
+///
+/// The column belongs to the base read and the position buffer; it is not the
+/// table's, so it must not reach a caller. Every schema derived from a base
+/// source's own schema goes through here.
+fn without_row_index(schema: SchemaRef) -> SchemaRef {
+ if schema
+ .column_with_name(ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ .is_none()
+ {
+ return schema;
+ }
+ Arc::new(arrow_schema::Schema::new(
+ schema
+ .fields()
+ .iter()
+ .filter(|f| f.name() != ROW_INDEX_TEMPORARY_COLUMN_NAME)
+ .cloned()
+ .collect::<Vec<_>>(),
+ ))
+}
+
+impl HoodieFileGroupReader {
+ /// Create a new file group reader.
+ ///
+ /// Mirrors Java's `HoodieFileGroupReader(readerContext, storage, tablePath,
+ /// latestCommitTime, dataSchema, requestedSchema, ...)` constructor.
+ ///
+ /// The constructor:
+ /// 1. Creates a [`FileGroupReaderSchemaHandler`] from `data_schema` +
+ /// `requested_schema` (Java lines 119-121)
+ /// 2. Calls `prepare_required_schema()` to compute the `required_schema`
+ /// (Java: automatic in `FileGroupReaderSchemaHandler` constructor, line 105)
+ /// 3. Obtains the `output_converter` from the schema handler (Java line 122)
+ ///
+ /// # Arguments
+ /// * `reader_context` — Engine context with merge mode, ordering fields, table config.
+ /// * `storage` — Storage layer for reading base files and log files.
+ /// * `input_split` — Describes what to read (base file path, log file paths, partition).
+ /// * `reader_parameters` — Reader flags (use_record_position, emit_delete, etc.).
+ /// * `data_schema` — Full table schema (what columns exist in the files).
+ /// Maps to Java's `dataSchema` / `tableSchema` parameter.
+ /// * `requested_schema` — Column projection requested by the caller.
+ /// Maps to Java's `requestedSchema` parameter. `None` means all columns.
+ pub fn new(
+ reader_context: Arc<ReaderContext>,
+ storage: Arc<Storage>,
+ input_split: InputSplit,
+ reader_parameters: ReaderParameters,
+ data_schema: Option<SchemaRef>,
+ requested_schema: Option<SchemaRef>,
+ ) -> Result<Self> {
+ log::debug!(
+ "HoodieFileGroupReader::new partition={} base_file={} log_files={} \
+ ordering_fields={:?} latest_commit_time={} record_key_field={}",
+ input_split.partition_path,
+ input_split.base_file_path.as_deref().unwrap_or("<none>"),
+ input_split.log_file_paths.len(),
+ reader_context.ordering_field_names(),
+ reader_context.latest_commit_time,
+ reader_context.record_key_field(),
+ );
+ for (i, lf) in input_split.log_file_paths.iter().enumerate() {
+ log::debug!(" log_file[{i}]: {lf}");
+ }
+
+ // Mirrors Java lines 119-121:
+ // readerContext.setSchemaHandler(
+ // new FileGroupReaderSchemaHandler(readerContext, dataSchema, requestedSchema, ...));
+ //
+ // When schemas are explicitly provided (direct construction / tests), create
+ // a new handler. When they are not provided (FFI path via builder), use the
+ // handler already on reader_context — which was populated by the FFI bridge
+ // from the Avro JSON schemas passed through the Substrait proto.
+ let mut schema_handler = if data_schema.is_some() || requested_schema.is_some() {
+ let mut handler = FileGroupReaderSchemaHandler::new();
+ if let Some(ds) = data_schema {
+ handler = handler.with_table_schema(ds.clone()).with_data_schema(ds);
+ }
+ if let Some(rs) = requested_schema {
+ handler = handler.with_requested_schema(rs);
+ }
+ handler
+ } else {
+ reader_context.schema_handler.clone()
+ };
+
+ // Mirrors Java FileGroupReaderSchemaHandler constructor line 105:
+ // this.requiredSchema = prepareRequiredSchema(this.deleteContext);
+ //
+ // Uses record_key_fields() (all key fields) instead of record_key_field()
+ // (single) to support composite record keys in virtual-key mode.
+ // Mirrors Java's getMandatoryFieldsForMerging() lines 250-258.
+ let has_instant_range = reader_context.instant_range.is_some();
+ schema_handler.prepare_required_schema(
+ input_split.has_log_files(),
+ &reader_context.record_key_fields(),
+ reader_context.ordering_field_names(),
+ &reader_context.table_config,
+ has_instant_range,
+ &reader_context.merge_mode,
+ )?;
+
+ // Schema-on-read (InternalSchema) evolution is not supported in hudi-rs
+ // (GAP-07). Gold loads an InternalSchema from the `.schema` folder and
+ // applies column renames / type changes through InternalSchema versioning
+ // when `hoodie.schema.on.read.enable=true`. hudi-rs only implements
+ // schema-on-write backward-compatible evolution, so silently honoring the
+ // flag would risk misreading evolved data. Reject it loudly at the same
+ // table-config chokepoint as the bootstrap gate below.
+ if reader_context
+ .table_config
+ .get("hoodie.schema.on.read.enable")
+ .map(|v| v.eq_ignore_ascii_case("true"))
+ .unwrap_or(false)
+ {
+ return Err(CoreError::Unsupported(format!(
+ "schema-on-read (InternalSchema) is not supported in hudi-rs. \
+ Table at '{}' has hoodie.schema.on.read.enable=true, which requires \
+ InternalSchema-based evolution (column renames / type changes) that \
+ hudi-rs does not implement; only schema-on-write backward-compatible \
+ evolution is supported.",
+ reader_context.table_path,
+ )));
+ }
+
+ // Bootstrap merge reordering is not yet supported in hudi-rs.
+ // Java's prepareRequiredSchema() (lines 280-288) partitions fields into
+ // meta and data columns and reorders them for bootstrap tables. Until
+ // that is implemented, reject bootstrap merge at construction time.
+ if reader_context.needs_bootstrap_merge {
+ // Reachable via table state (bootstrap base files), so this is a
+ // loud error rather than a panic.
+ return Err(CoreError::Unsupported(format!(
+ "Bootstrap merge is not yet supported in hudi-rs. \
+ Table at '{}' has bootstrap base files that require \
+ meta/data column reordering.",
+ reader_context.table_path,
+ )));
+ }
+
+ // Composite virtual keys ARE supported. With `hoodie.populate.meta.fields=false`
+ // and a multi-field recordkey, `RecordContext::record_key_array` reconstructs the
+ // full `field:val,field:val` merge key per row (mirroring Java
+ // `KeyGenerator.constructRecordKey`) on BOTH the base and log sides, so records
+ // sharing the first field but differing on a later one no longer collide. See
+ // `RecordContext::build_composite_record_key_array`.
+
+ // Multi-field (composite) precombine/ordering keys ARE supported.
+ // `RecordContext::new` splits a comma-separated `hoodie.table.precombine.field`
+ // / `hoodie.table.ordering.fields` into `ordering_field_names`, and
+ // `get_ordering_values` builds one `OrderingValue::Composite` per row from
+ // the per-field scalars (compared lexicographically field-by-field, mirroring
+ // Java `OrderingValues`). A field absent from a batch, an unsupported field
+ // type, or a null component falls back to natural order — matching the scalar
+ // path — so there is no silent first-field-only degradation. (Construction
+ // still rejects composite *virtual keys* above: that path reconstructs the
Review Comment:
This closing sentence — "Construction still rejects composite *virtual keys* above" — contradicts both the start of this same comment and `test_composite_virtual_keys_accepted_at_construction` below; the rejection it refers to no longer exists.
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/buffer/position_based.rs:
##########
@@ -0,0 +1,1491 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Some items exist only for parity
+//! with the Java reader and have no caller here.
+
+//! Position-based file-group record buffer.
+//!
+//! Mirrors Java `org.apache.hudi.common.table.read.buffer.PositionBasedFileGroupRecordBuffer`,
+//! which `extends KeyBasedFileGroupRecordBuffer`. Instead of keying the merge
+//! map by record key, it keys it by each log record's **position in the base
+//! file** — the row index recorded in the log block's `RECORD_POSITIONS` header.
+//! Base rows are matched to log records by their physical position (read from
+//! the synthetic row-index column attached to the base read), which is cheaper
+//! than key extraction + string hashing and is the layout Spark writes when
+//! `hoodie.merge.use.record.positions` is enabled.
+//!
+//! ## Composition
+//!
+//! Java extends the key-based buffer; here we **compose** it
+//! ([`inner`](PositionBasedFileGroupRecordBuffer::inner)) and reuse its
+//! vectorized/scalar merge kernels via the [`BaseMatch`] parameter, so the merge
+//! logic (winner selection, keep-mask, reconcile, update stats, spill map) is
+//! shared rather than duplicated. Positions are encoded as decimal-string map
+//! keys in the existing `String`-keyed spillable map; the map holds a single key
+//! domain at any instant — positions before fallback, record keys after.
+//!
+//! ## Fallback to key-based
+//!
+//! When a log block has no valid positions (missing / mismatched base-file
+//! instant, or an empty bitmap), the buffer permanently switches to key-based
+//! merge for the rest of the scan: it re-keys every already-buffered record from
+//! its position to its record key and delegates subsequent processing to the
+//! key-based super methods. This mirrors Java `fallbackToKeyBasedBuffer`.
+
+use std::collections::HashMap;
+use std::sync::Arc;
+
+use arrow_array::RecordBatch;
+use arrow_schema::SchemaRef;
+
+use crate::Result;
+use crate::error::CoreError;
+use crate::file_group::log_file::log_block::{LogBlock, LogBlockContent};
+use crate::file_group::reader_v2::buffer::key_based::{BaseMatch, KeyBasedFileGroupRecordBuffer};
+use crate::file_group::reader_v2::buffer::record_positions::{
+ base_row_position_array, decode_record_positions,
+};
+use crate::file_group::reader_v2::buffer::row_extraction::records_to_batch;
+use crate::file_group::reader_v2::buffer::spillable_map::{SpillConfig, SpillableRecordMap};
+use crate::file_group::reader_v2::buffer::{BufferType, HoodieFileGroupRecordBuffer};
+use crate::file_group::reader_v2::buffered_record::{BufferedRecord, DeleteRecord};
+use crate::file_group::reader_v2::merge_iterator::DEFAULT_BATCH_SIZE;
+use crate::file_group::reader_v2::reader_context::ReaderContext;
+use crate::file_group::reader_v2::update_processor::UpdateStats;
+
+/// A merge buffer that matches base rows to log records by base-file row
+/// position. See the module docs. Wraps a [`KeyBasedFileGroupRecordBuffer`] and
+/// falls back to it when positions are unavailable.
+#[derive(Debug)]
+pub struct PositionBasedFileGroupRecordBuffer {
+ /// The key-based buffer whose merge kernels/state this reuses. After a
+ /// fallback its records map holds record-key entries (position entries
+ /// otherwise).
+ inner: KeyBasedFileGroupRecordBuffer,
+ /// Commit/instant time of the base file being merged. Positions in a log
+ /// block are only usable when the block's
+ /// `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` header equals this.
+ base_file_instant_time: String,
+ /// Whether to still merge by position. Set true at construction; flipped
+ /// false permanently on the first fallback (mirrors Java
+ /// `readerContext.setShouldMergeUseRecordPosition(false)`).
+ should_use_position: bool,
+ /// Set during fallback when a position-only delete (a delete record without
+ /// a usable record key) could not be re-keyed and must keep being matched by
+ /// position. Effectively unreachable in hudi-rs (delete blocks always carry
+ /// record keys), but tracked for parity with Java's hybrid strategy.
+ needs_hybrid_strategy: bool,
+ /// Position-only deletes retained across a fallback (see
+ /// [`needs_hybrid_strategy`](Self::needs_hybrid_strategy)).
+ hybrid_deletes: HashMap<u64, BufferedRecord>,
+}
+
+impl PositionBasedFileGroupRecordBuffer {
+ /// Construct a position-based buffer. `base_file_instant_time` is the commit
+ /// time of the base file in this file slice (Java passes
+ /// `baseFile.getCommitTime()`), used to validate that a block's positions
+ /// were computed against this base file.
+ pub fn new(
+ reader_context: Arc<ReaderContext>,
+ merge_mode: String,
+ emit_delete: bool,
+ base_file_instant_time: String,
+ ) -> Result<Self> {
+ let inner = KeyBasedFileGroupRecordBuffer::new(reader_context, merge_mode, emit_delete)?;
+ Ok(Self {
+ inner,
+ base_file_instant_time,
+ should_use_position: true,
+ needs_hybrid_strategy: false,
+ hybrid_deletes: HashMap::new(),
+ })
+ }
+
+ /// The map key for a base-file position: its decimal string. Positions and
+ /// record keys never share the map at the same time (positions before
+ /// fallback, record keys after), so the string domains do not collide.
+ fn position_key(position: u64) -> String {
+ position.to_string()
+ }
+
+ /// Extract the block's record positions, validating the base-file instant.
+ ///
+ /// Mirrors Java `extractRecordPositions`: returns `Ok(None)` (→ fallback)
+ /// when the block's `BASE_FILE_INSTANT_TIME_OF_RECORD_POSITIONS` header is
+ /// missing / empty / different from this file group's base file instant, or
+ /// when the `RECORD_POSITIONS` bitmap is missing / empty.
+ fn extract_positions(&self, block: &LogBlock) -> Result<Option<Vec<u64>>> {
+ match block.base_file_instant_time_of_positions() {
+ Some(t) if !t.is_empty() && t == self.base_file_instant_time => {}
+ _ => {
+ log::debug!(
+ "[PositionBasedBuffer] falling back: block base-file instant of positions \
+ absent or != {}",
+ self.base_file_instant_time,
+ );
+ return Ok(None);
+ }
+ }
+ match block.record_positions_header() {
+ Some(encoded) if !encoded.is_empty() => {
+ // Writer-side invariant: `decode_record_positions` yields the
+ // Roaring64 bitmap's positions in ASCENDING order, and the
+ // caller zips them index-by-index with the block's records in
+ // file order. This matches Java's
+ // `PositionBasedFileGroupRecordBuffer`
+ // (`recordPositions.get(recordIndex++)`) and is only correct
+ // because the Hudi writer emits records within a block in
+ // ascending base-position order.
+ let positions = decode_record_positions(encoded)?;
+ if positions.is_empty() {
+ Ok(None)
+ } else {
+ Ok(Some(positions))
+ }
+ }
+ _ => Ok(None),
+ }
+ }
+
+ /// Permanently switch to key-based merge, re-keying every already-buffered
+ /// record from its position to its record key. Mirrors Java
+ /// `fallbackToKeyBasedBuffer`.
+ fn fallback_to_key_based(&mut self) -> Result<()> {
+ if !self.should_use_position {
+ return Ok(());
+ }
+ self.should_use_position = false;
+
+ // Rebuild the (possibly-spilled) position-keyed map as a record-key-keyed
+ // map. `drain_iter` yields values; the position keys are dropped.
+ //
+ // A record's `record_key` field is NOT trustworthy here: the spill tier
+ // reconstructs a drained record's `record_key` from its MAP key (see
+ // `RocksDbDiskMap::record_from_entry`), which in this buffer is the
+ // POSITION string, not the record key. So for data records the true key
+ // is re-extracted from the payload row itself. A delete tombstone has no
+ // payload to re-extract from — its `record_key` is only correct if the
+ // tombstone never spilled (hudi-rs delete blocks always carry record
+ // keys, and `process_next_deleted_record` preserves them in the
+ // in-memory tier). A delete without a usable record key cannot be
+ // re-keyed — Java keeps it position-keyed under the hybrid strategy, but
+ // that requires the position, which `drain_iter` does not surface; fail
+ // loudly rather than silently drop a delete (which would resurrect a
+ // deleted row). A SPILLED position-keyed delete comes back with its
+ // position string as `record_key` and is indistinguishable from a real
+ // numeric key — persisting the record key through the spill tier is the
+ // real fix (tracked as a follow-up outside this buffer).
Review Comment:
This looks already fixed: `record_from_entry` in `spillable_map.rs` persists and restores a delete's real key via `DiskLoc::Delete { record_key }`, with a comment naming this exact hazard — and I couldn't reproduce the mis-key described here. Could the comment be updated so the next reader doesn't chase a fixed bug?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
##########
crates/core/src/file_group/reader_v2/reader_context.rs:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+//! Ported from the merge-on-read reader. Nothing consumes it yet, so its
+//! items are unreachable from the crate's call graph until the reader wires in.
+
+//! Mirrors Java's `HoodieReaderContext<T>` — the engine-agnostic reader context
+//! that flows through the entire file group reader call stack.
+//!
+//! In Java, `HoodieReaderContext<T>` is an engine-specific object (Spark, Flink)
+//! that carries merge mode, instant range, schema handler, etc. In Rust, we work
+//! directly with Arrow `RecordBatch`es, so this is a plain data struct carrying
+//! the structured configuration that the reader stack needs.
+
+use super::record_context::RecordContext;
+use super::schema_handler::FileGroupReaderSchemaHandler;
+use crate::config::table::HudiTableConfig;
+use crate::storage::RowFilterBuilder;
+use crate::timeline::selector::InstantRange;
+use std::collections::{HashMap, HashSet};
+
+/// Owned inputs for the Gate-3 completed/inflight check (C-INFLIGHT-DELTA).
+///
+/// Mirrors the SET check Java's `BaseHoodieLogRecordReader.scanInternalV1` performs for
+/// `tableVersion < 8`: a data/delete block is skipped when its instant is in
+/// `filterInflights()` OR not in `filterCompletedInstants()` (with archived instants treated as
+/// committed). Held by the reader so it can be built once from the active timeline and borrowed for
+/// each scan.
+///
+/// The gate is only meaningful for tables with a v1 timeline layout (table version < 8). For v8+
+/// tables the per-delta-commit log files are already excluded at the completion-time file-slice
+/// level, so callers leave this `None` (see `forward_scan_pass1`), preserving prior behavior.
+///
+/// Lives here rather than with the scan that consumes it: [`ReaderContext`] holds it as a field, so
+/// defining it alongside the scanner would make the context and the scanner import each other.
+#[derive(Debug, Clone, Default)]
+pub struct CompletionGateInputs {
+ /// Active completed-commit instant times (Java `filterCompletedInstants()`).
+ pub completed_instants: HashSet<String>,
+ /// Active inflight/requested instant times (Java `filterInflights()`).
+ pub inflight_instants: HashSet<String>,
+ /// First active-timeline instant. An instant strictly before it is archived; archival only
+ /// removes completed instants, so an archived instant is committed. `None` disables the
+ /// archived fallback (every candidate must then be in `completed_instants`).
+ pub archived_boundary: Option<String>,
+}
+
+/// Config key selecting how a slice's base and log records are combined.
+/// `skip_merge` asks for them unmerged, which this reader does not implement and
+/// refuses in the buffer loader. Hudi's own spelling, so it is looked up by name
+/// rather than modelled as a [`HudiReadConfig`](crate::config::read::HudiReadConfig).
+pub const CONFIG_MERGE_TYPE: &str = "hoodie.datasource.merge.type";
+
+/// Reader context that flows through the file group reader call stack.
+///
+/// Mirrors Java's `HoodieReaderContext<T>`, carrying structured reader
+/// configuration instead of raw config key-value maps.
+///
+/// ## Java counterpart
+///
+/// | Java field / method | Rust field |
+/// |--------------------------------------------|-----------------------------------------|
+/// | `readerContext.getTablePath()` | `table_path` |
+/// | `readerContext.getLatestCommitTime()` | `latest_commit_time` |
+/// | `readerContext.getMergeMode()` | `merge_mode` |
+/// | `readerContext.getInstantRange()` | `instant_range` |
+/// | `readerContext.getRecordContext().format()` | `base_file_format` |
+/// | `readerContext.getHasLogFiles()` | `has_log_files` |
+/// | `readerContext.getRecordContext()` | `record_context` |
+/// | `readerContext.getSchemaHandler()` | `schema_handler` |
+/// | `metaClient.getTableConfig()` (config map) | `table_config` |
+/// | `props` (hoodie reader config overrides) | `hoodie_reader_config` |
+#[derive(Clone)]
+pub struct ReaderContext {
+ pub table_path: String,
+ pub latest_commit_time: String,
+ pub base_file_format: String,
+ pub has_log_files: bool,
+ pub has_bootstrap_base_file: bool,
+ pub needs_bootstrap_merge: bool,
+ pub should_merge_use_record_position: bool,
+ pub enable_logical_timestamp_field_repair: bool,
+ pub iterator_mode: String,
+ pub merge_mode: String,
+ pub merge_strategy_id: String,
+ pub instant_range: Option<InstantRange>,
+ /// The engine-specific record context for record-level operations.
+ ///
+ /// Mirrors Java's `HoodieReaderContext.recordContext` field.
+ /// In Java this is a persistent mutable field set at construction and
+ /// shared across all consumers. In Rust it is set once and shared via
+ /// `Arc<ReaderContext>`.
+ ///
+ /// Constructed from `table_config` + `partition_path`, mirroring Java's
+ /// `RecordContext(tableConfig, typeConverter)`.
+ pub record_context: RecordContext,
+ /// Schema management for the read pipeline.
+ ///
+ /// Mirrors Java's `HoodieReaderContext.schemaHandler` field
+ /// (`FileGroupReaderSchemaHandler<T>`).
+ pub schema_handler: FileGroupReaderSchemaHandler,
+ pub table_config: HashMap<String, String>,
+ /// Per-read overrides. Populated by
+ /// [`resolve_reader_context`](crate::file_group::reader_v2::resolver::resolve_reader_context)
+ /// from the `hoodie.read.*` keys plus the individually-named ones its
+ /// `READER_CONFIG_KEYS` lists — anything a consumer looks up here must
+ /// appear in one of those two, or it reads as unset.
+ pub hoodie_reader_config: HashMap<String, String>,
+ /// Optional parquet `RowFilter` builder for predicate pushdown into base
+ /// parquet files and parquet-format log blocks. Set by the FFI bridge from
Review Comment:
There are ~95 comments across `reader_v2/` referencing infrastructure that isn't in this repo — "the FFI bridge", "decoded substrait predicate", "Velox consumes the stream directly", `cpp/src/predicate.rs` (cpp/src has only lib.rs and util.rs), "#76/#95 review". For `row_filter_builder` and `completion_gate_inputs` it's more than tidiness: their only documented producer doesn't exist here, so inert-by-design and inert-by-accident read the same. Could these be rewritten against this crate's actual call graph?
_⚠️ AI-generated; verify before applying. React 👍/👎 to flag quality._
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(core): serve merge-on-read reads with file group reader version 2 [hudi-rs]
Posted by "yihua (via GitHub)" <gi...@apache.org>.
yihua commented on code in PR #660:
URL: https://github.com/apache/hudi-rs/pull/660#discussion_r3787738016
##########
crates/core/schemas/HoodieDeleteRecord.avsc:
##########
Review Comment:
Why is this `HoodieDeleteRecord` different than `HoodieDeleteRecordList.HoodieDeleteRecord`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org